Will your Tomcat instance not shut down because of long-running Quartz jobs?

I had that problem. Here is what I did…

Reproduce the problem

So first let’s reproduce the problem using Tomcat by creating a simple web app that runs a Quartz job.

We configure the Quartz Scheduler using the Spring SchedulerFactoryBean supplying a job detail and trigger bean as follows:

application-context.xml
 1 2 3 4 5 6 7 8 9101112131415161718192021222324252627282930313233
<bean id="simple.quartz.scheduler" class="org.springframework.scheduling.quartz.SchedulerFactoryBean"
    p:schedulerName="simple-scheduler"
    p:configLocation="classpath:quartz.properties"
    p:applicationContextSchedulerContextKey="applicationContext"
    p:autoStartup="true">

    <property name="jobFactory">
        <bean class="org.springframework.scheduling.quartz.SpringBeanJobFactory"/>
    </property>
    <property name="jobDetails">
        <list>
            <ref bean="simpleJob"/>
        </list>
    </property>
    <property name="triggers">
        <list>
            <ref bean="simpleJobTrigger"/>
        </list>
    </property>
</bean>

<bean id="simpleJobTrigger" class="org.springframework.scheduling.quartz.SimpleTriggerFactoryBean"
    p:jobDetail-ref="simpleJob"
    p:repeatInterval="60000"
    p:startDelay="10000"
    p:group="SIMPLE_JOB_GROUP"
    p:name="SIMPLE_JOB"/>

<bean id="simpleJob" class="org.springframework.scheduling.quartz.JobDetailFactoryBean"
    p:jobClass="com.iancollington.blog.simplequartzwebapp.SimpleJob"
    p:durability="true"
    p:group="SIMPLE_JOB_GROUP"
    p:name="SIMPLE_JOB"/>

The QuartzJobBean implementation is simple enough in that it runs forever:

SimpleJob.java

 1 2 3 4 5 6 7 8 91011121314151617
public class SimpleJob extends QuartzJobBean {

    private static final Logger LOGGER = LoggerFactory.getLogger(SimpleJob.class);

    @Override
    protected void executeInternal(JobExecutionContext context) throws JobExecutionException {
        while (true) {
            LOGGER.info("Long running job executing....");

            try {
                Thread.sleep(10000);
            } catch (InterruptedException e) {
                LOGGER.error("The thread sleep was interrupted.", e);
            }
        }
    }
}

Upon building and deploying the WAR file in Tomcat, you should see the following output every ten seconds:

INFO - Long running job executing....

If you gracefully stop Tomcat like so:

$ ./catalina.sh stop

then you will find that the above message continues until the Tomcat process is forcefully killed.

:
INFO: Stopping service Catalina
INFO: Stopping ProtocolHandler ["http-bio-8080"]
INFO: Stopping ProtocolHandler ["ajp-bio-8009"]
INFO: Destroying ProtocolHandler ["http-bio-8080"]
INFO - Long running job executing....
INFO: Destroying ProtocolHandler ["ajp-bio-8009"]
INFO - Long running job executing....
INFO - Long running job executing....
INFO - Long running job executing....
:

The complete source for this simple web app can be found on GitHub in the simple-quartz-webapp repository.

So how do we fix this?

There are two steps. First, we need to detect when the container is being shut down. Then we need to gracefully stop the Quartz jobs.

Detecting container shutdown

To detect when the container is being shut down we can hook into the Spring container lifecycle using an org.springframework.context.SmartLifecycle bean. The important part of this interface is the one it extends; org.springframework.context.Phased. This interface contains a single method named getPhase that returns a primitive integer.

During Spring container startup, Spring will call the start method on beans implementing this interface starting with the lowest phase value and ending with the highest phase value. The shutdown process is the reverse where Spring will call the stop method but starting with the highest phase value and ending with the lowest.

We want to be able to find out when the Spring container is stopping as quickly as possible such that we can stop the Quartz jobs as a soon as possible. The quicker the jobs are stopped the quicker Tomcat can shut down. We, therefore, need to create a bean that returns the highest possible phase value so that it is invoked as soon as possible when the Spring container starts to shut down.

SpringShutdownHook.java

 1 2 3 4 5 6 7 8 9101112131415161718192021222324252627282930313233
public class SpringShutdownHook implements SmartLifecycle {

    private static final Logger LOGGER = LoggerFactory.getLogger(SpringShutdownHook.class);

    private boolean isRunning = false;

    public boolean isAutoStartup() {
        return true;
    }

    public void stop(Runnable runnable) {
        stop();
        runnable.run();
    }

    public void start() {
        LOGGER.info("SpringShutdownHook started.");
        isRunning = true;
    }

    public void stop() {
        LOGGER.info("Spring container is shutting down.");
        isRunning = false;
    }

    public boolean isRunning() {
        return isRunning;
    }

    public int getPhase() {
        return Integer.MAX_VALUE;
    }
}
Add the new bean to the Spring config:

application-context.xml

123
<bean
  id="springShutdownHook"
  class="com.iancollington.blog.simplequartzwebapp.SpringShutdownHook" />

Upon starting Tomcat you will see the log output signalling that the SpringShutdownHook bean has been started:

INFO - SpringShutdownHook started.

You will then see the output from the Quartz job:

:
INFO - Long running job executing....
INFO - Long running job executing....
INFO - Long running job executing....
:
If you gracefully stop Tomcat like so:

$ ./catalina.sh stop

You will then see that the shutdown hook is invoked by the Spring framework.

INFO - Spring container is shutting down.

Now that we know when the Spring container is being shut down we can gracefully stop our Quartz jobs…

Gracefully stop the Quartz jobs

Quartz does not provide the facility to simply stop running Quartz jobs. Doing so could leave your application in a corrupt state. What it provides is a mechanism to interrupt running jobs allowing them to tidy up before stopping. The org.quartz.Scheduler class contains an interrupt(JobKey) method for this purpose. Any job that is interruptible should implement the org.quartz.InterruptableJob interface which provides an interrupt method.

We can modify the Quartz job class as follows to implement the InterruptableJob interface:

SimpleJob.java
 1 2 3 4 5 6 7 8 9101112131415161718192021222324252627282930313233
public class SimpleJob extends QuartzJobBean implements InterruptableJob {

    private static final Logger LOGGER = LoggerFactory.getLogger(SimpleJob.class);

    /**
     * Flag to indicate whether this job has been interrupted. When this flag becomes true the job should stop what it
     * is doing and finish.
     */
    private boolean hasBeenInterrupted = false;

    @Override
    protected void executeInternal(final JobExecutionContext context) throws JobExecutionException {
        while (!hasBeenInterrupted) {
            LOGGER.info("Long running job executing....");

            doSomeTask();
        }

        LOGGER.info("Long running job has been interrupted. Stopping.");
    }

    private void doSomeTask() {
        try {
            Thread.sleep(10000);
        } catch (InterruptedException e) {
            LOGGER.error("The thread sleep was interrupted.", e);
        }
    }

    public void interrupt() throws UnableToInterruptJobException {
        hasBeenInterrupted = true;
    }
}

So that the interrupt method can be invoked on the Scheduler class we need to inject it into our SpringShutdownHook bean:

SpringShutdownHook.java
123456
private Scheduler scheduler;

@Required
public void setScheduler(Scheduler scheduler) {
    this.scheduler = scheduler;
}
application-context.xml
1234
<bean
  id="springShutdownHook"
  class="com.iancollington.blog.simplequartzwebapp.SpringShutdownHook"
     p:scheduler-ref="simple.quartz.scheduler"/>

We can then modify the stop method to interrupt the jobs when the Spring container is being shut down like so:

SpringShutdownhook.java
 1 2 3 4 5 6 7 8 91011121314151617181920212223242526
public void stop() {
    LOGGER.info("Spring container is shutting down.");
    isRunning = false;

    try {
        interruptJobs();

        // Tell the scheduler to shutdown allowing jobs to complete
        scheduler.shutdown(true);
    } catch (SchedulerException e) {
        try {
            // Something has gone wrong so tell the scheduler to shutdown without allowing jobs to complete.
            scheduler.shutdown(false);
        } catch (SchedulerException ex) {
            LOGGER.error("Unable to shutdown the Quartz scheduler.", ex);
        }
    }
}

private void interruptJobs() throws SchedulerException {
    for (JobExecutionContext jobExecutionContext : scheduler.getCurrentlyExecutingJobs()) {
        final JobDetail jobDetail = jobExecutionContext.getJobDetail();
        LOGGER.info("Interrupting job key=[{}], group=[{}].", jobDetail.getKey().getName(), jobDetail.getKey().getGroup());
        scheduler.interrupt(jobDetail.getKey());
    }
}

So if we now build the code and start Tomcat, you should see the following output every ten seconds:

INFO - Long running job executing....
If you then gracefully stop Tomcat like so:

$ ./catalina.sh stop

You should see the SpringShutdownHook come into action and stop the running Quartz jobs before Tomcat stops.

INFO - Long running job executing....
INFO - Long running job executing....
INFO - Spring container is shutting down.
INFO - Interrupting job key=[SIMPLE_JOB], group=[SIMPLE_JOB_GROUP].
INFO - Long running job has been interrupted. Stopping.
INFO - Shutting down Quartz Scheduler

The complete code can be found in the simple-graceful-quartz-webapp repository.