4 Replies Latest reply on Dec 13, 2013 8:16 AM by mkouba

    Handling exceptions in Quartz Threads

    ibenjes

      Hi,

       

      I've got a pool of 10 threads and a few Cron jobs. These cron jobs check several different FTP/SFTP or POP connections at different intervals for incoming messages, add them to a processing queue or process the actual messages from the queue.

       

      That all works fine until I get an exception in one of the threads e.g. DefaultQuartzScheduler_Worker-1 (in this case it was a ValidationException).

      After that the thread is broken and no transaction is active when it tries to access the database:

       

      15:30:06,231 WARN  [][][org.hibernate.engine.jdbc.spi.SqlExceptionHelper][SqlExceptionHelper] (DefaultQuartzScheduler_Worker-1) SQLSQL Error: 0, SQLState: null

      15:30:06,231 ERROR [][][org.hibernate.engine.jdbc.spi.SqlExceptionHelper][SqlExceptionHelper] (DefaultQuartzScheduler_Worker-1) javax.resource.ResourceException: IJ000460: Error checking for a transaction

       

      The cron jobs continue to work fine on the Threads DefaultQuartzScheduler_Worker-2 to DefaultQuartzScheduler_Worker-10 but whenever DefaultQuartzScheduler_Worker-1 is used it fails to do anything involving the database.

       

      How can I recover from that? I basically either want to remove that thread from the pool or 'repair' it.

       

      I've seen the chapter on Handling Async Exceptions in the Seam documentation using the org.jboss.seam.async.asynchronousExceptionHandler:

      Chapter 22. Asynchronicity and messaging

       

      That is probably the right place to start but what do I do there? How can I access the thread pool and remove one thread from it or repair it?

       

      I don't want to cancel the other cron jobs.

       

      Environment: Seam 2.3 with Quartz on JBoss EAP 6.1

        • 1. Re: Handling exceptions in Quartz Threads
          mkouba

          Hi Immo,

          a few questions:

          1. What version of Quartz do you use?
          2. Do you use Seam async dispatcher (i.e. @Asynchronous) or Quartz API?
          3. Do you provide your own quartz configuration file (quartz.properties)?
          • 2. Re: Handling exceptions in Quartz Threads
            mkouba

            Also don't forget to have I/O socket timeouts defined. Otherwise the thread may hang in I/O wait indefinitely (i.e. will not be used for other job executions). I ran into a similar issue when using commons-mail to send an email via SMTP (there was no timeout by default - fixed in https://issues.apache.org/jira/browse/EMAIL-100).

            • 3. Re: Handling exceptions in Quartz Threads
              ibenjes

              I am not quite sure what Quartz version is used, I am using the one that is bundled with Seam. For Seam 2.2 this was Quartz 1.6, I am not sure whether that was updated for Seam 2.3 (1.6 sounds like it is very old).

               

              We don't use the Quartz API directly, we only use the Seam async dispatcher and we don't have any custom quartz configuration (no seam.quartz.properties file).

               

              Blocking sockets is not the problem here (although I had a similar problem with a misconfigured POP3 connection a while ago).

              • 4. Re: Handling exceptions in Quartz Threads
                mkouba

                Seam 2.3 depends on Quartz 1.6.5 and yes it's quite an outdated version. Anyway org.jboss.seam.async.QuartzDispatcher.QuartzJob is catching all exceptions and the default AsynchronousExceptionHandler just logs the exception. The log message you posted indicates a transaction was not completed correctly - i.e. no commit/rollback - and the thread has some invalid tx info associated. As far as I know there's no "clean" way to remove a thread from the thread pool in Quartz. So the best solution would be to handle a transaction rollback correctly (if ValidationException occurs). Do you use Seam managed transactions in your app?