Clustered Quartz wont't start in versions after AS 7.2.0.Alpha1

[bobzer]

Hello. I use Quartz 2.1.7 in my EAR deployment. Configuration is:

• quartz library placed in my.ear/lib folder
• quartz service started from .war placed under .ear
• quartz/jobs configuration files placed in .war
• using clustered config, so jobs readed from XML on service start saved in database

Under AS 7.2.0.Alpha1 (EAP 6.1.0 Alpha) started in Windows OS all works fine: Quartz service started, read jobs from XML config, save in DB, and do sheduler work, distributed between nodes of cluster. But since EAP 6.1.0 Beta there is some problem on start: Quartz service read and add 2 jobs (message in log) and stoped for long time. In database tables i see that actually added 1 job and no triggers. After 10 minutes in log added message about third job added, but no changes in database. After more 10 minutes in log i see fourth job added and second job added in database. Same behavior under the EAP 6.1.0 Final. Configuaration of my deployment is identical on all versions: on clean distribution of JBoss i deploy my app, performing the same operations (saved in my deployment instruction).

On the one hand, i understand that is maybe relates more to the Quartz engine, instead of JBoss, but the occurrence of error is related to change version of JBoss AS. I decided that maybe is related to Hibernate, because Hibernate version change has led to problems in my buisness logic too. I completely replace Hibernate module folder in JBoss, but have same Quartz behavior.

Some of the changes in JBoss since EAP 6.1.0 Beta affected the Quartz and i even don't know which way to look.

PS. My app is work fine on Alpha version, but IMHO - is bad practice to use Alpha on production environment when the Final version is out now.

[wdfink]

I'm not sure what the issue is. You should provide more details how you start the service. Maybe a simple example without DB access will show the behaviour.

But I would recommend to use the EJB3.1 compatible and portable way with @Schedule instead of Quartz.

BTW to use the 6.1.0.Final version in production you need to have a subscription.

[bobzer]

"you need to have a subscription" - thank you for pointed it out

"@Schedule instead of Quartz" - @Schedule additionaly need some clustering mechanism. I find some documentation about clustering EJB on JBoss, find import org.jboss.ejb3.annotation - this is not portable way (for example, where now are deploy-hasingleton folder or org.jboss.varia.scheduler from previos versions of JBoss?). Just placing quartz.lib in my EAR more portable, IMHO. In additional, i need way to manage jobs by system administrator, Quartz gives me the way to add/edit/delete jobs without programming - just edit config and restart.

I continue to replacing all modules and find "bad" module: it's modules\system\layers\base\org\jboss\ironjacamar. Its implementation changed from 1.0.15 to higher. If i replacing 1.0.17 that comes with JBoss Final to 1.0.15 that comes with JBoss Alpha, then Quartz work fine

[wdfink]

Thank you for isolate the problem so far.

Are you seeing any reason for that in the logfiles, maybe you need to add DEBUG or TRACE level?

If you find the difference and post the relevant part of log it will make it possible that someone with Ironjacamar knowlege can give you a hint or it is a bug and needs to be solved.

[bobzer]

OK, i turn on debug level on root-logger and have dosen of records. I had to cut part of the application to minimize the log. After quick analize i find out that probably afected connect to database process. There is no records in log direct from Ironjacamar, but suspisios info from org.jboss.jca.core.connectionmanager.pool.strategy.OnePool.

There is logs with Ironjacamar 1.0.15 deployed:

09:44:14,175 DEBUG [org.quartz.impl.jdbcjobstore.StdRowLockSemaphore] (ServerService Thread Pool -- 56) Lock 'TRIGGER_ACCESS' is desired by: ServerService Thread Pool -- 56

09:44:14,175 DEBUG [org.quartz.impl.jdbcjobstore.StdRowLockSemaphore] (ServerService Thread Pool -- 56) Lock 'TRIGGER_ACCESS' is being obtained: ServerService Thread Pool -- 56

09:44:14,176 DEBUG [org.quartz.impl.jdbcjobstore.StdRowLockSemaphore] (ServerService Thread Pool -- 56) Lock 'TRIGGER_ACCESS' given to: ServerService Thread Pool -- 56

09:44:14,183 DEBUG [org.quartz.impl.jdbcjobstore.StdRowLockSemaphore] (ServerService Thread Pool -- 56) Lock 'TRIGGER_ACCESS' returned by: ServerService Thread Pool -- 56

09:44:14,183 DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 56) OracleDS: returnConnection(10d1f3f, false) [1/500]

09:44:14,184 DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 56) OracleDS: getConnection(null, WrappedConnectionRequestInfo@1e3a949[userName=pfl_2]) [0/500]

09:44:14,214 DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 56) OracleDS: returnConnection(10d1f3f, false) [1/500]

09:44:14,214 DEBUG [org.quartz.xml.XMLSchedulingDataProcessor] (ServerService Thread Pool -- 56) Rescheduling job: DEFAULT.mq-actualize-abep with updated trigger: DEFAULT.mq-actualize-abep

09:44:14,214 DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 56) OracleDS: getConnection(null, WrappedConnectionRequestInfo@1e3a949[userName=pfl_2]) [0/500]

09:44:14,216 DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 56) OracleDS: returnConnection(10d1f3f, false) [1/500]

09:44:14,217 DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (ServerService Thread Pool -- 56) OracleDS: getConnection(null, WrappedConnectionRequestInfo@1e3a949[userName=pfl_2]) [0/500]

There is logs with Ironjacamar 1.0.17 deployed:

09:22:05,626 DEBUG [org.quartz.impl.jdbcjobstore.StdRowLockSemaphore] (ServerService Thread Pool -- 55) Lock 'TRIGGER_ACCESS' is desired by: ServerService Thread Pool -- 55

09:22:05,627 DEBUG [org.quartz.impl.jdbcjobstore.StdRowLockSemaphore] (ServerService Thread Pool -- 55) Lock 'TRIGGER_ACCESS' is being obtained: ServerService Thread Pool -- 55

09:22:22,707 DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (QuartzScheduler_QuartzGBDFL-WIN-66B0ADAQSCD1372130521956_ClusterManager) OracleDS: getConnection(null, WrappedConnectionRequestInfo@1758d2[userName=pfl_2]) [1/500]

09:22:22,719 DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (QuartzScheduler_QuartzGBDFL-WIN-66B0ADAQSCD1372130521956_ClusterManager) OracleDS: returnConnection(1e164e4, false) [2/500]

09:22:22,720 DEBUG [org.quartz.impl.jdbcjobstore.JobStoreCMT] (QuartzScheduler_QuartzGBDFL-WIN-66B0ADAQSCD1372130521956_ClusterManager) ClusterManager: Check-in complete.

09:22:42,708 DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (QuartzScheduler_QuartzGBDFL-WIN-66B0ADAQSCD1372130521956_ClusterManager) OracleDS: getConnection(null, WrappedConnectionRequestInfo@1758d2[userName=pfl_2]) [1/500]

09:22:42,721 DEBUG [org.jboss.jca.core.connectionmanager.pool.strategy.OnePool] (QuartzScheduler_QuartzGBDFL-WIN-66B0ADAQSCD1372130521956_ClusterManager) OracleDS: returnConnection(13c6c41, false) [2/500]

09:22:42,721 DEBUG [org.quartz.impl.jdbcjobstore.JobStoreCMT] (QuartzScheduler_QuartzGBDFL-WIN-66B0ADAQSCD1372130521956_ClusterManager) ClusterManager: Check-in complete.

09:22:51,862 DEBUG [org.apache.catalina.session] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Start expire sessions StandardManager at 1372130571862 sessioncount 0

The actions is similar but not identical, and i see that when the Quartz is request database connection, there is are lengthy delays.