We are using Quartz with Spring and our configuration is throwing deadlocks when quartz has more than 1 thread configured. I'm starting to believe that it's because we don't have our quartz configured correctly with Spring, but I can't find enough documentation on how to configure the two to play nicely. I'm also starting to believe that Quartz is not going to be our solution, unfortunately, because of this issue.
We are running tests under heavy load, starting as many as 1000 jobs in a matter of minutes, all scheduled to be picked up by 50 quartz threads after 10 second delays.
We are running on both Windows and Linux environments - pointing at MSSQL and Oracle DBs. With both OS, using either DB, we can throw the following deadlock errors...
2010-03-17 18:52:31,737   ERROR [DocFinityScheduler_Worker-42] core.ErrorLogger core.ErrorLogger (QuartzScheduler.java:2185) - An error occured while marking executed job complete. job= 'BPM.6e41a6567f0000020100362a51dc7a50'
org.quartz.JobPersistenceException: Couldn't remove trigger: Transaction (Process ID 87) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. [See nested exception: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 87) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.]
at org.quartz.impl.jdbcjobstore.JobStoreSupport.removeTrigger(JobStoreSupport.java:1469)at org.quartz.impl.jdbcjobstore.JobStoreSupport.triggeredJobComplete(JobStoreSupport.java:2978)at org.quartz.impl.jdbcjobstore.JobStoreSupport$39.execute(JobStoreSupport.java:2962) at org.quartz.impl.jdbcjobstore.JobStoreSupport$41.execute(JobStoreSupport.java:3713)at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3747)
at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3709)at org.quartz.impl.jdbcjobstore.JobStoreSupport.triggeredJobComplete(JobStoreSupport.java:2958)at org.quartz.core.QuartzScheduler.notifyJobStoreJobComplete(QuartzScheduler.java:1727)at org.quartz.core.JobRunShell.run(JobRunShell.java:273)
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 87) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(Unknown Source at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(Unknown Source) at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(Unknown Source) at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(Unknown Source) at com.microsoft.sqlserver.jdbc.TDSCommand.execute(Unknown Source) at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(Unknown Source) at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(Unknown Source)at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(Unknown Source) at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeUpdate(Unknown Source)at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeUpdate(NewProxyPreparedStatement.java:105) at org.quartz.impl.jdbcjobstore.StdJDBCDelegate.deleteSimpleTrigger(StdJDBCDelegate.java:1820) at org.quartz.impl.jdbcjobstore.JobStoreSupport.deleteTriggerAndChildren(JobStoreSupport.java:1345 at org.quartz.impl.jdbcjobstore.JobStoreSupport.removeTrigger(JobStoreSupport.java:1453 ... 9 more
I have attached as many files as I could think of that relate to the issue. Can anybody help me out here? Do I have it configured incorrectly to not use a JobStoreCMT versus a JobStoreTX? Do I have it configured incorrectly in terms of threads or something else?
I'm really at a loss. We've been scouring the web for this stuff for documentation for two days and are still scratching our heads.