1 Reply Latest reply on May 30, 2007 10:32 AM by timfox

deadlocks when stressing jms application on MS SQL 2005

pkorros May 30, 2007 9:52 AM

I get several database deadlocks like the one bellow, when i am trying to stress my application.

I am using JBoss 4.0.5.GA with JBoss messaging 1.2.0GA configured to use MS SqlServer 2005 as a persistent storage.

2007-05-30 16:34:17,426 WARN [org.jboss.messaging.core.plugin.JDBCPersistenceManager] (JMS SessionPool Worker-6) SQLException caught - assuming deadlock detected, try:1
java.sql.SQLException: Transaction (Process ID 148) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:365)
at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2781)
at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2224)
at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:628)
at net.sourceforge.jtds.jdbc.JtdsStatement.processResults(JtdsStatement.java:525)
at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:487)
at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeUpdate(JtdsPreparedStatement.java:421)
at net.sourceforge.jtds.jdbcx.proxy.PreparedStatementProxy.executeUpdate(PreparedStatementProxy.java:75)
at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:251)
at org.jboss.messaging.core.plugin.JDBCPersistenceManager.updateWithRetry(JDBCPersistenceManager.java:3522)
at org.jboss.messaging.core.plugin.JDBCPersistenceManager.updateWithRetry(JDBCPersistenceManager.java:3394)
at org.jboss.messaging.core.plugin.JDBCPersistenceManager.commitPreparedTransaction(JDBCPersistenceManager.java:3063)
at org.jboss.messaging.core.plugin.JDBCPersistenceManager.handleBeforeCommit2PC(JDBCPersistenceManager.java:2349)
at org.jboss.messaging.core.plugin.JDBCPersistenceManager$TransactionCallback.beforeCommit(JDBCPersistenceManager.java:3781)
at org.jboss.messaging.core.tx.Transaction.commit(Transaction.java:208)
at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendTransaction(ServerConnectionEndpoint.java:466)
at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.org$jboss$jms$server$endpoint$advised$ConnectionAdvised$sendTransaction$aop(ConnectionAdvised.java:101)
at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$sendTransaction_N3268650789275322226.invokeNext(ConnectionAdvised$sendTransaction_N3268650789275322226.java)
at org.jboss.jms.server.container.ServerLogInterceptor.invoke(ServerLogInterceptor.java:105)
at org.jboss.jms.server.endpoint.advised.ConnectionAdvised$sendTransaction_N3268650789275322226.invokeNext(ConnectionAdvised$sendTransaction_N3268650789275322226.java)
at org.jboss.jms.server.endpoint.advised.ConnectionAdvised.sendTransaction(ConnectionAdvised.java)
at org.jboss.jms.wireformat.ConnectionSendTransactionRequest.serverInvoke(ConnectionSendTransactionRequest.java:85)
at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:125)
at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
at org.jboss.remoting.Client.invoke(Client.java:1544)
at org.jboss.remoting.Client.invoke(Client.java:530)
at org.jboss.remoting.Client.invoke(Client.java:518)
at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:184)
at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:155)
at org.jboss.jms.client.delegate.ClientConnectionDelegate.org$jboss$jms$client$delegate$ClientConnectionDelegate$sendTransaction$aop(ClientConnectionDelegate.java:219)
at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N3268650789275322226.invokeNext(ClientConnectionDelegate$sendTransaction_N3268650789275322226.java)
at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:91)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N3268650789275322226.invokeNext(ClientConnectionDelegate$sendTransaction_N3268650789275322226.java)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:171)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N3268650789275322226.invokeNext(ClientConnectionDelegate$sendTransaction_N3268650789275322226.java)
at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N3268650789275322226.invokeNext(ClientConnectionDelegate$sendTransaction_N3268650789275322226.java)
at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
at org.jboss.jms.client.delegate.ClientConnectionDelegate$sendTransaction_N3268650789275322226.invokeNext(ClientConnectionDelegate$sendTransaction_N3268650789275322226.java)
at org.jboss.jms.client.delegate.ClientConnectionDelegate.sendTransaction(ClientConnectionDelegate.java)
at org.jboss.jms.tx.ResourceManager.sendTransactionXA(ResourceManager.java:627)
at org.jboss.jms.tx.ResourceManager.commit(ResourceManager.java:383)
at org.jboss.jms.tx.MessagingXAResource.commit(MessagingXAResource.java:235)
at org.jboss.tm.TransactionImpl$Resource.commit(TransactionImpl.java:2253)
at org.jboss.tm.TransactionImpl.commitResources(TransactionImpl.java:1784)
at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java:358)
at org.jboss.tm.TxManager.commit(TxManager.java:240)
at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:351)
at org.jboss.jms.client.remoting.MessageCallbackHandler.callOnMessage(MessageCallbackHandler.java:153)
at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:791)
at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:171)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:71)
at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:194)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:748)
at java.lang.Thread.run(Thread.java:595)
2007-05-30 16:34:17,426 WARN [org.jboss.messaging.core.plugin.JDBCPersistenceManager] (JMS SessionPool Worker-20) Update worked after retry
2007-05-30 16:34:17,441 WARN [org.jboss.messaging.core.plugin.JDBCPersistenceManager] (JMS SessionPool Worker-22) Trying again after a pause
2007-05-30 16:34:17,441 WARN [org.jboss.messaging.core.plugin.JDBCPersistenceManager] (JMS SessionPool Worker-6) Trying again after a pause
2007-05-30 16:34:17,441 WARN [org.jboss.messaging.core.plugin.JDBCPersistenceManager] (JMS SessionPool Worker-17) Update worked after retry
2007-05-30 16:34:17,457 WARN [org.jboss.messaging.core.plugin.JDBCPersistenceManager] (JMS SessionPool Worker-18) Update worked after retry
2007-05-30 16:34:17,551 WARN [org.jboss.messaging.core.plugin.JDBCPersistenceManager] (JMS SessionPool Worker-15) Update worked after retry
2007-05-30 16:34:17,707 WARN [org.jboss.messaging.core.plugin.JDBCPersistenceManager] (JMS SessionPool Worker-23) Update worked after retry
2007-05-30 16:34:17,832 WARN [org.jboss.messaging.core.plugin.JDBCPersistenceManager] (JMS SessionPool Worker-22) Update worked after retry
2007-05-30 16:34:17,894 WARN [org.jboss.messaging.core.plugin.JDBCPersistenceManager] (JMS SessionPool Worker-21) Update worked after retry
2007-05-30 16:34:17,925 WARN [org.jboss.messaging.core.plugin.JDBCPersistenceManager] (JMS SessionPool Worker-6) Update worked after retry

This greatly hurts the performance of my application, any ideas on what is causing the problem?

1. Re: deadlocks when stressing jms application on MS SQL 2005

timfox May 30, 2007 10:32 AM (in response to pkorros)

Deadlocks are often inevitable depending on your application usage patterns, this is why we defensively program for this, and catch deadlock and retry after an interval, which is a very standard way of dealing with deadlock.

It should be harmless.

If you are getting a lot, you should think whether you're doing a lot of stuff on a competing resource. Maybe you have many consumers consuming from the same queue and acknowledging, committng or rolling back concurrently?

Moreover with some databases, (I don't know about SQL Server), but with MySQL you can actually get deadlock even if the two competing transactions are only updating a single row.
Actions