Deadlock during JBoss transaction processing
Hi,
I have deployed WebMethods Broker in jboss-eap-5.1. We are testing case where transactions times out and new transactions will be started, these also times out so on..This process will repeat as long as JBoss is running.
During this we have noticed deadlock sometimes. Transactions never timeout.
Captured thread dump is attached.
Deadlock is as below:
Found one Java-level deadlock:
=============================
"WorkManager(2)-1":
waiting to lock monitor 0x00000000561c5490 (object 0x00000000cc460c08, a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction),
which is held by "Thread-10"
"Thread-10":
waiting for ownable synchronizer 0x00000000cc461658, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
which is held by "WorkManager(2)-1"
Java stack information for the threads listed above:
===================================================
"WorkManager(2)-1":
at com.arjuna.ats.arjuna.coordinator.BasicAction.removeChildThread(BasicAction.java:916)
- waiting to lock <0x00000000cc460c08> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)
at com.arjuna.ats.internal.arjuna.thread.ThreadActionData.purgeActions(ThreadActionData.java:248)
at com.arjuna.ats.internal.arjuna.thread.ThreadActionData.purgeActions(ThreadActionData.java:224)
at com.arjuna.ats.arjuna.AtomicAction.suspend(AtomicAction.java:334)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionManagerImple.suspend(TransactionManagerImple.java:76)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.suspend(BaseTransactionManagerDelegate.java:143)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:363)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
at org.jboss.ejb.plugins.RunAsSecurityInterceptor.process(RunAsSecurityInterceptor.java:133)
at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:103)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:420)
at org.jboss.ejb.Container.invoke(Container.java:1092)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:283)
at org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)
at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)
at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)
at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)
at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)
at org.jboss.ejb.plugins.inflow.MessageEndpointInterceptor.delivery(MessageEndpointInterceptor.java:249)
at org.jboss.ejb.plugins.inflow.MessageEndpointInterceptor.invoke(MessageEndpointInterceptor.java:128)
at org.jboss.proxy.ClientMethodInterceptor.invoke(ClientMethodInterceptor.java:74)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:101)
at $Proxy111.onMessage(Unknown Source)
at com.sun.genericra.inbound.async.DeliveryHelper.deliverMessage(DeliveryHelper.java:301)
at com.sun.genericra.inbound.async.DeliveryHelper.deliver(DeliveryHelper.java:196)
at com.sun.genericra.inbound.async.DeliveryHelper.deliver(DeliveryHelper.java:186)
at com.sun.genericra.inbound.async.EndpointConsumer.consumeMessage(EndpointConsumer.java:176)
at com.sun.genericra.inbound.async.MessageListener.onMessage(MessageListener.java:56)
at com.webmethods.jms.loadbalance.connection.WmClusterSessionImpl.run(WmClusterSessionImpl.java:2014)
at com.sun.genericra.inbound.async.WorkImpl.run(WorkImpl.java:53)
at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:205)
at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
"Thread-10":
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000cc461658> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
at com.webmethods.locks.BasicJMSLockManager.takeObjectWriteLock(BasicJMSLockManager.java:53)
at com.webmethods.jms.loadbalance.connection.WmClusterXASessionImpl.rollback(WmClusterXASessionImpl.java:1585)
at com.sun.genericra.inbound.InboundXAResourceProxy.rollback(InboundXAResourceProxy.java:260)
at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelAbort(XAResourceRecord.java:420)
at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3270)
at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3248)
at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1933)
- locked <0x00000000cc460c08> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:114)
at com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:213)
at com.arjuna.ats.arjuna.coordinator.TransactionReaper.doCancellations(TransactionReaper.java:444)
at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:91)
Found 1 deadlock.
Any idea what caused this deadlock?
Found a issue reported similar to this: https://community.jboss.org/message/509851
-
stack_deadlock.txt.zip 5.3 KB
