JBossDeveloper

This is the deadlock:

ReceiverAspect.handleReceive is trying to communicate back to the server, and the aspect is holding its communication (as it's supposed to do), while handleFailoverOnConsumer is trying to change MessageCallbackHandler properties.

"LocalThreadConsumer-1" prio=1 tid=0x8b9cf568 nid=0x228e in Object.wait() [0x8a66f000..0x8a66feb0]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x91652838> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
 at java.lang.Object.wait(Object.java:474)
 at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(WriterPreferenceReadWriteLock.java:240)
 - locked <0x91652838> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
 at org.jboss.jms.client.container.ValveAspect.invoke(ValveAspect.java:107)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$acknowledgeDelivery_N3172014476730533936.invokeNext(ClientSessionDelegate$acknowledgeDelivery_N3172014476730533936.java)
 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:177)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$acknowledgeDelivery_N3172014476730533936.invokeNext(ClientSessionDelegate$acknowledgeDelivery_N3172014476730533936.java)
 at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:68)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$acknowledgeDelivery_N3172014476730533936.invokeNext(ClientSessionDelegate$acknowledgeDelivery_N3172014476730533936.java)
 at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$acknowledgeDelivery_N3172014476730533936.invokeNext(ClientSessionDelegate$acknowledgeDelivery_N3172014476730533936.java)
 at org.jboss.jms.client.delegate.ClientSessionDelegate.acknowledgeDelivery(ClientSessionDelegate.java)
 at org.jboss.jms.client.container.SessionAspect.ackDelivery(SessionAspect.java:453)
 at org.jboss.jms.client.container.SessionAspect.handlePostDeliver(SessionAspect.java:283)
 at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect7.invoke(SessionAspect7.java)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$postDeliver_5319211143798977162.invokeNext(ClientSessionDelegate$postDeliver_5319211143798977162.java)
 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:177)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$postDeliver_5319211143798977162.invokeNext(ClientSessionDelegate$postDeliver_5319211143798977162.java)
 at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:68)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$postDeliver_5319211143798977162.invokeNext(ClientSessionDelegate$postDeliver_5319211143798977162.java)
 at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$postDeliver_5319211143798977162.invokeNext(ClientSessionDelegate$postDeliver_5319211143798977162.java)
 at org.jboss.jms.client.delegate.ClientSessionDelegate.postDeliver(ClientSessionDelegate.java)
 at org.jboss.jms.client.remoting.MessageCallbackHandler.receive(MessageCallbackHandler.java:419)
 - locked <0x91652338> (a java.lang.Object)
 at org.jboss.jms.client.container.ReceiverAspect.handleReceive(ReceiverAspect.java:63)
 at org.jboss.aop.advice.org.jboss.jms.client.container.ReceiverAspect25.invoke(ReceiverAspect25.java)
 at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)
 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:177)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:117)
 at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)
 at org.jboss.jms.client.container.ExceptionInterceptor.invoke(ExceptionInterceptor.java:68)
 at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)
 at org.jboss.jms.client.container.ClientLogInterceptor.invoke(ClientLogInterceptor.java:107)
 at org.jboss.jms.client.delegate.ClientConsumerDelegate$receive_N8299950230150603585.invokeNext(ClientConsumerDelegate$receive_N8299950230150603585.java)
 at org.jboss.jms.client.delegate.ClientConsumerDelegate.receive(ClientConsumerDelegate.java)
 at org.jboss.jms.client.JBossMessageConsumer.receive(JBossMessageConsumer.java:86)
 at org.jboss.test.messaging.jms.clustering.ValveTest$LocalThreadConsumer.run(ValveTest.java:94)


 "Thread-6" daemon prio=1 tid=0x08183840 nid=0x22d2 waiting for monitor entry [0x8aefc000..0x8aefd030]
 at org.jboss.jms.client.remoting.MessageCallbackHandler.copyState(MessageCallbackHandler.java:500)
 - waiting to lock <0x91652338> (a java.lang.Object)
 at org.jboss.jms.client.container.HAAspect.handleFailoverOnConsumer(HAAspect.java:605)
 at org.jboss.jms.client.container.HAAspect.performClientSideFailover(HAAspect.java:451)
 at org.jboss.jms.client.container.HAAspect.handleConnectionFailure(HAAspect.java:337)
 at org.jboss.jms.client.container.ValveAspect.handleConnectionFailure(ValveAspect.java:202)
 at org.jboss.jms.client.container.ValveAspect$ConnectionFailureListener.handleConnectionException(ValveAspect.java:241)
 at org.jboss.jms.client.remoting.ConsolidatedRemotingConnectionListener.handleConnectionException(ConsolidatedRemotingConnectionListener.java:83)
 at org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:139)

This dead lock has also to do with another issue I'm investigating, not necessarily with Valves but with the behavior on failover itself.

It looks like ACK is called as soon as the message arrives on MessageCallbackHandler, what might be causing the reference to be deleted.

In case of a failure, the buffer is being cleared at this point, and we will call recoverDeliverie, but as the message was previously acknowledged already it won't exist on the channel, what will cause an exception on ChannelSupport::recoverDeliveries.

The dead lock was fixed:

We have created a JIRA issue to track the change:

http://jira.jboss.com/jira/browse/JBMESSAGING-702


We still need to verify HAStressTest failures regarding ACKs.