BisocketClientInvoker deadlock| JBoss.org Content Archive (Read Only)

15. Re: BisocketClientInvoker deadlock

clebert.suconic May 13, 2008 5:44 PM (in response to ykonrad)

This one is all yours :-)

You have one thread holding a lock here:

- locked <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
 at com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl.send(GWCommandListenerImpl.java:207)

While... all the others:

at com.cisco.nm.cmp.gjc.GWConnectionImpl.executeCommand(GWConnectionImpl.java:118)
 - waiting to lock <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)

When I said all yours.. I assumed you work for cisco.
You should contact the developers for com.cisco.nm.cmp.gjc.GWConnectionImp.

16. Re: BisocketClientInvoker deadlock

ykonrad May 14, 2008 2:32 AM (in response to ykonrad)

Thank you for the quick and the helpful response.

17. Re: BisocketClientInvoker deadlock

ykonrad May 14, 2008 9:59 AM (in response to ykonrad)

Hi

All other threads are waiting for lock <0x471a79e0> and yes, it is locked by the following thread in:

 - locked <0x471a79e0> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
 at com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl.send(GWCommandListenerImpl.java:207)

But, I still can't understand why this thread is waiting on

 at java.lang.Object.wait(Native Method)
 - waiting on <0x47080560> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
 at java.lang.Object.wait(Object.java:474)
 at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:163)
 - locked <0x47080560> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)
 at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2075)
 at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
 at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:741)
 at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)

While no other thread is holding 0x47080560

Can you please advise?

18. Re: BisocketClientInvoker deadlock

clebert.suconic May 14, 2008 2:30 PM (in response to ykonrad)

Well... maybe this means the lock is not totally yours.

This thread is trying to acquire a writeLock:

"Timer-8" daemon prio=10 tid=0x0238ff18 nid=0x93 in Object.wait() [0xd6caf000..0xd6cafaf0]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x47080570> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
 at java.lang.Object.wait(Object.java:474)
 at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock.acquire(WriterPreferenceReadWriteLock.java:240)
 - locked <0x47080570> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$WriterLock)
 at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.removeBindingInMemory(MessagingPostOffice.java:2280)
 at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.internalRemoveBinding(MessagingPostOffice.java:1902)
 at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.removeBinding(MessagingPostOffice.java:464)
 at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.close(ServerConnectionEndpoint.java:400)
 - locked <0x4b9a2598> (a java.util.HashSet)
 at org.jboss.jms.server.connectionmanager.SimpleConnectionManager.closeConsumersForClientVMID(SimpleConnectionManager.java:377)
 - locked <0x47196dc8> (a org.jboss.jms.server.connectionmanager.SimpleConnectionManager)

While this thread, holding a readLock is trying to connect back to the client:

"WorkManager(2)-10" daemon prio=10 tid=0x02d1e5a8 nid=0xc7 in Object.wait() [0xd6dea000..0xd6def8f0]
 at java.lang.Object.wait(Native Method)
 - waiting on <0xb94ac400> (a java.util.HashSet)
 at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:473)
 - locked <0xb94ac400> (a java.util.HashSet)
 at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:801)
 at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:551)
 at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:418)
 at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
 at org.jboss.remoting.Client.invoke(Client.java:1634)
 at org.jboss.remoting.Client.invoke(Client.java:548)
 at org.jboss.remoting.Client.invokeOneway(Client.java:598)
 at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:826)
 at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:697)
 at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1432)
 at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1344)
 - locked <0x4ba02568> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint)
 at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:322)
 - locked <0x4ba347f0> (a java.lang.Object)
 at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)

Do you think you could give it a try with the SVN:

svn co http://anonsvn.jboss.org/repos/messaging/branches/Branch_JBossMessaging_1_4_0_SP3_CP/ jbm
cd jbm
ant release-bundle

and let me know how it goes?

19. Re: BisocketClientInvoker deadlock

clebert.suconic May 14, 2008 2:42 PM (in response to ykonrad)

I believe you have a support contract.
If you do, have you tried using the support portal?

20. Re: BisocketClientInvoker deadlock

azafran May 14, 2008 3:27 PM (in response to ykonrad)

I'll elaborate on the previous:

The "Notify Thread" locks other threads since it acquires the lock on UserConnectionImpl. What we're worried about is why this thread "hangs" while trying to send a message. When reproducing this phenomena, we've witnessed two scenarios:
1) "Notify Thread" hangs on MessagingPostOffice line 2075, as previously posted
2) "Notify Thread" hangs on BisocketClientInvoker line 473, as follows

"Notify Thread" prio=10 tid=0x029796e8 nid=0x58 in Object.wait() [0xda78c000..0xda78fb70]
 at java.lang.Object.wait(Native Method)
 - waiting on <0x4df481b8> (a java.util.HashSet)
 at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.createSocket(BisocketClientInvoker.java:473)
 - locked <0x4df481b8> (a java.util.HashSet)
 at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:801)
 at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:551)
 at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:418)
 at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
 at org.jboss.remoting.Client.invoke(Client.java:1634)
 at org.jboss.remoting.Client.invoke(Client.java:548)
 at org.jboss.remoting.Client.invokeOneway(Client.java:598)
 at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:826)
 at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:697)
 at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1432)
 at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1344)
 - locked <0x4e4b8988> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint)
 at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:322)
 - locked <0x4e4c1d08> (a java.lang.Object)
 at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
 at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:582)
 at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
 at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:606)
 at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:505)
 at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:238)
 - locked <0x4e4ba7f0> (a java.lang.Object)
 at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2203)
 at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:489)
 at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:741)
 at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:383)
 at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)
 at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
 at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
 at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
 at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_7280680627620114891.java)
 at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
 at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
 at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
 at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
 at org.jboss.remoting.Client.invoke(Client.java:1634)
 at org.jboss.remoting.Client.invoke(Client.java:548)
 at org.jboss.remoting.Client.invoke(Client.java:536)
 at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
 at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
 at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:477)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
 at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:632)
 at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
 at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
 at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send_6145266547759487588.java)
 at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
 at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:278)
 at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect39.invoke(ProducerAspect39.java)
 at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
 at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$send_3961598017717988886.java)
 at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
 at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
 at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
 at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
 at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
 at com.cisco.nm.cmp.gjc.cm.UserConnectionImpl.sendMsg(UserConnectionImpl.java:319)
 - locked <0x47864780> (a com.cisco.nm.cmp.gjc.cm.UserConnectionImpl)
 at com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl.send(GWCommandListenerImpl.java:207)
 - locked <0x4e2e5838> (a com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl)
 at com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl.notifications(GWCommandListenerImpl.java:179)
 - locked <0x4e2e5838> (a com.cisco.nm.cmp.gjc.cm.GWCommandListenerImpl)
 at com.cisco.nm.cmp.client.j2ee.connector.DnaCommand$CommandListener.notify(DnaCommand.java:838)
 at com.cisco.nm.cmp.gw.ptpclient.MessageListenerAdapter.onMessage(MessageListenerAdapter.java:49)
 at com.cisco.nm.cmp.gw.ptpclient.protocol.NotifyThread.notifyListeners(NotifyThread.java:193)
 at com.cisco.nm.cmp.gw.ptpclient.protocol.NotifyThread.run(NotifyThread.java:148)

There are many JMS clients connecting to the server and ungraceful disconnections are common, due to poor network, firewall etc.. It seems that the BisocketClientInvoker is waiting on the sockets set forever.

21. Re: BisocketClientInvoker deadlock

clebert.suconic May 14, 2008 4:36 PM (in response to ykonrad)

Did you see my post asking for you to do a test using the SVN version?

22. Re: BisocketClientInvoker deadlock

azafran May 14, 2008 4:42 PM (in response to ykonrad)

I saw it only after I posted my latest reply.

23. Re: BisocketClientInvoker deadlock

ykonrad May 15, 2008 3:51 AM (in response to ykonrad)

Hi Clebert,

After reviewing BisocketClientInvoker code, and the remoting-bisocket-service.xml I believe that we can solve it by changing the pingFrequency to 30000 http://lists.jboss.org/pipermail/jboss-cvs-commits/2007-July/038249.html.
Right now it is 214748364 (~ 1 hour)

Can you please advise?

24. Re: BisocketClientInvoker deadlock

ykonrad May 15, 2008 5:59 AM (in response to ykonrad)

Of course, along with setting the DisableRemotingChecks to true

<attribute name="DisableRemotingChecks">false</attribute>

25. Re: BisocketClientInvoker deadlock

timfox May 15, 2008 6:07 AM (in response to ykonrad)

This may be

http://jira.jboss.org/jira/browse/JBREM-947.
http://jira.jboss.org/jira/browse/JBMESSAGING-1268

26. Re: BisocketClientInvoker deadlock

ykonrad May 15, 2008 7:35 AM (in response to ykonrad)

The topic http://jira.jboss.org/jira/browse/JBMESSAGING-1268 is exactly our case. But setting the timeout to zero is not recommended in the xml comments (as well as setting the ping frequency).

My questions:
1. What will do the job - timeout=3000 or ping frequency=30000?
2. How can I recognize this case from the deployed application side?. When application is sending the message - The IOException from the BisocketClientInvoker (upon ping failure or timeout) is not carried out back to the application (as JMSException).

Thanks ahead.

JBossDeveloper