Dead Lock in JBoss Messaging

[dasmurali]

Hi All,

We had two applications(let's call them as A and B)

running in tow different JBoss 5.1.0 AS servers. Both are using default JBoss Messaging(1.4.3.GA) comes with AS.

Client (actually an MDB) running in

A is trying to connect to JMS Queue hosted by Application B (and will send messages). Some times when client trying to create JMS Connection from server B, the thread is not coming back. When verified in server B, there are multiple threads are in BLOCK state.

Please find the below client code which is trying to create connection (this is where my client is not return back).

[code]

QueueConnection connect = factory.createQueueConnection();

[/code]

Please find the below thread dump taken from server B(which is hosted JMS Queue)

Thread: WorkerThread#2[<IP>:40561] : priority:5, demon:false, threadId:291, threadState:BLOCKED

- waiting on <0x1ad437e> (a org.jboss.aop.AspectManager)
org.jboss.jms.server.endpoint.ServerConnectionFactoryEndpoint.createConnectionDelegateInternal(ServerConnectionFactoryEndpoint.java:288)
org.jboss.jms.server.endpoint.ServerConnectionFactoryEndpoint.createConnectionDelegate(ServerConnectionFactoryEndpoint.java:171)
org.jboss.jms.server.endpoint.advised.ConnectionFactoryAdvised.org$jboss$jms$server$endpoint$advised$ConnectionFactoryAdvised$createConnectionDelegate$aop(ConnectionFactoryAdvised.java:108)
org.jboss.jms.server.endpoint.advised.ConnectionFactoryAdvised.createConnectionDelegate(ConnectionFactoryAdvised.java)
org.jboss.jms.wireformat.ConnectionFactoryCreateConnectionDelegateRequest.serverInvoke(ConnectionFactoryCreateConnectionDelegateRequest.java:91)
org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:891)
org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:744)
org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:697)
org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:551)
org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232)

Thread: WorkerThread#3[<IP>:38371] : priority:5, demon:false, threadId:292, threadState:BLOCKED

- waiting on <0x1ad437e> (a org.jboss.aop.AspectManager)
org.jboss.jms.server.endpoint.ServerConnectionFactoryEndpoint.createConnectionDelegateInternal(ServerConnectionFactoryEndpoint.java:288)
org.jboss.jms.server.endpoint.ServerConnectionFactoryEndpoint.createConnectionDelegate(ServerConnectionFactoryEndpoint.java:171)
org.jboss.jms.server.endpoint.advised.ConnectionFactoryAdvised.org$jboss$jms$server$endpoint$advised$ConnectionFactoryAdvised$createConnectionDelegate$aop(ConnectionFactoryAdvised.java:108)
org.jboss.jms.server.endpoint.advised.ConnectionFactoryAdvised.createConnectionDelegate(ConnectionFactoryAdvised.java)
org.jboss.jms.wireformat.ConnectionFactoryCreateConnectionDelegateRequest.serverInvoke(ConnectionFactoryCreateConnectionDelegateRequest.java:91)
org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:891)
org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:744)
org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:697)
org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:551)
org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232)

Thanks and Regards

Murali Reddy

[wdfink]

I don't understand your use case exactly ...

You run both instances in a cluster?

Where the queue(s) hosted

What clients do you have?

But from your thread dump you show only two blocked threads, what is the thread <0x1ad437e> doing? Maybe that will give you a hint.

[dasmurali]

Hi,

Thank  you for your response.

You run both instances in a cluster? --> Both applications are different and not running in cluster. Both running in different JBoss servers on different m/c's.

Where the queue(s) hosted -> Queues are hosted in servers A and B.

What clients do you have?--> My Client is an MessageDrivenBean (deployed in server A listening to Queue hosted in server A) and trying to send message to queue hosted by server B.

But from your thread dump you show only two blocked threads, what is the thread <0x1ad437e> doing? Maybe that will give you a hint.

--> I am not sure about waiting on <0x1ad437e> (a org.jboss.aop.AspectManager).

I verified org.jboss.jms.server.endpoint.ServerConnectionFactoryEndpoint.createConnectionDelegateInternal(ServerConnectionFactoryEndpoint.java:288) code. Please find the below snippet of code that is where the thread is blocking.

// Need to synchronized to prevent a deadlock

// See http://jira.jboss.com/jira/browse/JBMESSAGING-797

synchronized (AspectManager.instance())

{

connAdvised =

new ConnectionAdvised(endpoint); --> BLOCKED here.

}

And please find below the thread dump of client from node A.

Thread: WorkManager(2)-21 : priority:5, demon:true, threadId:5491, threadState:RUNNABLE

java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.FilterInputStream.read(FilterInputStream.java:111)
com.wily.introscope.agent.probe.net.ManagedSocketInputStream.read(ManagedSocketInputStream.java:214)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
java.io.FilterInputStream.read(FilterInputStream.java:66)
org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:1263)
org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:838)
org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:426)
org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:165)
org.jboss.remoting.Client.invoke(Client.java:1724)
org.jboss.remoting.Client.invoke(Client.java:629)
org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:171)
org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeTarget(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:81)
org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect_z_handleCreateConnectionDelegate_361025953.invoke(StateCreationAspect_z_handleCreateConnectionDelegate_361025953.java)
org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101)
org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95)
com.adp.ebs.dasmonitor.ejb.payrollprocessed.PayrollProcessedHandler.sendPublishPayrollEvent(PayrollProcessedHandler.java:585)
com.adp.ebs.dasmonitor.ejb.payrollprocessed.PayrollProcessedHandler.sendGliPayroll(PayrollProcessedHandler.java:511)
com.adp.ebs.dasmonitor.ejb.payrollprocessed.PayrollProcessedHandler.handle(PayrollProcessedHandler.java:157)
com.adp.ebs.dasmonitor.ejb.GenericMDB.onMessage(GenericMDB.java:143)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:585)
org.jboss.invocation.Invocation.performCall(Invocation.java:386)
org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:513)
org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:156)
org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:115)
org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
org.jboss.ejb.plugins.RunAsSecurityInterceptor.process(RunAsSecurityInterceptor.java:133)
org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:103)
org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:415)
org.jboss.ejb.Container.invoke(Container.java:1029)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:585)
org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)
org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)
org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)
org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)
org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)
org.jboss.ejb.plugins.inflow.MessageEndpointInterceptor.delivery(MessageEndpointInterceptor.java:249)
org.jboss.ejb.plugins.inflow.MessageEndpointInterceptor.invoke(MessageEndpointInterceptor.java:128)
org.jboss.proxy.ClientMethodInterceptor.invoke(ClientMethodInterceptor.java:74)
org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:101)
$Proxy236.onMessage(Unknown Source)
org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
org.jboss.jms.client.container.ClientConsumer.callOnMessageStatic(ClientConsumer.java:160)
org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:831)
org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handleRun_361025953.invoke(SessionAspect_z_handleRun_361025953.java)
org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:234)
org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:205)
org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
java.lang.Thread.run(Thread.java:595)

Thanks and Regards

Murali Reddy