0 Replies Latest reply on Nov 14, 2014 5:26 AM by rasmuslund

On JBoss 5.1 JMS related method calls sometimes hangs for a long time in our JMS clients

rasmuslund Nov 14, 2014 5:26 AM

On JBoss 5.1 JMS related method calls sometimes hangs for a long time in our JMS clients - e.g.:

jndiContext.lookup(topicConnectionFactoryJndiName): takes about 70sec
topicConnectionFactory.createTopicConnection(): takes about 5 minutes
Furthermore EJBs that try to publish messages on the Topic also hangs/freezes.

One way of provoking this situation is to let a client connect, and then kill the client. When this has been done about 5 times we start to see the behavior described above.

More information:

Our client programs use durable topic subscriptions
About 4 messages per minute is sent to the topic (most not in tx and non-persistent)
We often read attributes of mbeans related to JMS
No transactions seem to be hanging in the database (Postgres 9.1)
While the clients calls are hanging, we see the following via JMX:
- ConnectionCount=1 on the mbean jboss.jca:name=JmsXA,service=ManagedConnectionPool
- RunningTransactionCount=1 on the mbean jboss:service=TransactionManager
The versions we use:
- jboss.messaging.jboss-messaging_1.4.8.SP9
- org.jboss.remoting.jboss-remoting_2.5.4.SP4
We see the problem when running standalone and cluster
In [remoting-bisocket-service.xml] and in [remoting-sslbisocket-service.xml] we have tried to add the following attribute - which doesn’t seem to make any difference:
<attribute name="writeTimeout" isParam="true">30000</attribute>

Just before we see a client hanging in JMS, we see a thread hanging for a long time - I have included its stacktrace below:

---

Thread-58 <--- Frozen for at least 47 sec

org.jboss.messaging.core.impl.ChannelSupport.deliver() ChannelSupport.java:382

org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run() ServerSessionEndpoint.java:1717

org.jboss.messaging.util.OrderedExecutorFactory$ChildExecutor.run() OrderedExecutorFactory.java:111

java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Runnable) ThreadPoolExecutor.java:895

java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:918

java.lang.Thread.run() Thread.java:662

---

Then when we start the client, we see 3+ threads freeze on the server - sometimes more threads than that. Many seems to be waiting for the release of the thread lock on org.jboss.jms.server.connectionmanager.SimpleConnectionManager.

---

Thread-114 <--- Frozen for at least 2m 56 sec

org.jboss.messaging.core.impl.ChannelSupport.deliver() ChannelSupport.java:382

org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run() ServerSessionEndpoint.java:1717

org.jboss.messaging.util.OrderedExecutorFactory$ChildExecutor.run() OrderedExecutorFactory.java:111

java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Runnable) ThreadPoolExecutor.java:895

java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:918

java.lang.Thread.run() Thread.java:662

WorkerThread#0[127.0.0.1:37110] <--- Frozen for at least 2m 55 sec

org.jboss.jms.server.connectionmanager.SimpleConnectionManager.registerConnectionFactoryCallback(String, String, InvokerCallbackHandler) SimpleConnectionManager.java:127

org.jboss.jms.server.endpoint.ServerConnectionFactoryEndpoint.addCallback(String, String, InvokerCallbackHandler) ServerConnectionFactoryEndpoint.java:319

org.jboss.jms.server.endpoint.advised.ConnectionFactoryAdvised.addCallback(String, String, ServerInvokerCallbackHandler) ConnectionFactoryAdvised.java:81

org.jboss.jms.wireformat.ConnectionFactoryAddCallbackRequest.serverInvoke() ConnectionFactoryAddCallbackRequest.java:80

org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(InvocationRequest) JMSServerInvocationHandler.java:165

org.jboss.remoting.ServerInvoker.invoke(InvocationRequest) ServerInvoker.java:967

org.jboss.remoting.transport.socket.ServerThread.completeInvocation(SocketWrapper, InputStream, OutputStream, boolean, int) ServerThread.java:791

org.jboss.remoting.transport.socket.ServerThread.processInvocation(SocketWrapper, InputStream, OutputStream) ServerThread.java:744

org.jboss.remoting.transport.socket.ServerThread.dorun() ServerThread.java:586

org.jboss.remoting.transport.socket.ServerThread.run() ServerThread.java:234

WorkerThread#2[127.0.0.1:37224] <--- Frozen for at least 32 sec

org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleClientFailure(String) SimpleConnectionManager.java:201

org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleConnectionException(Throwable, Client) SimpleConnectionManager.java:241

org.jboss.remoting.ConnectionNotifier.connectionTerminated(String, String, Map) ConnectionNotifier.java:91

org.jboss.remoting.Lease.notifyClientTermination(String) Lease.java:295

org.jboss.remoting.Lease.terminateLease(String) Lease.java:187

org.jboss.remoting.ServerInvoker.terminateLease(InvocationRequest) ServerInvoker.java:2032

org.jboss.remoting.ServerInvoker.invoke(InvocationRequest) ServerInvoker.java:924

org.jboss.remoting.transport.socket.ServerThread.completeInvocation(SocketWrapper, InputStream, OutputStream, boolean, int) ServerThread.java:791

org.jboss.remoting.transport.socket.ServerThread.processInvocation(SocketWrapper, InputStream, OutputStream) ServerThread.java:744

org.jboss.remoting.transport.socket.ServerThread.dorun() ServerThread.java:548

org.jboss.remoting.transport.socket.ServerThread.run() ServerThread.java:234

WorkerThread#3[127.0.0.1:37120] <--- Frozen for at least 2m 45 sec

org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleClientFailure(String) SimpleConnectionManager.java:201

org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleConnectionException(Throwable, Client) SimpleConnectionManager.java:241

org.jboss.remoting.ConnectionNotifier.connectionTerminated(String, String, Map) ConnectionNotifier.java:91

org.jboss.remoting.Lease.notifyClientTermination(String) Lease.java:265

org.jboss.remoting.Lease.terminateLease(String) Lease.java:195

org.jboss.remoting.ServerInvoker.terminateLease(InvocationRequest) ServerInvoker.java:2023

org.jboss.remoting.ServerInvoker.invoke(InvocationRequest) ServerInvoker.java:924

org.jboss.remoting.transport.socket.ServerThread.completeInvocation(SocketWrapper, InputStream, OutputStream, boolean, int) ServerThread.java:791

org.jboss.remoting.transport.socket.ServerThread.processInvocation(SocketWrapper, InputStream, OutputStream) ServerThread.java:744

org.jboss.remoting.transport.socket.ServerThread.dorun() ServerThread.java:586

org.jboss.remoting.transport.socket.ServerThread.run() ServerThread.java:234

WorkerThread#4[127.0.0.1:37182] <--- Frozen for at least 1m 35 sec