0 Replies Latest reply on Nov 14, 2014 5:26 AM by Rasmus Lund

    On JBoss 5.1 JMS related method calls sometimes hangs for a long time in our JMS clients

    Rasmus Lund Newbie

      On JBoss 5.1 JMS related method calls sometimes hangs for a long time in our JMS clients - e.g.:


      • jndiContext.lookup(topicConnectionFactoryJndiName): takes about 70sec
      • topicConnectionFactory.createTopicConnection(): takes about 5 minutes
      • Furthermore EJBs that try to publish messages on the Topic also hangs/freezes.


      One way of provoking this situation is to let a client connect, and then kill the client. When this has been done about 5 times we start to see the behavior described above.


      More information:

      • Our client programs use durable topic subscriptions
      • About 4 messages per minute is sent to the topic (most not in tx and non-persistent)
      • We often read attributes of mbeans related to JMS
      • No transactions seem to be hanging in the database (Postgres 9.1)
      • While the clients calls are hanging, we see the following via JMX:
        • ConnectionCount=1 on the mbean jboss.jca:name=JmsXA,service=ManagedConnectionPool
        • RunningTransactionCount=1 on the mbean jboss:service=TransactionManager
      • The versions we use:
        • jboss.messaging.jboss-messaging_1.4.8.SP9
        • org.jboss.remoting.jboss-remoting_2.5.4.SP4
      • We see the problem when running standalone and cluster
      • In [remoting-bisocket-service.xml] and in [remoting-sslbisocket-service.xml] we have tried to add the following attribute - which doesn’t seem to make any difference:
        <attribute name="writeTimeout" isParam="true">30000</attribute>


      Just before we see a client hanging in JMS, we see a thread hanging for a long time - I have included its stacktrace below:

      ---

      Thread-58 <--- Frozen for at least 47 sec

      org.jboss.messaging.core.impl.ChannelSupport.deliver() ChannelSupport.java:382

      org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run() ServerSessionEndpoint.java:1717

      org.jboss.messaging.util.OrderedExecutorFactory$ChildExecutor.run() OrderedExecutorFactory.java:111

      java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Runnable) ThreadPoolExecutor.java:895

      java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:918

      java.lang.Thread.run() Thread.java:662

      ---

      Then when we start the client, we see 3+ threads freeze on the server - sometimes more threads than that. Many seems to be waiting for the release of the thread lock on org.jboss.jms.server.connectionmanager.SimpleConnectionManager.

      ---

      Thread-114 <--- Frozen for at least 2m 56 sec

      org.jboss.messaging.core.impl.ChannelSupport.deliver() ChannelSupport.java:382

      org.jboss.jms.server.endpoint.ServerSessionEndpoint$2.run() ServerSessionEndpoint.java:1717

      org.jboss.messaging.util.OrderedExecutorFactory$ChildExecutor.run() OrderedExecutorFactory.java:111

      java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Runnable) ThreadPoolExecutor.java:895

      java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:918

      java.lang.Thread.run() Thread.java:662


      WorkerThread#0[127.0.0.1:37110] <--- Frozen for at least 2m 55 sec

      org.jboss.jms.server.connectionmanager.SimpleConnectionManager.registerConnectionFactoryCallback(String, String, InvokerCallbackHandler) SimpleConnectionManager.java:127

      org.jboss.jms.server.endpoint.ServerConnectionFactoryEndpoint.addCallback(String, String, InvokerCallbackHandler) ServerConnectionFactoryEndpoint.java:319

      org.jboss.jms.server.endpoint.advised.ConnectionFactoryAdvised.addCallback(String, String, ServerInvokerCallbackHandler) ConnectionFactoryAdvised.java:81

      org.jboss.jms.wireformat.ConnectionFactoryAddCallbackRequest.serverInvoke() ConnectionFactoryAddCallbackRequest.java:80

      org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(InvocationRequest) JMSServerInvocationHandler.java:165

      org.jboss.remoting.ServerInvoker.invoke(InvocationRequest) ServerInvoker.java:967

      org.jboss.remoting.transport.socket.ServerThread.completeInvocation(SocketWrapper, InputStream, OutputStream, boolean, int) ServerThread.java:791

      org.jboss.remoting.transport.socket.ServerThread.processInvocation(SocketWrapper, InputStream, OutputStream) ServerThread.java:744

      org.jboss.remoting.transport.socket.ServerThread.dorun() ServerThread.java:586

      org.jboss.remoting.transport.socket.ServerThread.run() ServerThread.java:234


      WorkerThread#2[127.0.0.1:37224] <--- Frozen for at least 32 sec

      org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleClientFailure(String) SimpleConnectionManager.java:201

      org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleConnectionException(Throwable, Client) SimpleConnectionManager.java:241

      org.jboss.remoting.ConnectionNotifier.connectionTerminated(String, String, Map) ConnectionNotifier.java:91

      org.jboss.remoting.Lease.notifyClientTermination(String) Lease.java:295

      org.jboss.remoting.Lease.terminateLease(String) Lease.java:187

      org.jboss.remoting.ServerInvoker.terminateLease(InvocationRequest) ServerInvoker.java:2032

      org.jboss.remoting.ServerInvoker.invoke(InvocationRequest) ServerInvoker.java:924

      org.jboss.remoting.transport.socket.ServerThread.completeInvocation(SocketWrapper, InputStream, OutputStream, boolean, int) ServerThread.java:791

      org.jboss.remoting.transport.socket.ServerThread.processInvocation(SocketWrapper, InputStream, OutputStream) ServerThread.java:744

      org.jboss.remoting.transport.socket.ServerThread.dorun() ServerThread.java:548

      org.jboss.remoting.transport.socket.ServerThread.run() ServerThread.java:234


      WorkerThread#3[127.0.0.1:37120] <--- Frozen for at least 2m 45 sec

      org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleClientFailure(String) SimpleConnectionManager.java:201

      org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleConnectionException(Throwable, Client) SimpleConnectionManager.java:241

      org.jboss.remoting.ConnectionNotifier.connectionTerminated(String, String, Map) ConnectionNotifier.java:91

      org.jboss.remoting.Lease.notifyClientTermination(String) Lease.java:265

      org.jboss.remoting.Lease.terminateLease(String) Lease.java:195

      org.jboss.remoting.ServerInvoker.terminateLease(InvocationRequest) ServerInvoker.java:2023

      org.jboss.remoting.ServerInvoker.invoke(InvocationRequest) ServerInvoker.java:924

      org.jboss.remoting.transport.socket.ServerThread.completeInvocation(SocketWrapper, InputStream, OutputStream, boolean, int) ServerThread.java:791

      org.jboss.remoting.transport.socket.ServerThread.processInvocation(SocketWrapper, InputStream, OutputStream) ServerThread.java:744

      org.jboss.remoting.transport.socket.ServerThread.dorun() ServerThread.java:586

      org.jboss.remoting.transport.socket.ServerThread.run() ServerThread.java:234


      WorkerThread#4[127.0.0.1:37182] <--- Frozen for at least 1m 35 sec

      org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleClientFailure(String) SimpleConnectionManager.java:201

      org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleConnectionException(Throwable, Client) SimpleConnectionManager.java:241

      org.jboss.remoting.ConnectionNotifier.connectionTerminated(String, String, Map) ConnectionNotifier.java:91

      org.jboss.remoting.Lease.notifyClientTermination(String) Lease.java:265

      org.jboss.remoting.Lease.terminateLease(String) Lease.java:195

      org.jboss.remoting.ServerInvoker.terminateLease(InvocationRequest) ServerInvoker.java:2023

      org.jboss.remoting.ServerInvoker.invoke(InvocationRequest) ServerInvoker.java:924

      org.jboss.remoting.transport.socket.ServerThread.completeInvocation(SocketWrapper, InputStream, OutputStream, boolean, int) ServerThread.java:791

      org.jboss.remoting.transport.socket.ServerThread.processInvocation(SocketWrapper, InputStream, OutputStream) ServerThread.java:744

      org.jboss.remoting.transport.socket.ServerThread.dorun() ServerThread.java:586

      org.jboss.remoting.transport.socket.ServerThread.run() ServerThread.java:234


      WorkerThread#5[127.0.0.1:37175] <--- Frozen for at least 1m 45 sec

      org.jboss.jms.server.connectionmanager.SimpleConnectionManager.registerConnection(String, String, ConnectionEndpoint) SimpleConnectionManager.java:104

      org.jboss.jms.server.endpoint.ServerConnectionEndpoint.<init>(ServerPeer, String, String, String, int, int, int, int, int, ServerConnectionFactoryEndpoint, String, String, byte, ServerInvokerCallbackHandler, int) ServerConnectionEndpoint.java:195

      org.jboss.jms.server.endpoint.ServerConnectionFactoryEndpoint.createConnectionDelegateInternal(String, String, int, String, String, byte, ServerInvokerCallbackHandler) ServerConnectionFactoryEndpoint.java:270

      org.jboss.jms.server.endpoint.ServerConnectionFactoryEndpoint.createConnectionDelegate(String, String, int, String, String, byte, ServerInvokerCallbackHandler) ServerConnectionFactoryEndpoint.java:170

      org.jboss.jms.server.endpoint.advised.ConnectionFactoryAdvised.org$jboss$jms$server$endpoint$advised$ConnectionFactoryAdvised$createConnectionDelegate$aop(String, String, int, String, String, byte, ServerInvokerCallbackHandler) ConnectionFactoryAdvised.java:108

      org.jboss.jms.server.endpoint.advised.ConnectionFactoryAdvised.createConnectionDelegate(String, String, int, String, String, byte, ServerInvokerCallbackHandler) ConnectionFactoryAdvised.java

      org.jboss.jms.wireformat.ConnectionFactoryCreateConnectionDelegateRequest.serverInvoke() ConnectionFactoryCreateConnectionDelegateRequest.java:91

      org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(InvocationRequest) JMSServerInvocationHandler.java:165

      org.jboss.remoting.ServerInvoker.invoke(InvocationRequest) ServerInvoker.java:967

      org.jboss.remoting.transport.socket.ServerThread.completeInvocation(SocketWrapper, InputStream, OutputStream, boolean, int) ServerThread.java:791

      org.jboss.remoting.transport.socket.ServerThread.processInvocation(SocketWrapper, InputStream, OutputStream) ServerThread.java:744

      org.jboss.remoting.transport.socket.ServerThread.dorun() ServerThread.java:586

      org.jboss.remoting.transport.socket.ServerThread.run() ServerThread.java:234


      WorkerThread#6[127.0.0.1:37205] <--- Frozen for at least 1m 2 sec

      org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleClientFailure(String) SimpleConnectionManager.java:201

      org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleConnectionException(Throwable, Client) SimpleConnectionManager.java:241

      org.jboss.remoting.ConnectionNotifier.connectionTerminated(String, String, Map) ConnectionNotifier.java:91

      org.jboss.remoting.Lease.notifyClientTermination(String) Lease.java:265

      org.jboss.remoting.Lease.terminateLease(String) Lease.java:195

      org.jboss.remoting.ServerInvoker.terminateLease(InvocationRequest) ServerInvoker.java:2023

      org.jboss.remoting.ServerInvoker.invoke(InvocationRequest) ServerInvoker.java:924

      org.jboss.remoting.transport.socket.ServerThread.completeInvocation(SocketWrapper, InputStream, OutputStream, boolean, int) ServerThread.java:791

      org.jboss.remoting.transport.socket.ServerThread.processInvocation(SocketWrapper, InputStream, OutputStream) ServerThread.java:744

      org.jboss.remoting.transport.socket.ServerThread.dorun() ServerThread.java:586

      org.jboss.remoting.transport.socket.ServerThread.run() ServerThread.java:234


      WorkerThread#7[127.0.0.1:37146] <--- Frozen for at least 2m 15 sec

      org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleClientFailure(String) SimpleConnectionManager.java:201

      org.jboss.jms.server.connectionmanager.SimpleConnectionManager.handleConnectionException(Throwable, Client) SimpleConnectionManager.java:241

      org.jboss.remoting.ConnectionNotifier.connectionTerminated(String, String, Map) ConnectionNotifier.java:91

      org.jboss.remoting.Lease.notifyClientTermination(String) Lease.java:295

      org.jboss.remoting.Lease.terminateLease(String) Lease.java:187

      org.jboss.remoting.ServerInvoker.terminateLease(InvocationRequest) ServerInvoker.java:2032

      org.jboss.remoting.ServerInvoker.invoke(InvocationRequest) ServerInvoker.java:924

      org.jboss.remoting.transport.socket.ServerThread.completeInvocation(SocketWrapper, InputStream, OutputStream, boolean, int) ServerThread.java:791

      org.jboss.remoting.transport.socket.ServerThread.processInvocation(SocketWrapper, InputStream, OutputStream) ServerThread.java:744

      org.jboss.remoting.transport.socket.ServerThread.dorun() ServerThread.java:548

      org.jboss.remoting.transport.socket.ServerThread.run() ServerThread.java:234


      WorkerThread#8[127.0.0.1:37195] <--- Frozen for at least 1m 12 sec

      org.jboss.jms.server.connectionmanager.SimpleConnectionManager.registerConnectionFactoryCallback(String, String, InvokerCallbackHandler) SimpleConnectionManager.java:127

      org.jboss.jms.server.endpoint.ServerConnectionFactoryEndpoint.addCallback(String, String, InvokerCallbackHandler) ServerConnectionFactoryEndpoint.java:319

      org.jboss.jms.server.endpoint.advised.ConnectionFactoryAdvised.addCallback(String, String, ServerInvokerCallbackHandler) ConnectionFactoryAdvised.java:81

      org.jboss.jms.wireformat.ConnectionFactoryAddCallbackRequest.serverInvoke() ConnectionFactoryAddCallbackRequest.java:80

      org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(InvocationRequest) JMSServerInvocationHandler.java:165

      org.jboss.remoting.ServerInvoker.invoke(InvocationRequest) ServerInvoker.java:967

      org.jboss.remoting.transport.socket.ServerThread.completeInvocation(SocketWrapper, InputStream, OutputStream, boolean, int) ServerThread.java:791

      org.jboss.remoting.transport.socket.ServerThread.processInvocation(SocketWrapper, InputStream, OutputStream) ServerThread.java:744

      org.jboss.remoting.transport.socket.ServerThread.dorun() ServerThread.java:586

      org.jboss.remoting.transport.socket.ServerThread.run() ServerThread.java:234