0 Replies Latest reply on Jan 13, 2016 7:53 AM by veenaonnet

    Threads getting blocked on MicroRemoteClientInvoker.establishLease and MicroRemoteClientInvoker.terminateLease

    veenaonnet Newbie

      Hi Team,

      We are running 2 Jboss Nodes (4.2.2 GA) in a cluster with Jboss Messaging 1.4.4.GA and Jboss remoting ( 2.2.3).

      After running the setup for a month we see the application dragging. Every operation related to JMS queue (either dispatching a message to queue to retrieving a message from JMS queue) is taking considerable time approx. 7/10 mins.

      When we analysed the thread dumps, we found that all the threads are getting into BLOCKED state as either of following place:

       

      Name: Thread-100406008  Id: 103030603

      State: BLOCKED

      Lock name: java.lang.Object@74cbe038

      Lock owner: Thread-100402565    Lock owner id: 103026056

      Total blocked: 1         Total waited: 1

      Stack trace:

      org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:442)

      org.jboss.remoting.Client.setupClientLease(Client.java:1845)

      org.jboss.remoting.Client.connect(Client.java:1732)

      org.jboss.remoting.Client.connect(Client.java:613)

      org.jboss.jms.client.remoting.JMSRemotingConnection$1.run(JMSRemotingConnection.java:362)

      java.security.AccessController.doPrivileged(Native Method)

      org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:356)

      org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165)

      org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)

      org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80)

      org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)

      org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)

      org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)

      org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:134)

      sun.reflect.GeneratedMethodAccessor297.invoke(Unknown Source)

      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

      java.lang.reflect.Method.invoke(Method.java:597)

      org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)

      org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)

      org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)

      org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)

      org.jboss.jms.client.JBossConnectionFactory.createConnection(JBossConnectionFactory.java:87)

      org.jboss.jms.client.JBossConnectionFactory.createConnection(JBossConnectionFactory.java:82)

      com.mypack.common.msg.MessageFactory.handleDispatchMessage(MessageFactory.java:378)

      com.mypack.common.msg.MessageFactory.dispatchMessage(MessageFactory.java:342)

      com.mypack.common.msg.MessageFactory.dispatchMessage(MessageFactory.java:326)

      com.mypack.common.msg.MessageFactory.dispatchMessage(MessageFactory.java:148)

      com.mypack.dam.tasks.FileChangeNotificationTask.dispatchMessage(FileChangeNotificationTask.java:780)

      com.mypack.dam.tasks.FileChangeNotificationTask.performTaskNow(FileChangeNotificationTask.java:581)

      com.mypack.dam.tasks.DeleteFileFromStorageTask$1.run(DeleteFileFromStorageTask.java:402)

      java.lang.Thread.run(Thread.java:662)

       

       

      Name: Thread-100402494  Id: 103025979

      State: BLOCKED

      Lock name: java.lang.Object@74cbe038

      Lock owner: Thread-100402565    Lock owner id: 103026056

      Total blocked: 2         Total waited: 1

      Stack trace:

      org.jboss.remoting.MicroRemoteClientInvoker.terminateLease(MicroRemoteClientInvoker.java:353)

      org.jboss.remoting.MicroRemoteClientInvoker.terminateLease(MicroRemoteClientInvoker.java:348)

      org.jboss.remoting.Client.disconnect(Client.java:633)

      org.jboss.jms.client.remoting.JMSRemotingConnection.stop(JMSRemotingConnection.java:418)

      org.jboss.jms.client.container.ConnectionAspect.handleClose(ConnectionAspect.java:192)

      sun.reflect.GeneratedMethodAccessor300.invoke(Unknown Source)

      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

      java.lang.reflect.Method.invoke(Method.java:597)

      org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)

      org.jboss.jms.client.delegate.ClientConnectionDelegate$close_N4742752445160157748.invokeNext(ClientConnectionDelegate$close_N4742752445160157748.java)

      org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114)

      org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)

      org.jboss.jms.client.delegate.ClientConnectionDelegate$close_N4742752445160157748.invokeNext(ClientConnectionDelegate$close_N4742752445160157748.java)

      org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)

      org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)

      org.jboss.jms.client.delegate.ClientConnectionDelegate$close_N4742752445160157748.invokeNext(ClientConnectionDelegate$close_N4742752445160157748.java)

      org.jboss.jms.client.delegate.ClientConnectionDelegate.close(ClientConnectionDelegate.java)

      org.jboss.jms.client.JBossConnection.close(JBossConnection.java:132)

      com.mypack.common.msg.MessageFactory.handleDispatchMessage(MessageFactory.java:390)

      com.mypack.common.msg.MessageFactory.dispatchMessage(MessageFactory.java:342)

      com.mypack.common.msg.MessageFactory.dispatchMessage(MessageFactory.java:326)

      com.mypack.common.msg.MessageFactory.dispatchMessage(MessageFactory.java:148)

      com.mypack.dam.tasks.FileChangeNotificationTask.dispatchMessage(FileChangeNotificationTask.java:780)

      com.mypack.dam.tasks.FileChangeNotificationTask.performTaskNow(FileChangeNotificationTask.java:581)

      com.mypack.dam.tasks.DeleteFileFromStorageTask$1.run(DeleteFileFromStorageTask.java:402)

      java.lang.Thread.run(Thread.java:662)

       

      The stack trace of Lock owner thread is as below :

       

      Name: Thread-100402565  Id: 103026056

      State: RUNNABLE

      Lock name: null

      Lock owner: null        Lock owner id: -1

      Total blocked: 17        Total waited: 7

      Stack trace:

      java.net.SocketInputStream.socketRead0(Native Method)

      java.net.SocketInputStream.read(SocketInputStream.java:129)

      java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

      java.io.BufferedInputStream.read(BufferedInputStream.java:237)

      java.io.FilterInputStream.read(FilterInputStream.java:66)

      org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:957)

      org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:590)

      org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:422)

      org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141)

      org.jboss.remoting.LeasePinger.sendClientPing(LeasePinger.java:369)

      org.jboss.remoting.LeasePinger.addClient(LeasePinger.java:175)

      org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:448)

      org.jboss.remoting.Client.setupClientLease(Client.java:1845)

      org.jboss.remoting.Client.connect(Client.java:1732)

      org.jboss.remoting.Client.connect(Client.java:613)

      org.jboss.jms.client.remoting.JMSRemotingConnection$1.run(JMSRemotingConnection.java:362)

      java.security.AccessController.doPrivileged(Native Method)

      org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:356)

      org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165)

       

      Can you please suggest what could lead to this condition?

      Please be informed that upgrade to any later version of JBOSS is not an option for us because of some constraints.

      Restarting both the nodes fixes the problem for one month approx.

       

      Thanks,

      Veena