0 Replies Latest reply on Jul 25, 2011 7:07 AM by Prabakaran Periasamy

    JBoss Messaging stops working after few hours

    Prabakaran Periasamy Newbie

      Hello All,

       

      We are using Jboss Messaging 1.4.5 ,Jboss-remoting 2.2.3 SP3 with JBoss 4.2.2 .

      We are having two nodes in cluster both are in different physical servers.

      Our application is having around 8 different cluster queue and some of the meesages are produced by node 1 and some are from node 2.

      The messaging is working fine with normal load,but if we increase the load the system stoped working after 24 hours or 30 hours.

       

      Just to isolate the issue ,we have removed the queue clustering and found system is working with out any issue.

      We have tried with jboss-remoting -2.2.4 and faced the same issue.

       

      Attaching the remoting-bisocket-service.xml

       

      Can some one please help me to solve this?

       

      Regards,

      Prabakar.

       

      Some part of Thread Dump :

       

       

          *Thread: RMI RenewClean-[10.21.11.44:1201]* : priority:5,

          demon:true, threadId:446, threadState:TIMED_WAITING,

          lockName:java.lang.ref.ReferenceQueue$Lock@5351f3bf

       

              java.lang.Object.wait(Native Method)

              java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)

              sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:501)

              java.lang.Thread.run(Thread.java:595)

       

      *Thread: http-10.21.11.33-8080-4* : priority:5, demon:true,

                  threadId:281, threadState:RUNNABLE, lockName:null

       

                      org.jboss.remoting.MicroRemoteClientInvoker.terminateLease(MicroRemoteClientInvoker.java:367)

                      org.jboss.remoting.MicroRemoteClientInvoker.terminateLease(MicroRemoteClientInvoker.java:360)

                      org.jboss.remoting.Client.disconnect(Client.java:675)

                      org.jboss.jms.client.remoting.JMSRemotingConnection.stop(JMSRemotingConnection.java:430)

                      org.jboss.jms.client.container.ConnectionAspect.handleClose(ConnectionAspect.java:192)

                      sun.reflect.GeneratedMethodAccessor657.invoke(Unknown

                      Source)

                      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                      java.lang.reflect.Method.invoke(Method.java:585)

                      org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)

                      org.jboss.jms.client.delegate.ClientConnectionDelegate$close_N4742752445160157748.invokeNext(ClientConnectionDelegate$close_N4742752445160157748.java)

                      org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114)

                      org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)

                      org.jboss.jms.client.delegate.ClientConnectionDelegate$close_N4742752445160157748.invokeNext(ClientConnectionDelegate$close_N4742752445160157748.java)

                      org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)

                      org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)

                      org.jboss.jms.client.delegate.ClientConnectionDelegate$close_N4742752445160157748.invokeNext(ClientConnectionDelegate$close_N4742752445160157748.java)

                      org.jboss.jms.client.delegate.ClientConnectionDelegate.close(ClientConnectionDelegate.java)

                      org.jboss.jms.client.JBossConnection.close(JBossConnection.java:132)

       

      *Thread: http-10.21.11.33-8080-6* : priority:5, demon:true,

                  threadId:431, threadState:BLOCKED,

                  lockName:java.lang.Class@54bceb91

       

                      java.beans.PropertyEditorManager.findEditor(PropertyEditorManager.java:75)

                      org.jboss.util.propertyeditor.PropertyEditors.mapJavaBeanProperties(PropertyEditors.java:365)

                      org.jboss.remoting.transport.socket.SocketServerInvoker.setup(SocketServerInvoker.java:157)

                      org.jboss.remoting.transport.bisocket.BisocketServerInvoker.setup(BisocketServerInvoker.java:517)

                      org.jboss.remoting.ServerInvoker.create(ServerInvoker.java:904)

                      org.jboss.remoting.transport.Connector.init(Connector.java:412)

                      org.jboss.remoting.transport.Connector.create(Connector.java:829)

                      org.jboss.remoting.transport.Connector.start(Connector.java:304)

                      org.jboss.remoting.Client.addListener(Client.java:1158)

                      org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:259)

                      org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:400)

                      org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165)

                      org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)

                      org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80)

                      org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)

                      org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)

                      org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)

                      org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:134)

                      sun.reflect.GeneratedMethodAccessor652.invoke(Unknown

                      Source)

                      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                      java.lang.reflect.Method.invoke(Method.java:585)

                      org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)

                      org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)

                      org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)

                      org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)

                      org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101)

                      org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95)

       

       

       

      *Thread: http-10.21.11.33-8080-14* : priority:5, demon:true,

                  threadId:441, threadState:BLOCKED,

                  lockName:java.lang.Object@31e50754

       

                      org.jboss.remoting.InvokerRegistry.destroyClientInvoker(InvokerRegistry.java:229)

                      org.jboss.remoting.Client.disconnect(Client.java:706)

                      org.jboss.remoting.Client.addCallbackListener(Client.java:2001)

                      org.jboss.remoting.Client.addListener(Client.java:1161)

                      org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:259)

                      org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:400)

                      org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165)

                      org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)

                      org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80)

                      org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)

                      org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)

                      org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)

                      org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:134)

                      sun.reflect.GeneratedMethodAccessor652.invoke(Unknown

                      Source)

                      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                      java.lang.reflect.Method.invoke(Method.java:585)

                      org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)

                      org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)

                      org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)

                      org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)

                      org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101)

                      org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95)