JBoss Messaging stops working after few hours
Hello All,
We are using Jboss Messaging 1.4.5 ,Jboss-remoting 2.2.3 SP3 with JBoss 4.2.2 .
We are having two nodes in cluster both are in different physical servers.
Our application is having around 8 different cluster queue and some of the meesages are produced by node 1 and some are from node 2.
The messaging is working fine with normal load,but if we increase the load the system stoped working after 24 hours or 30 hours.
Just to isolate the issue ,we have removed the queue clustering and found system is working with out any issue.
We have tried with jboss-remoting -2.2.4 and faced the same issue.
Attaching the remoting-bisocket-service.xml
Can some one please help me to solve this?
Regards,
Prabakar.
Some part of Thread Dump :
*Thread: RMI RenewClean-[10.21.11.44:1201]* : priority:5,
demon:true, threadId:446, threadState:TIMED_WAITING,
lockName:java.lang.ref.ReferenceQueue$Lock@5351f3bf
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:501)
java.lang.Thread.run(Thread.java:595)
*Thread: http-10.21.11.33-8080-4* : priority:5, demon:true,
threadId:281, threadState:RUNNABLE, lockName:null
org.jboss.remoting.MicroRemoteClientInvoker.terminateLease(MicroRemoteClientInvoker.java:367)
org.jboss.remoting.MicroRemoteClientInvoker.terminateLease(MicroRemoteClientInvoker.java:360)
org.jboss.remoting.Client.disconnect(Client.java:675)
org.jboss.jms.client.remoting.JMSRemotingConnection.stop(JMSRemotingConnection.java:430)
org.jboss.jms.client.container.ConnectionAspect.handleClose(ConnectionAspect.java:192)
sun.reflect.GeneratedMethodAccessor657.invoke(Unknown
Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:585)
org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
org.jboss.jms.client.delegate.ClientConnectionDelegate$close_N4742752445160157748.invokeNext(ClientConnectionDelegate$close_N4742752445160157748.java)
org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:114)
org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
org.jboss.jms.client.delegate.ClientConnectionDelegate$close_N4742752445160157748.invokeNext(ClientConnectionDelegate$close_N4742752445160157748.java)
org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
org.jboss.jms.client.delegate.ClientConnectionDelegate$close_N4742752445160157748.invokeNext(ClientConnectionDelegate$close_N4742752445160157748.java)
org.jboss.jms.client.delegate.ClientConnectionDelegate.close(ClientConnectionDelegate.java)
org.jboss.jms.client.JBossConnection.close(JBossConnection.java:132)
*Thread: http-10.21.11.33-8080-6* : priority:5, demon:true,
threadId:431, threadState:BLOCKED,
lockName:java.lang.Class@54bceb91
java.beans.PropertyEditorManager.findEditor(PropertyEditorManager.java:75)
org.jboss.util.propertyeditor.PropertyEditors.mapJavaBeanProperties(PropertyEditors.java:365)
org.jboss.remoting.transport.socket.SocketServerInvoker.setup(SocketServerInvoker.java:157)
org.jboss.remoting.transport.bisocket.BisocketServerInvoker.setup(BisocketServerInvoker.java:517)
org.jboss.remoting.ServerInvoker.create(ServerInvoker.java:904)
org.jboss.remoting.transport.Connector.init(Connector.java:412)
org.jboss.remoting.transport.Connector.create(Connector.java:829)
org.jboss.remoting.transport.Connector.start(Connector.java:304)
org.jboss.remoting.Client.addListener(Client.java:1158)
org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:259)
org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:400)
org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165)
org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80)
org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:134)
sun.reflect.GeneratedMethodAccessor652.invoke(Unknown
Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:585)
org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101)
org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95)
*Thread: http-10.21.11.33-8080-14* : priority:5, demon:true,
threadId:441, threadState:BLOCKED,
lockName:java.lang.Object@31e50754
org.jboss.remoting.InvokerRegistry.destroyClientInvoker(InvokerRegistry.java:229)
org.jboss.remoting.Client.disconnect(Client.java:706)
org.jboss.remoting.Client.addCallbackListener(Client.java:2001)
org.jboss.remoting.Client.addListener(Client.java:1161)
org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:259)
org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:400)
org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165)
org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80)
org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:134)
sun.reflect.GeneratedMethodAccessor652.invoke(Unknown
Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:585)
org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101)
org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95)