Dealing with lost JMS sessions
Hi,
We are facing the following situation in our jboss server:
1. Client are using temporary queues for registrations.
2. Clients are sometimes disconnecting without closing the connections.
As result:
1. We have huge amount of waiting thread in the Thread Group: "JBM-threads" (About 1200)
For example from the jmx-console:
Thread: jbm-server-session-x2-0q6bpggf-1-xfs9pggf-uuwe9e-c5k1k1a : priority:5, demon:true, threadId:75, threadState:WAITING, lockName:java.lang.Object@11ad3b3 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:474) EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122) EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83) java.lang.Thread.run(Thread.java:595)
(We have also jbm-client-session threads)
2. We have many of those messages in the log (I know this is our fault - trying to send messages, but I think it can help illuminating the problem)
May 21 2008 14:05:51.963 IDT org.jboss.remoting.ServerInvoker ERROR Error executing server oneway invocation request: Invocat ionRequest[13d4aff, JMS, org.jboss.jms.wireformat.SessionSendRequest@106d2f6] javax.jms.JMSException: Failed to route Reference[19847052524901780]:NON-RELIABLE to l1-vq5bqggf-1-z7eaqggf-htbpnh-100a at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:757) at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:391) at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop (SessionAdvised.java:87) at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_72806 80627620114891.java) at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157) at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121) at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_72806 80627620114891.java) at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java) at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95) at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809) at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815) at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63) at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758) at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101) at org.jboss.remoting.Client.invoke(Client.java:1634) at org.jboss.remoting.Client.invoke(Client.java:548) at org.jboss.remoting.Client.invokeOneway(Client.java:598) at org.jboss.remoting.Client.invokeOneway(Client.java:786) at org.jboss.remoting.Client.invokeOneway(Client.java:776) at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:180) at org.jboss.jms.client.delegate.DelegateSupport.doInvokeOneway(DelegateSupport.java:165) at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(C lientSessionDelegate.java:473) at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send _6145266547759487588.java) at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:661) at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java) at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send _6145266547759487588.java) at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105) at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send _6145266547759487588.java) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105) at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send _6145266547759487588.java) at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java) at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:269) at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect46.invoke(ProducerAspect46.java) at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$se nd_3961598017717988886.java) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105) at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$se nd_3961598017717988886.java) at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145) at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136) at com.cisco.nm.dmm.ejb.DMMJ2EEQueueNtfsTask.publishMessage(DMMJ2EEQueueNtfsTask.java:134) at com.cisco.nm.dmm.common.services.QueueTask.publishSynchronizedPageGroups(QueueTask.java:141) at com.cisco.nm.dmm.mdb.SynchronizedPageGroupsMDB.onMessage(SynchronizedPageGroupsMDB.java:117) at sun.reflect.GeneratedMethodAccessor523.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112) at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166) at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.j ava:54) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.tx.BMTInterceptor.handleStateless(BMTInterceptor.java:71) at org.jboss.ejb3.tx.BMTInterceptor.invoke(BMTInterceptor.java:131) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62) at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101) at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:245) at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268) at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138) at $Proxy170.onMessage(Unknown Source) at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178) at org.jboss.jms.client.container.ClientConsumer.callOnMessage(ClientConsumer.java:160) at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:831) at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java) at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_ N8003352271541955702.java) at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170) at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105) at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_ N8003352271541955702.java) at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java) at org.jboss.jms.client.JBossSession.run(JBossSession.java:199) at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237) at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204) at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743) at java.lang.Thread.run(Thread.java:595)
My questions:
1. What are all those threads? Why we have so many?
2. Can we use any kind of purging mechanism for lost sessions thus reducing thread numbers?
We are using jboss AS 4.2.0.GA with:
JBM 1.4.0.SP3.CP02
JBR 2.2.2.SP7
