4 Replies Latest reply on Jun 26, 2008 12:13 PM by Tim Fox

    Dealing with lost JMS sessions

    yuval konrad Newbie

      Hi,

      We are facing the following situation in our jboss server:

      1. Client are using temporary queues for registrations.
      2. Clients are sometimes disconnecting without closing the connections.

      As result:

      1. We have huge amount of waiting thread in the Thread Group: "JBM-threads" (About 1200)
      For example from the jmx-console:

      Thread: jbm-server-session-x2-0q6bpggf-1-xfs9pggf-uuwe9e-c5k1k1a : priority:5, demon:true, threadId:75, threadState:WAITING,
      lockName:java.lang.Object@11ad3b3
       java.lang.Object.wait(Native Method)
       java.lang.Object.wait(Object.java:474)
       EDU.oswego.cs.dl.util.concurrent.LinkedQueue.take(LinkedQueue.java:122)
       EDU.oswego.cs.dl.util.concurrent.QueuedExecutor$RunLoop.run(QueuedExecutor.java:83)
       java.lang.Thread.run(Thread.java:595)
      


      (We have also jbm-client-session threads)

      2. We have many of those messages in the log (I know this is our fault - trying to send messages, but I think it can help illuminating the problem)

      May 21 2008 14:05:51.963 IDT org.jboss.remoting.ServerInvoker ERROR Error executing server oneway invocation request: Invocat
      ionRequest[13d4aff, JMS, org.jboss.jms.wireformat.SessionSendRequest@106d2f6]
      javax.jms.JMSException: Failed to route Reference[19847052524901780]:NON-RELIABLE to l1-vq5bqggf-1-z7eaqggf-htbpnh-100a
       at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:757)
       at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:391)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop
      (SessionAdvised.java:87)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_72806
      80627620114891.java)
       at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)
       at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeNext(SessionAdvised$send_72806
      80627620114891.java)
       at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)
       at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)
       at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
       at org.jboss.remoting.ServerInvoker$1.run(ServerInvoker.java:1815)
       at org.jboss.jms.server.remoting.DirectThreadPool.run(DirectThreadPool.java:63)
       at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:1826)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:758)
       at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:101)
       at org.jboss.remoting.Client.invoke(Client.java:1634)
       at org.jboss.remoting.Client.invoke(Client.java:548)
       at org.jboss.remoting.Client.invokeOneway(Client.java:598)
       at org.jboss.remoting.Client.invokeOneway(Client.java:786)
       at org.jboss.remoting.Client.invokeOneway(Client.java:776)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:180)
       at org.jboss.jms.client.delegate.DelegateSupport.doInvokeOneway(DelegateSupport.java:165)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(C
      lientSessionDelegate.java:473)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send
      _6145266547759487588.java)
       at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:661)
       at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect28.invoke(SessionAspect28.java)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send
      _6145266547759487588.java)
       at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send
      _6145266547759487588.java)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeNext(ClientSessionDelegate$send
      _6145266547759487588.java)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)
       at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:269)
       at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect46.invoke(ProducerAspect46.java)
       at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$se
      nd_3961598017717988886.java)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
       at org.jboss.jms.client.delegate.ClientProducerDelegate$send_3961598017717988886.invokeNext(ClientProducerDelegate$se
      nd_3961598017717988886.java)
       at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)
       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:164)
       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:207)
       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:145)
       at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:136)
       at com.cisco.nm.dmm.ejb.DMMJ2EEQueueNtfsTask.publishMessage(DMMJ2EEQueueNtfsTask.java:134)
       at com.cisco.nm.dmm.common.services.QueueTask.publishSynchronizedPageGroups(QueueTask.java:141)
       at com.cisco.nm.dmm.mdb.SynchronizedPageGroupsMDB.onMessage(SynchronizedPageGroupsMDB.java:117)
       at sun.reflect.GeneratedMethodAccessor523.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
       at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
       at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.j
      ava:54)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.ejb3.tx.BMTInterceptor.handleStateless(BMTInterceptor.java:71)
       at org.jboss.ejb3.tx.BMTInterceptor.invoke(BMTInterceptor.java:131)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:245)
       at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:268)
       at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:138)
       at $Proxy170.onMessage(Unknown Source)
       at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:178)
       at org.jboss.jms.client.container.ClientConsumer.callOnMessage(ClientConsumer.java:160)
       at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:831)
       at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_
      N8003352271541955702.java)
       at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
       at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
       at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_
      N8003352271541955702.java)
       at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
       at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
       at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:237)
       at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:204)
       at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
       at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
       at java.lang.Thread.run(Thread.java:595)
      


      My questions:
      1. What are all those threads? Why we have so many?
      2. Can we use any kind of purging mechanism for lost sessions thus reducing thread numbers?

      We are using jboss AS 4.2.0.GA with:
      JBM 1.4.0.SP3.CP02
      JBR 2.2.2.SP7