1 Reply Latest reply on Nov 9, 2013 9:38 PM by Ron Sigal

    Big readwrite lock in the jboss messaging service [EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire]

    Raghavendra Venkatakrishnan Newbie

      Hi All,

       

      Greetings !!!

       

      Problem :

      ---------------------

      Things are getting stuck stuck on the JBM side. There is a big readwrite lock in the jboss messaging server. All message delivery and sending involves getting the read lock [EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire]. Adding a new connection needs the write lock.

      I see in the jstacks (an extract from the jstack as shown below) a massive pile up of server side threads that need the read lock

       

       

      The locked resource 0xfffffffd3cdaa0d0 is not in RUNNABLE state in any of the threads.Several threads are hanging in waiting state in order to acquire the reader or writer lock eventhough there is no other thread holding on the lock.

       

       

      Jstack_info:

      ------------------------

      "XXXX" prio=3 tid=0x000000015b0eb800 nid=0x733 in Object.wait() [0xfffffffc5a539000]

      java.lang.Thread.State: WAITING (on object monitor)

      at java.lang.Object.wait(Native Method)

      at java.lang.Object.wait(Object.java:485)

      at EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock.acquire(WriterPreferenceReadWriteLock.java:163)

      - locked <0xfffffffd3cdaa0d0> (a EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock$ReaderLock)

      at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2302)

      at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:520)

      at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:777)

      at org.jboss.jms.server.endpoint.ServerSessionEndpoint.send(ServerSessionEndpoint.java:399)

      at org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$send$aop(SessionAdvised.java:87)

      at org.jboss.jms.server.endpoint.advised.SessionAdvised$send_7280680627620114891.invokeTarget(SessionAdvised$send_7280680627620114891.java)

      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)

      at org.jboss.jms.server.container.SecurityAspect.handleSend(SecurityAspect.java:157)

      at sun.reflect.GeneratedMethodAccessor368.invoke(Unknown Source)

      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

      at java.lang.reflect.Method.invoke(Method.java:597)

      at org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:122)

      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

      at org.jboss.jms.server.endpoint.advised.SessionAdvised.send(SessionAdvised.java)

      at org.jboss.jms.wireformat.SessionSendRequest.serverInvoke(SessionSendRequest.java:95)

      at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:157)

      at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:967)

      at org.jboss.remoting.ServerInvoker$4.run(ServerInvoker.java:2171)

      at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:147)

      at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1746)

      at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)

      at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)

      at org.jboss.util.threadpool.BasicThreadPool.execute(BasicThreadPool.java:487)

      at org.jboss.util.threadpool.BasicThreadPool.runTaskWrapper(BasicThreadPool.java:206)

      at org.jboss.util.threadpool.BasicThreadPool.run(BasicThreadPool.java:226)

      at org.jboss.util.threadpool.BasicThreadPool.run(BasicThreadPool.java:220)

      at org.jboss.remoting.ServerInvoker.handleOnewayInvocation(ServerInvoker.java:2182)

      at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:937)

      at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:106)

      at org.jboss.remoting.Client.invoke(Client.java:2077)

      at org.jboss.remoting.Client.invoke(Client.java:880)

      at org.jboss.remoting.Client$9.run(Client.java:934)

      at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:147)

      at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1746)

      at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)

      at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)

      at org.jboss.util.threadpool.BasicThreadPool.execute(BasicThreadPool.java:487)

      at org.jboss.util.threadpool.BasicThreadPool.runTaskWrapper(BasicThreadPool.java:206)

      at org.jboss.util.threadpool.BasicThreadPool.run(BasicThreadPool.java:226)

      at org.jboss.util.threadpool.BasicThreadPool.run(BasicThreadPool.java:220)

      at org.jboss.remoting.Client.invokeOneway(Client.java:944)

      at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:186)

      at org.jboss.jms.client.delegate.DelegateSupport.doInvokeOneway(DelegateSupport.java:166)

      at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$send$aop(ClientSessionDelegate.java:495)

      at org.jboss.jms.client.delegate.ClientSessionDelegate$send_6145266547759487588.invokeTarget(ClientSessionDelegate$send_6145266547759487588.java)

      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)

      at org.jboss.jms.client.container.SessionAspect.handleSend(SessionAspect.java:661)

      at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handleSend_500571876.invoke(SessionAspect_z_handleSend_500571876.java)

      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

      at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)

      at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)

      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

      at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)

      at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)

      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

      at org.jboss.jms.client.delegate.ClientSessionDelegate.send(ClientSessionDelegate.java)

      at org.jboss.jms.client.container.ProducerAspect.handleSend(ProducerAspect.java:276)

      at org.jboss.aop.advice.org.jboss.jms.client.container.ProducerAspect_z_handleSend_500571876.invoke(ProducerAspect_z_handleSend_500571876.java)

      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

      at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)

      at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)

      at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

      at org.jboss.jms.client.delegate.ClientProducerDelegate.send(ClientProducerDelegate.java)

      at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:165)

      at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:208)

      at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:146)

      at org.jboss.jms.client.JBossMessageProducer.send(JBossMessageProducer.java:137)

      at org.jboss.resource.adapter.jms.JmsMessageProducer.send(JmsMessageProducer.java:142)

       

       

      Related java bug:

      -------------------------------------

      Is this related to the below java bug:

      http://bugs.sun.com/view_bug.do?bug_id=6822370

       

       

      The restart of jboss server even with java VM option turned on : -XX:+UseMembar

      also didnt solve the above issue though.

       

       

      Environment and Version details:

      -------------------------------------------------------

      java -version:

      "1.6.0_22"

      Java(TM) SE Runtime Environment (build 1.6.0_22-b04)

      Java HotSpot(TM) Server VM (build 17.1-b03, mixed mode)

       

      JBM:

      jboss-messaging-1.4.6.GA

       

      JBR:

      jboss-remoting-2.5.4.SP1.jar

       

      OS_Arch:

      SunOS sys-46 5.10 Generic_147440-23 sun4v sparc SUNW,SPARC-Enterprise-T5220

       

       

      Help me out.

       

       

      Thanks,

      Raghav