4 Replies Latest reply on May 27, 2010 6:28 AM by cullendw

    Possible Deadlock Jboss 4.2.3

    cullendw

      Hi,

       

      We have been running jboss 4.2.3 for a while now. Recently we have been getting deadlocks and we are battling to narrow down the exact cause. This happens randomly and can take hours to occur. When doing a stack trace, we find the following culprits :

       

       

      Thread: WorkManager(2)-38 : priority:5, demon:true, threadId:479, threadState:BLOCKED, lockName:java.lang.Class@741b31f2

      java.lang.Class.initAnnotationsIfNecessary(Class.java:3067)
      java.lang.Class.getAnnotation(Class.java:3029)
      org.jboss.aop.annotation.AnnotationElement.getVisibleAnnotation(AnnotationElement.java:88)
      org.jboss.aop.Advisor.resolveAnnotation(Advisor.java:321)
      org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:74)
      org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
      org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
      org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
      org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
      org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
      org.jboss.ejb3.service.ServiceContainer.localInvoke(ServiceContainer.java:329)
      org.jboss.ejb3.service.ServiceContainer.localInvoke(ServiceContainer.java:294)
      org.jboss.ejb3.service.ServiceMBeanDelegate.invoke(ServiceMBeanDelegate.java:215)
      org.jboss.mx.server.RawDynamicInvoker.invoke(RawDynamicInvoker.java:164)
      org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
      $Proxy101.get(Unknown Source)

       

      Thread: AsyncNotifier-401 : priority:5, demon:false, threadId:812, threadState:BLOCKED, lockName:java.lang.Class@741b31f2

      java.lang.Class.initAnnotationsIfNecessary(Class.java:3067)
      java.lang.Class.getAnnotation(Class.java:3029)
      org.jboss.aop.annotation.AnnotationElement.getVisibleAnnotation(AnnotationElement.java:88)
      org.jboss.aop.Advisor.resolveAnnotation(Advisor.java:321)
      org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:74)
      org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
      org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
      org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
      org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
      org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
      org.jboss.ejb3.service.ServiceContainer.localInvoke(ServiceContainer.java:329)
      org.jboss.ejb3.service.ServiceContainer.localInvoke(ServiceContainer.java:294)
      org.jboss.ejb3.service.ServiceMBeanDelegate.invoke(ServiceMBeanDelegate.java:215)
      org.jboss.mx.server.RawDynamicInvoker.invoke(RawDynamicInvoker.java:164)
      org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
      $Proxy101.get(Unknown Source)

       

       

      The method  java.lang.Class.initAnnotationsIfNecessary is synchronized and it seems that both threads are hanging on the same class  java.lang.Class@741b31f2.

       

      Any ideas why this would start happening all of a sudden? We upgraded to the latest java 6 patch recently and I am curious if any changes were made to the Class class ....

        • 1. Re: Possible Deadlock Jboss 4.2.3
          cullendw

          I forgot to mention, we recently upgraded from Spring 2.5 to Spring 3 (which we have deployed using the spring deployer)

          • 2. Re: Possible Deadlock Jboss 4.2.3
            jaikiran

            Could you try this with JBoss AS 5.1.0 with the latest EJB3 plugin? There have been numerous changes since AS 4.2.3 in JBoss EJB3 code.

            1 of 1 people found this helpful
            • 3. Re: Possible Deadlock Jboss 4.2.3
              cullendw

              Hi Jaikiran,

               

              Thanks for the response. I take your point that we are running on an old implementation of ejb3. However, we have been for some time without any issues, so we found it strange that we are all of a sudden getting deadlocks in our application. We attempted to move over to jboss 5.1 some time ago, but we encountered some issues which forced us to move back to 4.2.3.

               

              We will look at the change logs to see if there are any relevant bug fixes/upgrades that pertain to this issue.

              • 4. Issue was caused by a bottleneck
                cullendw

                It looks like we did not have a deadlock at all, but rather a bottleneck. Using JBoss Profiler  (http://www.jboss.org/jbossprofiler) we determined that a bottleneck caused a backlog and since we use Semaphores to enable FIFO processing of messages, the locked threads built up causing the appearance of a deadlock.