1 Reply Latest reply on Aug 7, 2013 3:21 AM by Mikal Henriksen

    ClassLoader bottleneck in 5.1.0.GA

    Mikal Henriksen Newbie

      So it looks like our app is affected by JBCL-185 and/or JBCL-100. We were running visualvm during load testing, and discovered that EJB proxy instances had a very high self-time. This is a SLSB with a @Local interface, so it should be pretty fast.

       

      We did a thread dump, and it turns out most of the servlet threads were blocked waiting for a class loader. Some examples:

       

      Several are like this:

      Thread: ajp-0.0.0.0-8009-11 : priority:5, demon:true, threadId:198, threadState:BLOCKED

      - waiting on <0x22d22fe3> (a org.jboss.web.tomcat.service.WebCtxLoader$ENCLoader)

      java.lang.ClassLoader.loadClass(ClassLoader.java:291)

      java.lang.ClassLoader.loadClass(ClassLoader.java:295)

      - locked <0x2c25bf62> (a org.jboss.ejb3.common.classloader.PrimitiveAwareClassLoader)

      java.lang.ClassLoader.loadClass(ClassLoader.java:247)

      org.jboss.ejb3.common.lang.SerializableMethod.getClassFromName(SerializableMethod.java:307)

      org.jboss.ejb3.common.lang.SerializableMethod.getClassType(SerializableMethod.java:282)

      org.jboss.ejb3.common.lang.SerializableMethod.toMethod(SerializableMethod.java:233)

      org.jboss.ejb3.common.lang.SerializableMethod.toMethod(SerializableMethod.java:220)

      org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.handleInvocationDirectly(SessionProxyInvocationHandlerBase.java:240)

      org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:182)

      org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:164)

      $Proxy815.getAdaptedItem(Unknown Source)

      [servlet method here]

       

      The thread owning 0x22d22fe3:

       

      Thread: ajp-0.0.0.0-8009-18 : priority:5, demon:true, threadId:205, threadState:BLOCKED

      - waiting on <0x1b4ddacc> (a org.jboss.classloader.spi.base.BaseClassLoader)

      java.lang.Class.forName0(Native Method)

      java.lang.Class.forName(Class.java:247)

      org.jboss.classloader.plugins.loader.ClassLoaderToLoaderAdapter.loadClass(ClassLoaderToLoaderAdapter.java:172)

      org.jboss.classloader.spi.ClassLoaderDomain.loadClassFromParent(ClassLoaderDomain.java:352)

      org.jboss.classloader.spi.ClassLoaderDomain.loadClassAfter(ClassLoaderDomain.java:323)

      org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:275)

      org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1119)

      org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:798)

      org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:441)

      - locked <0x357d9553> (a org.jboss.classloader.spi.base.BaseClassLoader)

      java.lang.ClassLoader.loadClass(ClassLoader.java:295)

      - locked <0x22d22fe3> (a org.jboss.web.tomcat.service.WebCtxLoader$ENCLoader)

      java.lang.ClassLoader.loadClass(ClassLoader.java:295)

      - locked <0x7866dc25> (a org.jboss.ejb3.common.classloader.PrimitiveAwareClassLoader)

      java.lang.ClassLoader.loadClass(ClassLoader.java:247)

      org.jboss.ejb3.common.lang.SerializableMethod.getClassFromName(SerializableMethod.java:307)

      org.jboss.ejb3.common.lang.SerializableMethod.toMethod(SerializableMethod.java:239)

      org.jboss.ejb3.common.lang.SerializableMethod.toMethod(SerializableMethod.java:220)

      org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:182)

      org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:164)

      $Proxy815.getAdaptedItem(Unknown Source)

      [servlet method here]

       

      The thread owning 0x1b4ddacc:

       

      Thread: ajp-0.0.0.0-8009-5 : priority:5, demon:true, threadId:192, threadState:RUNNABLE

      java.util.Arrays.copyOfRange(Arrays.java:3209)

      java.lang.String.(String.java:215)

      java.lang.StringBuilder.toString(StringBuilder.java:430)

      org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:448)

      - locked <0x1b4ddacc> (a org.jboss.classloader.spi.base.BaseClassLoader)

      java.lang.ClassLoader.loadClass(ClassLoader.java:295)

      - locked <0x24c3e995> (a org.jboss.ejb3.common.classloader.PrimitiveAwareClassLoader)

      java.lang.ClassLoader.loadClass(ClassLoader.java:247)

      org.jboss.ejb3.common.lang.SerializableMethod.getClassFromName(SerializableMethod.java:307)

      org.jboss.ejb3.common.lang.SerializableMethod.toMethod(SerializableMethod.java:239)

      org.jboss.ejb3.session.SessionSpecContainer.invoke(SessionSpecContainer.java:135)

      org.jboss.ejb3.session.SessionSpecContainer.invoke(SessionSpecContainer.java:216)

      org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:207)

      org.jboss.ejb3.proxy.impl.handler.session.SessionProxyInvocationHandlerBase.invoke(SessionProxyInvocationHandlerBase.java:164)

      $Proxy815.getAdaptedItem(Unknown Source)

      [servlet method here]

       

       

      According to what I've read on the issue pages and in discussions here, the root cause is the classes aren't properly cached, and thus have to be loaded on every single session bean invocation.

       

      Is there a way to work around this issue? Upgrading JBoss is not an option, but if upgrading only the class loader component is possible, we could do that.