1 Reply Latest reply on Dec 14, 2012 6:09 AM by adesanctis

    JBoss AS 5.1 Classloader Deadlock issue

    adesanctis Newbie

      We're experiencing random deadlock related to classloading problems. Below part of the thread dump that show deadlock source:

       

      Found one Java-level deadlock

       

      "ajp-10.220.35.49-8009-301":

        waiting to lock monitor 0x00002aae00b851e8 (object 0x00002aab3a8eb050, a com.livinglogic.struts.workflow.WorkflowContainer),

        which is held by "ajp-10.220.35.49-8009-67"

      "ajp-10.220.35.49-8009-67":

        waiting to lock monitor 0x00002aae35d2d578 (object 0x00002aab0f8c72b0, a java.util.HashMap),

        which is held by "ajp-10.220.35.49-8009-28"

      "ajp-10.220.35.49-8009-28":

        waiting to lock monitor 0x00002aae37d1f210 (object 0x00002aab0a7e49c8, a org.jboss.web.tomcat.service.WebCtxLoader$ENCLoader),

        which is held by "ajp-10.220.35.49-8009-42"

      "ajp-10.220.35.49-8009-42":

        waiting to lock monitor 0x00002aae35f66060 (object 0x00002aaaf1a33d00, a org.jboss.bootstrap.NoAnnotationURLClassLoader),

        which is held by "WorkManager(2)-27"

      "WorkManager(2)-27":

        waiting to lock monitor 0x00002aae35f65f10 (object 0x00002aaaf1a33b60, a sun.misc.Launcher$AppClassLoader),

        which is held by "ajp-10.220.35.49-8009-42"

       

       

      Java stack information for the threads listed above

       

      "ajp-10.220.35.49-8009-301":

          at com.livinglogic.struts.workflow.WorkflowRequestProcessorLogic.process(WorkflowRequestProcessorLogic.java:513)

          - waiting to lock <0x00002aab3a8eb050> (a com.livinglogic.struts.workflow.WorkflowContainer)

          at com.livinglogic.struts.workflow.TilesWorkflowRequestProcessor.process(TilesWorkflowRequestProcessor.java:179)

      ...

          at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)

          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

          at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)

          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

          at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)

          at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

          at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:183)

          at org.jboss.web.tomcat.service.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:135)

          at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:94)

          at org.jboss.web.tomcat.service.session.JvmRouteValve.invoke(JvmRouteValve.java:88)

          at org.jboss.web.tomcat.service.session.LockingValve.invoke(LockingValve.java:62)

          at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)

          at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:95)

          at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)

          at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)

          at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

          at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

          at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)

          at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

          at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)

          at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:436)

          at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:451)

          at java.lang.Thread.run(Thread.java:619)

      "ajp-10.220.35.49-8009-67":

          at org.apache.struts.action.RequestProcessor.processActionCreate(RequestProcessor.java:266)

          - waiting to lock <0x00002aab0f8c72b0> (a java.util.HashMap)

          at com.livinglogic.struts.workflow.TilesWorkflowRequestProcessor.processActionCreate(TilesWorkflowRequestProcessor.java:152)

          at com.livinglogic.struts.workflow.WorkflowRequestProcessorLogic.process(WorkflowRequestProcessorLogic.java:562)

          - locked <0x00002aab3a8eb050> (a com.livinglogic.struts.workflow.WorkflowContainer)

          at com.livinglogic.struts.workflow.TilesWorkflowRequestProcessor.process(TilesWorkflowRequestProcessor.java:179)

      ...

          at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)

          at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)

          at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)

          at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

          at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:638)

          at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:444)

          at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:382)

          at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:310)

          at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1063)

          at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:263)

          at org.apache.struts.action.RequestProcessor.internalModuleRelativeForward(RequestProcessor.java:1001)

          at org.apache.struts.tiles.TilesRequestProcessor.internalModuleRelativeForward(TilesRequestProcessor.java:345)

          at org.apache.struts.action.RequestProcessor.processForward(RequestProcessor.java:560)

          at com.livinglogic.struts.workflow.TilesWorkflowRequestProcessor.processForward(TilesWorkflowRequestProcessor.java:140)

          at com.livinglogic.struts.workflow.WorkflowRequestProcessorLogic.process(WorkflowRequestProcessorLogic.java:554)

          - locked <0x00002aab3a8eb050> (a com.livinglogic.struts.workflow.WorkflowContainer)

          at com.livinglogic.struts.workflow.TilesWorkflowRequestProcessor.process(TilesWorkflowRequestProcessor.java:179)

      ...

          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

          at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)

          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

          at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)

          at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

          at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:183)

          at org.jboss.web.tomcat.service.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:135)

          at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:94)

          at org.jboss.web.tomcat.service.session.JvmRouteValve.invoke(JvmRouteValve.java:88)

          at org.jboss.web.tomcat.service.session.LockingValve.invoke(LockingValve.java:62)

          at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)

          at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:95)

          at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)

          at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)

          at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

          at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

          at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)

          at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

          at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)

          at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:436)

          at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:451)

          at java.lang.Thread.run(Thread.java:619)

      "ajp-10.220.35.49-8009-28":

          at java.lang.ClassLoader.loadClass(ClassLoader.java:292)

          - waiting to lock <0x00002aab0a7e49c8> (a org.jboss.web.tomcat.service.WebCtxLoader$ENCLoader)

          at java.lang.ClassLoader.loadClass(ClassLoader.java:248)

          at org.apache.struts.util.RequestUtils.applicationClass(RequestUtils.java:117)

          at org.apache.struts.util.RequestUtils.applicationInstance(RequestUtils.java:143)

          at org.apache.struts.action.RequestProcessor.processActionCreate(RequestProcessor.java:280)

          - locked <0x00002aab0f8c72b0> (a java.util.HashMap)

          at com.livinglogic.struts.workflow.TilesWorkflowRequestProcessor.processActionCreate(TilesWorkflowRequestProcessor.java:152)

          at com.livinglogic.struts.workflow.WorkflowRequestProcessorLogic.process(WorkflowRequestProcessorLogic.java:562)

          - locked <0x00002aab6d0f25c0> (a com.livinglogic.struts.workflow.WorkflowContainer)

          at com.livinglogic.struts.workflow.TilesWorkflowRequestProcessor.process(TilesWorkflowRequestProcessor.java:179)

      ...

          at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)

          at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)

          at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)

          at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

      ...

          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

      ...

          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

          at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)

          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

          at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)

          at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

          at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:183)

          at org.jboss.web.tomcat.service.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:135)

          at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:94)

          at org.jboss.web.tomcat.service.session.JvmRouteValve.invoke(JvmRouteValve.java:88)

          at org.jboss.web.tomcat.service.session.LockingValve.invoke(LockingValve.java:62)

          at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)

          at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:95)

          at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)

          at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)

          at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

          at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

          at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)

          at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

          at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)

          at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:436)

          at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:451)

          at java.lang.Thread.run(Thread.java:619)

      "ajp-10.220.35.49-8009-42":

          at java.lang.Class.forName0(Native Method)

          at java.lang.Class.forName(Class.java:169)

          at org.jboss.net.protocol.URLStreamHandlerFactory.createURLStreamHandler(URLStreamHandlerFactory.java:183)

          at java.net.URL.getURLStreamHandler(URL.java:1107)

          at java.net.URL. (URL.java:572)

          at java.net.URL. (URL.java:464)

          at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:977)

          at sun.misc.URLClassPath.getResource(URLClassPath.java:169)

          at java.net.URLClassLoader$1.run(URLClassLoader.java:194)

          at java.security.AccessController.doPrivileged(Native Method)

          at java.net.URLClassLoader.findClass(URLClassLoader.java:190)

          at java.lang.ClassLoader.loadClass(ClassLoader.java:307)

          - locked <0x00002aaaf1a33b60> (a sun.misc.Launcher$AppClassLoader)

          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)

          - locked <0x00002aaaf1a33b60> (a sun.misc.Launcher$AppClassLoader)

          at java.lang.ClassLoader.loadClass(ClassLoader.java:248)

          at java.lang.Class.forName0(Native Method)

          at java.lang.Class.forName(Class.java:247)

          at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:300)

          at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1138)

          at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:862)

          at org.jboss.classloader.spi.base.BaseClassLoader.doLoadClass(BaseClassLoader.java:502)

          - locked <0x00002aab009b7c98> (a org.jboss.classloader.spi.base.BaseClassLoader)

          at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:447)

          at java.lang.ClassLoader.loadClass(ClassLoader.java:296)

          - locked <0x00002aab0a7e49c8> (a org.jboss.web.tomcat.service.WebCtxLoader$ENCLoader)

          at java.lang.ClassLoader.loadClass(ClassLoader.java:248)

          at org.apache.xalan.extensions.ObjectFactory.findProviderClass(ObjectFactory.java:508)

          at org.apache.xalan.extensions.ExtensionHandler.getClassForName(ExtensionHandler.java:65)

          at org.apache.xalan.extensions.ExtensionNamespacesManager.defineJavaNamespace(ExtensionNamespacesManager.java:182)

          at org.apache.xalan.extensions.ExtensionNamespacesManager.defineJavaNamespace(ExtensionNamespacesManager.java:157)

          at org.apache.xalan.extensions.ExtensionNamespacesManager.registerUnregisteredNamespaces(ExtensionNamespacesManager.java:132)

          at org.apache.xalan.templates.StylesheetRoot.recompose(StylesheetRoot.java:346)

          at org.apache.xalan.processor.StylesheetHandler.endDocument(StylesheetHandler.java:471)

          at org.apache.xerces.parsers.AbstractSAXParser.endDocument(Unknown Source)

          at org.apache.xerces.impl.XMLDocumentScannerImpl.endEntity(Unknown Source)

          at org.apache.xerces.impl.XMLEntityManager.endEntity(Unknown Source)

          at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)

          at org.apache.xerces.impl.XMLEntityScanner.skipSpaces(Unknown Source)

          at org.apache.xerces.impl.XMLDocumentScannerImpl$TrailingMiscDispatcher.dispatch(Unknown Source)

          at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)

          at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)

          at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)

          at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)

          at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)

          at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)

          at org.apache.xalan.processor.TransformerFactoryImpl.newTemplates(TransformerFactoryImpl.java:922)

          at org.apache.xalan.processor.TransformerFactoryImpl.newTransformer(TransformerFactoryImpl.java:776)

      ....

          at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)

          at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)

          at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)

          at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)

          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

      ....

          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

          at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)

          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

          at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)

          at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

          at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:183)

          at org.jboss.web.tomcat.service.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:135)

          at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:94)

          at org.jboss.web.tomcat.service.session.JvmRouteValve.invoke(JvmRouteValve.java:88)

          at org.jboss.web.tomcat.service.session.LockingValve.invoke(LockingValve.java:62)

          at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)

          at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:95)

          at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)

          at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)

          at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

          at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

          at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)

          at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

          at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)

          at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:436)

          at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384)

          at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:451)

          at java.lang.Thread.run(Thread.java:619)

      "WorkManager(2)-27":

          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:293)

          - waiting to lock <0x00002aaaf1a33b60> (a sun.misc.Launcher$AppClassLoader)

          at java.lang.ClassLoader.loadClass(ClassLoader.java:296)

          - locked <0x00002aaaf1a33d00> (a org.jboss.bootstrap.NoAnnotationURLClassLoader)

          at java.lang.ClassLoader.loadClass(ClassLoader.java:248)

          at java.beans.Introspector.instantiate(Introspector.java:1453)

          at java.beans.PropertyEditorManager.findEditor(PropertyEditorManager.java:91)

          - locked <0x00002aaab3074668> (a java.lang.Class for java.beans.PropertyEditorManager)

          at org.jboss.util.propertyeditor.PropertyEditors.mapJavaBeanProperties(PropertyEditors.java:369)

          at org.jboss.remoting.transport.socket.SocketServerInvoker.mapJavaBeanProperties(SocketServerInvoker.java:1120)

          at org.jboss.remoting.transport.socket.SocketServerInvoker.setup(SocketServerInvoker.java:213)

          at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.setup(BisocketServerInvoker.java:653)

          at org.jboss.remoting.ServerInvoker.create(ServerInvoker.java:971)

          at org.jboss.remoting.transport.Connector.init(Connector.java:482)

          at org.jboss.remoting.transport.Connector.create(Connector.java:1212)

          at org.jboss.remoting.transport.Connector.start(Connector.java:348)

          at org.jboss.remoting.Client.addListener(Client.java:1246)

          at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:259)

          at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:400)

          at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165)

          at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeTarget(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)

          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)

          at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80)

          at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect_z_handleCreateConnectionDelegate_865870853.invoke(StateCreationAspect_z_handleCreateConnectionDelegate_865870853.java)

          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

          at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)

          at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)

          at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101)

          at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95)

      ...

          at sun.reflect.GeneratedMethodAccessor4024.invoke(Unknown Source)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at org.jboss.invocation.Invocation.performCall(Invocation.java:386)

          at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:518)

          at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:156)

          at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:115)

          at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)

          at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)

          at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)

          at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)

          at org.jboss.ejb.plugins.RunAsSecurityInterceptor.process(RunAsSecurityInterceptor.java:133)

          at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:103)

          at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)

          at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)

          at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:420)

          at org.jboss.ejb.Container.invoke(Container.java:1092)

          at sun.reflect.GeneratedMethodAccessor405.invoke(Unknown Source)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

          at java.lang.reflect.Method.invoke(Method.java:597)

          at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)

          at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)

          at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)

          at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)

          at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)

          at org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)

          at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)

          at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)

          at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)

          at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)

          at org.jboss.ejb.plugins.inflow.MessageEndpointInterceptor.delivery(MessageEndpointInterceptor.java:249)

          at org.jboss.ejb.plugins.inflow.MessageEndpointInterceptor.invoke(MessageEndpointInterceptor.java:128)

          at org.jboss.proxy.ClientMethodInterceptor.invoke(ClientMethodInterceptor.java:74)

          at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:101)

          at $Proxy311.onMessage(Unknown Source)

          at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:179)

          at org.jboss.jms.client.container.ClientConsumer.callOnMessageStatic(ClientConsumer.java:160)

          at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:831)

          at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handleRun_865870853.invoke(SessionAspect_z_handleRun_865870853.java)

          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

          at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172)

          at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)

          at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)

          at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)

          at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)

          at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:236)

          at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:205)

          at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

          at java.lang.Thread.run(Thread.java:619)

       

      It is clear to me that some problems are related to livinglogic struts warkflow, but they appear because we have already a lock.

       

      The  real problem is the race condition between:

       

      "ajp-10.220.35.49-8009-42":

        waiting to lock monitor 0x00002aae35f66060 (object 0x00002aaaf1a33d00, a org.jboss.bootstrap.NoAnnotationURLClassLoader),

        which is held by "WorkManager(2)-27"

      "WorkManager(2)-27":

        waiting to lock monitor 0x00002aae35f65f10 (object 0x00002aaaf1a33b60, a sun.misc.Launcher$AppClassLoader),

        which is held by "ajp-10.220.35.49-8009-42"

       

      The different classloader order generates a deadlock.

       

      How can I solve this?

       

      It appears many times in a day, and sometime for a long period nothing happens.

       

      I finish my ideas

       

      Thank you

       

      Alessandro

        • 1. Re: JBoss AS 5.1 Classloader Deadlock issue
          adesanctis Newbie

          Some enviroment varaibles:

           

          08:46:31,072 INFO  [ServerInfo] Java version: 1.6.0_18,Sun Microsystems Inc.

          08:46:31,072 INFO  [ServerInfo] Java Runtime: Java(TM) SE Runtime Environment (build 1.6.0_18-b07)

          08:46:31,072 INFO  [ServerInfo] Java VM: Java HotSpot(TM) 64-Bit Server VM 16.0-b13,Sun Microsystems Inc.

          08:46:31,072 INFO  [ServerInfo] OS-System: Linux 2.6.18-274.el5,amd64

           

          Alessandro