JBoss AS 5.1 Classloader Deadlock issue
We're experiencing random deadlock related to classloading problems. Below part of the thread dump that show deadlock source:
Found one Java-level deadlock
"ajp-10.220.35.49-8009-301":
waiting to lock monitor 0x00002aae00b851e8 (object 0x00002aab3a8eb050, a com.livinglogic.struts.workflow.WorkflowContainer),
which is held by "ajp-10.220.35.49-8009-67"
"ajp-10.220.35.49-8009-67":
waiting to lock monitor 0x00002aae35d2d578 (object 0x00002aab0f8c72b0, a java.util.HashMap),
which is held by "ajp-10.220.35.49-8009-28"
"ajp-10.220.35.49-8009-28":
waiting to lock monitor 0x00002aae37d1f210 (object 0x00002aab0a7e49c8, a org.jboss.web.tomcat.service.WebCtxLoader$ENCLoader),
which is held by "ajp-10.220.35.49-8009-42"
"ajp-10.220.35.49-8009-42":
waiting to lock monitor 0x00002aae35f66060 (object 0x00002aaaf1a33d00, a org.jboss.bootstrap.NoAnnotationURLClassLoader),
which is held by "WorkManager(2)-27"
"WorkManager(2)-27":
waiting to lock monitor 0x00002aae35f65f10 (object 0x00002aaaf1a33b60, a sun.misc.Launcher$AppClassLoader),
which is held by "ajp-10.220.35.49-8009-42"
Java stack information for the threads listed above
"ajp-10.220.35.49-8009-301":
at com.livinglogic.struts.workflow.WorkflowRequestProcessorLogic.process(WorkflowRequestProcessorLogic.java:513)
- waiting to lock <0x00002aab3a8eb050> (a com.livinglogic.struts.workflow.WorkflowContainer)
at com.livinglogic.struts.workflow.TilesWorkflowRequestProcessor.process(TilesWorkflowRequestProcessor.java:179)
...
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:183)
at org.jboss.web.tomcat.service.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:135)
at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:94)
at org.jboss.web.tomcat.service.session.JvmRouteValve.invoke(JvmRouteValve.java:88)
at org.jboss.web.tomcat.service.session.LockingValve.invoke(LockingValve.java:62)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:95)
at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:436)
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:451)
at java.lang.Thread.run(Thread.java:619)
"ajp-10.220.35.49-8009-67":
at org.apache.struts.action.RequestProcessor.processActionCreate(RequestProcessor.java:266)
- waiting to lock <0x00002aab0f8c72b0> (a java.util.HashMap)
at com.livinglogic.struts.workflow.TilesWorkflowRequestProcessor.processActionCreate(TilesWorkflowRequestProcessor.java:152)
at com.livinglogic.struts.workflow.WorkflowRequestProcessorLogic.process(WorkflowRequestProcessorLogic.java:562)
- locked <0x00002aab3a8eb050> (a com.livinglogic.struts.workflow.WorkflowContainer)
at com.livinglogic.struts.workflow.TilesWorkflowRequestProcessor.process(TilesWorkflowRequestProcessor.java:179)
...
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:638)
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:444)
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:382)
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:310)
at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1063)
at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:263)
at org.apache.struts.action.RequestProcessor.internalModuleRelativeForward(RequestProcessor.java:1001)
at org.apache.struts.tiles.TilesRequestProcessor.internalModuleRelativeForward(TilesRequestProcessor.java:345)
at org.apache.struts.action.RequestProcessor.processForward(RequestProcessor.java:560)
at com.livinglogic.struts.workflow.TilesWorkflowRequestProcessor.processForward(TilesWorkflowRequestProcessor.java:140)
at com.livinglogic.struts.workflow.WorkflowRequestProcessorLogic.process(WorkflowRequestProcessorLogic.java:554)
- locked <0x00002aab3a8eb050> (a com.livinglogic.struts.workflow.WorkflowContainer)
at com.livinglogic.struts.workflow.TilesWorkflowRequestProcessor.process(TilesWorkflowRequestProcessor.java:179)
...
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:183)
at org.jboss.web.tomcat.service.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:135)
at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:94)
at org.jboss.web.tomcat.service.session.JvmRouteValve.invoke(JvmRouteValve.java:88)
at org.jboss.web.tomcat.service.session.LockingValve.invoke(LockingValve.java:62)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:95)
at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:436)
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:451)
at java.lang.Thread.run(Thread.java:619)
"ajp-10.220.35.49-8009-28":
at java.lang.ClassLoader.loadClass(ClassLoader.java:292)
- waiting to lock <0x00002aab0a7e49c8> (a org.jboss.web.tomcat.service.WebCtxLoader$ENCLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at org.apache.struts.util.RequestUtils.applicationClass(RequestUtils.java:117)
at org.apache.struts.util.RequestUtils.applicationInstance(RequestUtils.java:143)
at org.apache.struts.action.RequestProcessor.processActionCreate(RequestProcessor.java:280)
- locked <0x00002aab0f8c72b0> (a java.util.HashMap)
at com.livinglogic.struts.workflow.TilesWorkflowRequestProcessor.processActionCreate(TilesWorkflowRequestProcessor.java:152)
at com.livinglogic.struts.workflow.WorkflowRequestProcessorLogic.process(WorkflowRequestProcessorLogic.java:562)
- locked <0x00002aab6d0f25c0> (a com.livinglogic.struts.workflow.WorkflowContainer)
at com.livinglogic.struts.workflow.TilesWorkflowRequestProcessor.process(TilesWorkflowRequestProcessor.java:179)
...
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)
at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
...
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
...
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:183)
at org.jboss.web.tomcat.service.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:135)
at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:94)
at org.jboss.web.tomcat.service.session.JvmRouteValve.invoke(JvmRouteValve.java:88)
at org.jboss.web.tomcat.service.session.LockingValve.invoke(LockingValve.java:62)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:95)
at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:436)
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:451)
at java.lang.Thread.run(Thread.java:619)
"ajp-10.220.35.49-8009-42":
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at org.jboss.net.protocol.URLStreamHandlerFactory.createURLStreamHandler(URLStreamHandlerFactory.java:183)
at java.net.URL.getURLStreamHandler(URL.java:1107)
at java.net.URL. (URL.java:572)
at java.net.URL. (URL.java:464)
at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:977)
at sun.misc.URLClassPath.getResource(URLClassPath.java:169)
at java.net.URLClassLoader$1.run(URLClassLoader.java:194)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
- locked <0x00002aaaf1a33b60> (a sun.misc.Launcher$AppClassLoader)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
- locked <0x00002aaaf1a33b60> (a sun.misc.Launcher$AppClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:300)
at org.jboss.classloader.spi.base.BaseClassLoaderDomain.loadClass(BaseClassLoaderDomain.java:1138)
at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:862)
at org.jboss.classloader.spi.base.BaseClassLoader.doLoadClass(BaseClassLoader.java:502)
- locked <0x00002aab009b7c98> (a org.jboss.classloader.spi.base.BaseClassLoader)
at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:447)
at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
- locked <0x00002aab0a7e49c8> (a org.jboss.web.tomcat.service.WebCtxLoader$ENCLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at org.apache.xalan.extensions.ObjectFactory.findProviderClass(ObjectFactory.java:508)
at org.apache.xalan.extensions.ExtensionHandler.getClassForName(ExtensionHandler.java:65)
at org.apache.xalan.extensions.ExtensionNamespacesManager.defineJavaNamespace(ExtensionNamespacesManager.java:182)
at org.apache.xalan.extensions.ExtensionNamespacesManager.defineJavaNamespace(ExtensionNamespacesManager.java:157)
at org.apache.xalan.extensions.ExtensionNamespacesManager.registerUnregisteredNamespaces(ExtensionNamespacesManager.java:132)
at org.apache.xalan.templates.StylesheetRoot.recompose(StylesheetRoot.java:346)
at org.apache.xalan.processor.StylesheetHandler.endDocument(StylesheetHandler.java:471)
at org.apache.xerces.parsers.AbstractSAXParser.endDocument(Unknown Source)
at org.apache.xerces.impl.XMLDocumentScannerImpl.endEntity(Unknown Source)
at org.apache.xerces.impl.XMLEntityManager.endEntity(Unknown Source)
at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)
at org.apache.xerces.impl.XMLEntityScanner.skipSpaces(Unknown Source)
at org.apache.xerces.impl.XMLDocumentScannerImpl$TrailingMiscDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
at org.apache.xalan.processor.TransformerFactoryImpl.newTemplates(TransformerFactoryImpl.java:922)
at org.apache.xalan.processor.TransformerFactoryImpl.newTransformer(TransformerFactoryImpl.java:776)
....
at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1194)
at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:432)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
....
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:183)
at org.jboss.web.tomcat.service.session.ClusteredSessionValve.handleRequest(ClusteredSessionValve.java:135)
at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:94)
at org.jboss.web.tomcat.service.session.JvmRouteValve.invoke(JvmRouteValve.java:88)
at org.jboss.web.tomcat.service.session.LockingValve.invoke(LockingValve.java:62)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:95)
at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:436)
at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:451)
at java.lang.Thread.run(Thread.java:619)
"WorkManager(2)-27":
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:293)
- waiting to lock <0x00002aaaf1a33b60> (a sun.misc.Launcher$AppClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:296)
- locked <0x00002aaaf1a33d00> (a org.jboss.bootstrap.NoAnnotationURLClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.beans.Introspector.instantiate(Introspector.java:1453)
at java.beans.PropertyEditorManager.findEditor(PropertyEditorManager.java:91)
- locked <0x00002aaab3074668> (a java.lang.Class for java.beans.PropertyEditorManager)
at org.jboss.util.propertyeditor.PropertyEditors.mapJavaBeanProperties(PropertyEditors.java:369)
at org.jboss.remoting.transport.socket.SocketServerInvoker.mapJavaBeanProperties(SocketServerInvoker.java:1120)
at org.jboss.remoting.transport.socket.SocketServerInvoker.setup(SocketServerInvoker.java:213)
at org.jboss.remoting.transport.bisocket.BisocketServerInvoker.setup(BisocketServerInvoker.java:653)
at org.jboss.remoting.ServerInvoker.create(ServerInvoker.java:971)
at org.jboss.remoting.transport.Connector.init(Connector.java:482)
at org.jboss.remoting.transport.Connector.create(Connector.java:1212)
at org.jboss.remoting.transport.Connector.start(Connector.java:348)
at org.jboss.remoting.Client.addListener(Client.java:1246)
at org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:259)
at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:400)
at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:165)
at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeTarget(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80)
at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect_z_handleCreateConnectionDelegate_865870853.invoke(StateCreationAspect_z_handleCreateConnectionDelegate_865870853.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:101)
at org.jboss.jms.client.JBossConnectionFactory.createQueueConnection(JBossConnectionFactory.java:95)
...
at sun.reflect.GeneratedMethodAccessor4024.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.invocation.Invocation.performCall(Invocation.java:386)
at org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:518)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:156)
at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:115)
at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
at org.jboss.ejb.plugins.RunAsSecurityInterceptor.process(RunAsSecurityInterceptor.java:133)
at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:103)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:420)
at org.jboss.ejb.Container.invoke(Container.java:1092)
at sun.reflect.GeneratedMethodAccessor405.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
at org.jboss.invocation.local.LocalInvoker$MBeanServerAction.invoke(LocalInvoker.java:169)
at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:118)
at org.jboss.invocation.InvokerInterceptor.invokeLocal(InvokerInterceptor.java:209)
at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:195)
at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)
at org.jboss.ejb.plugins.inflow.MessageEndpointInterceptor.delivery(MessageEndpointInterceptor.java:249)
at org.jboss.ejb.plugins.inflow.MessageEndpointInterceptor.invoke(MessageEndpointInterceptor.java:128)
at org.jboss.proxy.ClientMethodInterceptor.invoke(ClientMethodInterceptor.java:74)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:101)
at $Proxy311.onMessage(Unknown Source)
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:179)
at org.jboss.jms.client.container.ClientConsumer.callOnMessageStatic(ClientConsumer.java:160)
at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:831)
at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handleRun_865870853.invoke(SessionAspect_z_handleRun_865870853.java)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172)
at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:236)
at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:205)
at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
It is clear to me that some problems are related to livinglogic struts warkflow, but they appear because we have already a lock.
The real problem is the race condition between:
"ajp-10.220.35.49-8009-42":
waiting to lock monitor 0x00002aae35f66060 (object 0x00002aaaf1a33d00, a org.jboss.bootstrap.NoAnnotationURLClassLoader),
which is held by "WorkManager(2)-27"
"WorkManager(2)-27":
waiting to lock monitor 0x00002aae35f65f10 (object 0x00002aaaf1a33b60, a sun.misc.Launcher$AppClassLoader),
which is held by "ajp-10.220.35.49-8009-42"
The different classloader order generates a deadlock.
How can I solve this?
It appears many times in a day, and sometime for a long period nothing happens.
I finish my ideas
Thank you
Alessandro