JBoss AOP 2.0.0 possible deadlock?
So, we've been progressing with integrating JBoss AOP 2.0.0 into JBoss AS 4.2.2 out of necessity to pull in JBoss Cache 2.1.1 for a critical bug fix. We're not ready to make the move to JBoss AS 5 so that's what brings us here. We've followed the instructions in the JBoss 4 area of the package by running the ant script appropriately. Additionally, we found that we needed to pull out the org.jboss.aop package from client/jbossall-client as it was using I presume JBoss AOP 1.5. We also replaced client/jboss-aop-jdk50-client.jar with the one from lib-50 in the AOP 2.0.0 download. Since pulling in JBoss AOP 2.0.0 (initially CR8, now updated to CR13), we are seeing an intermittent deadlock upon AS startup. We use Spring injection in some of our MBeans and there is 1 specific MBean that gets stuck on startup when parsing the Spring XML configuration. Here is the last line in our log:
21:32:51,653 INFO [XmlBeanDefinitionReader] Loading XML bean definitions from class path resource [spring/applicationContext-policymgr.xml]
When I attach my debugger to JBoss I find that this particular thread (the JBoss main thread) is stuck at AspectManager.java:967. That seems to relate to this line of code:
if (isNonAdvisableClassName(className))
inside of this method:
public synchronized byte[] translate(String className, ClassLoader loader, byte[] classfileBuffer) throws Exception
After inspecting some of the other threads, I see there is another thread in the same method (even though this is supposedly synchronized). My thread (Thread-10) is stuck at line 988 in AspectManager which is that same exact method. So, I'm wondering if someone can give me some insight as to what might be going on here since the method is synchronized. This is a big problem as it essentially deadlocks the entire AS (everything stops loading and never finishes startup). Below are the stack traces for the 2 threads in question. This is an intermittent problem as it appears the timing of the two threads on my side need to enter at almost the same instant. I'm going to play around with dependencies on my side which could possibly avoid this altogether.
The "main" thread:
Thread [main] (Suspended) AspectManager.translate(String, ClassLoader, byte[]) line: 967 AspectManager.transform(ClassLoader, String, Class, ProtectionDomain, byte[]) line: 934 AOPTransformer.aspectTransform(String, ClassLoader, Class<?>, ProtectionDomain, byte[]) line: 85 AOPTransformer.transform(ClassLoader, String, Class<?>, ProtectionDomain, byte[]) line: 73 TransformerManager.transform(ClassLoader, String, Class, ProtectionDomain, byte[]) line: 122 InstrumentationImpl.transform(ClassLoader, String, Class, ProtectionDomain, byte[]) line: 155 CMNodeFactory.getCMLeafNode(int, Object, int, int) line: not available CMBuilder.buildSyntaxTree(XSParticleDecl) line: not available CMBuilder.buildSyntaxTree(XSParticleDecl) line: not available CMBuilder.createDFACM(XSParticleDecl) line: not available CMBuilder.getContentModel(XSComplexTypeDecl) line: not available XSComplexTypeDecl.getContentModel(CMBuilder) line: not available XMLSchemaValidator.handleStartElement(QName, XMLAttributes, Augmentations) line: not available XMLSchemaValidator.startElement(QName, XMLAttributes, Augmentations) line: not available XMLNSDocumentScannerImpl.scanStartElement() line: not available XMLNSDocumentScannerImpl$NSContentDispatcher.scanRootElementHook() line: not available XMLNSDocumentScannerImpl$NSContentDispatcher(XMLDocumentFragmentScannerImpl$FragmentContentDispatcher).dispatch(boolean) line: not available XMLNSDocumentScannerImpl(XMLDocumentFragmentScannerImpl).scanDocument(boolean) line: not available XIncludeAwareParserConfiguration(XML11Configuration).parse(boolean) line: not available XIncludeAwareParserConfiguration(XML11Configuration).parse(XMLInputSource) line: not available DOMParser(XMLParser).parse(XMLInputSource) line: not available DOMParser.parse(InputSource) line: not available DocumentBuilderImpl.parse(InputSource) line: not available DefaultDocumentLoader.loadDocument(InputSource, EntityResolver, ErrorHandler, int, boolean) line: 78 XmlBeanDefinitionReader.doLoadBeanDefinitions(InputSource, Resource) line: 361 XmlBeanDefinitionReader.loadBeanDefinitions(EncodedResource) line: 313 XmlBeanDefinitionReader.loadBeanDefinitions(Resource) line: 290 XmlBeanDefinitionReader(AbstractBeanDefinitionReader).loadBeanDefinitions(Resource[]) line: 142 XmlBeanDefinitionReader(AbstractBeanDefinitionReader).loadBeanDefinitions(String) line: 158 XmlBeanDefinitionReader(AbstractBeanDefinitionReader).loadBeanDefinitions(String[]) line: 184 ClassPathXmlApplicationContext(AbstractXmlApplicationContext).loadBeanDefinitions(XmlBeanDefinitionReader) line: 112 ClassPathXmlApplicationContext(AbstractXmlApplicationContext).loadBeanDefinitions(DefaultListableBeanFactory) line: 79 ClassPathXmlApplicationContext(AbstractRefreshableApplicationContext).refreshBeanFactory() line: 97 ClassPathXmlApplicationContext(AbstractApplicationContext).obtainFreshBeanFactory() line: 411 ClassPathXmlApplicationContext(AbstractApplicationContext).refresh() line: 338 ClassPathXmlApplicationContext.<init>(String[], boolean, ApplicationContext) line: 122 ClassPathXmlApplicationContext.<init>(String[]) line: 76 PolicyManagerDelegateImpl.<init>() line: 28 PolicyManagerService.startService() line: 53 PolicyManagerService(ServiceMBeanSupport).jbossInternalStart() line: 289 PolicyManagerService(ServiceMBeanSupport).jbossInternalLifecycle(String) line: 245 GeneratedMethodAccessor3.invoke(Object, Object[]) line: not available DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25 Method.invoke(Object, Object...) line: 585 ReflectedDispatcher.invoke(Invocation) line: 155 Invocation.dispatch() line: 94 Invocation.invoke() line: 86 XMBean(AbstractMBeanInvoker).invoke(String, Object[], String[]) line: 264 MBeanServerImpl.invoke(ObjectName, String, Object[], String[]) line: 659 ServiceController$ServiceProxy.invoke(Object, Method, Object[]) line: 978 $Proxy0.start() line: not available ServiceController.start(ObjectName) line: 417 ServiceController.start(ObjectName) line: 435 GeneratedMethodAccessor9.invoke(Object, Object[]) line: not available DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25 Method.invoke(Object, Object...) line: 585 ReflectedDispatcher.invoke(Invocation) line: 155 Invocation.dispatch() line: 94 Invocation.invoke() line: 86 XMBean(AbstractMBeanInvoker).invoke(String, Object[], String[]) line: 264 MBeanServerImpl.invoke(ObjectName, String, Object[], String[]) line: 659 MBeanProxyExt.invoke(Object, Method, Object[]) line: 210 $Proxy4.start(ObjectName) line: not available SARDeployer.start(DeploymentInfo) line: 302 MainDeployer.start(DeploymentInfo) line: 1025 MainDeployer.start(DeploymentInfo) line: 1015 MainDeployer.deploy(DeploymentInfo) line: 819 MainDeployer.deploy(URL) line: 782 GeneratedMethodAccessor20.invoke(Object, Object[]) line: not available DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25 Method.invoke(Object, Object...) line: 585 ReflectedDispatcher.invoke(Invocation) line: 155 Invocation.dispatch() line: 94 NullInterceptor(AbstractInterceptor).invoke(Invocation) line: 133 Invocation.invoke() line: 88 ModelMBeanOperationInterceptor.invoke(Invocation) line: 142 Invocation.invoke() line: 88 XMBean(AbstractMBeanInvoker).invoke(String, Object[], String[]) line: 264 MBeanServerImpl.invoke(ObjectName, String, Object[], String[]) line: 659 MBeanProxyExt.invoke(Object, Method, Object[]) line: 210 $Proxy9.deploy(URL) line: not available URLDeploymentScanner.deploy(URLDeploymentScanner$DeployedURL) line: 421 URLDeploymentScanner.scan() line: 634 AbstractDeploymentScanner$ScannerThread.doScan() line: 263 URLDeploymentScanner(AbstractDeploymentScanner).startService() line: 336 URLDeploymentScanner(ServiceMBeanSupport).jbossInternalStart() line: 289 URLDeploymentScanner(ServiceMBeanSupport).jbossInternalLifecycle(String) line: 245 GeneratedMethodAccessor3.invoke(Object, Object[]) line: not available DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25 Method.invoke(Object, Object...) line: 585 ReflectedDispatcher.invoke(Invocation) line: 155 Invocation.dispatch() line: 94 Invocation.invoke() line: 86 XMBean(AbstractMBeanInvoker).invoke(String, Object[], String[]) line: 264 MBeanServerImpl.invoke(ObjectName, String, Object[], String[]) line: 659 ServiceController$ServiceProxy.invoke(Object, Method, Object[]) line: 978 $Proxy0.start() line: not available ServiceController.start(ObjectName) line: 417 GeneratedMethodAccessor9.invoke(Object, Object[]) line: not available DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25 Method.invoke(Object, Object...) line: 585 ReflectedDispatcher.invoke(Invocation) line: 155 Invocation.dispatch() line: 94 Invocation.invoke() line: 86 XMBean(AbstractMBeanInvoker).invoke(String, Object[], String[]) line: 264 MBeanServerImpl.invoke(ObjectName, String, Object[], String[]) line: 659 MBeanProxyExt.invoke(Object, Method, Object[]) line: 210 $Proxy4.start(ObjectName) line: not available SARDeployer.start(DeploymentInfo) line: 302 MainDeployer.start(DeploymentInfo) line: 1025 MainDeployer.deploy(DeploymentInfo) line: 819 MainDeployer.deploy(URL) line: 782 MainDeployer.deploy(String) line: 766 NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method] NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39 DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25 Method.invoke(Object, Object...) line: 585 ReflectedDispatcher.invoke(Invocation) line: 155 Invocation.dispatch() line: 94 NullInterceptor(AbstractInterceptor).invoke(Invocation) line: 133 Invocation.invoke() line: 88 ModelMBeanOperationInterceptor.invoke(Invocation) line: 142 Invocation.invoke() line: 88 XMBean(AbstractMBeanInvoker).invoke(String, Object[], String[]) line: 264 MBeanServerImpl.invoke(ObjectName, String, Object[], String[]) line: 659 MBeanProxyExt.invoke(Object, Method, Object[]) line: 210 $Proxy5.deploy(String) line: not available ServerImpl.doStart() line: 482 ServerImpl.start() line: 362 Main.boot(String[]) line: 200 Main$1.run() line: 508 Thread.run() line: 595
"Thread 10"
Thread [Thread-10] (Suspended) ClassLoader.findBootstrapClass(String) line: not available [native method] Launcher$ExtClassLoader(ClassLoader).findBootstrapClass0(String) line: 891 Launcher$ExtClassLoader(ClassLoader).loadClass(String, boolean) line: 301 Launcher$AppClassLoader(ClassLoader).loadClass(String, boolean) line: 299 Launcher$AppClassLoader.loadClass(String, boolean) line: 268 NoAnnotationURLClassLoader(ClassLoader).loadClass(String, boolean) line: 299 UnifiedClassLoader3(ClassLoader).loadClass(String, boolean) line: 299 UnifiedClassLoader3(RepositoryClassLoader).loadClassLocally(String, boolean) line: 200 ClassLoadingTask$ThreadTask.run() line: 131 LoadMgr3.nextTask(Thread, ClassLoadingTask, UnifiedLoaderRepository3) line: 399 UnifiedClassLoader3(RepositoryClassLoader).loadClassImpl(String, boolean, int) line: 527 UnifiedClassLoader3(RepositoryClassLoader).loadClass(String, boolean) line: 415 UnifiedClassLoader3(ClassLoader).loadClass(String) line: 251 JBossClassPool.toClass(CtClass, ClassLoader, ProtectionDomain) line: 109 CtNewNestedClass(CtClass).toClass(ClassLoader, ProtectionDomain) line: 1094 TransformerCommon$ToClassAction$2.toClass(CtClass, ClassLoader, ProtectionDomain) line: 293 TransformerCommon.compileOrLoadClass(CtClass, CtClass, boolean) line: 103 TransformerCommon.compileOrLoadClass(CtClass, CtClass) line: 64 FieldJoinPointGenerator$WriteBaseClassGenerator(FieldJoinPointGenerator$BaseClassGenerator).generate() line: 265 FieldJoinPointGenerator.createWriteJoinpointBaseClass(GeneratedAdvisorInstrumentor, CtClass, CtField, String, int) line: 221 GeneratedAdvisorFieldAccessTransformer.createWriteJoinPointClass(CtField, String, int) line: 184 GeneratedAdvisorFieldAccessTransformer.addWriteJoinPoint(CtField, String, int) line: 172 GeneratedAdvisorFieldAccessTransformer.addFieldWriteInfoFieldToGeneratedAdvisor(CtField, int) line: 161 GeneratedAdvisorFieldAccessTransformer.doBuildFieldWrappers(CtClass, CtField, int, boolean, JoinpointClassification, JoinpointClassification) line: 77 GeneratedAdvisorFieldAccessTransformer(FieldAccessTransformer).buildFieldWrappers(CtClass, ClassAdvisor, boolean) line: 103 GeneratedAdvisorInstrumentor(Instrumentor).transform(CtClass, ClassAdvisor) line: 771 GeneratedAdvisorInstrumentor.transform(CtClass, ClassAdvisor) line: 117 SuperClassesFirstWeavingStrategy.instrumentClass(AspectManager, AOPClassPool, CtClass, boolean) line: 204 SuperClassesFirstWeavingStrategy.translate(AspectManager, String, ClassLoader, byte[]) line: 70 AspectManager.translate(String, ClassLoader, byte[]) line: 988 AspectManager.transform(ClassLoader, String, Class, ProtectionDomain, byte[]) line: 934 AOPTransformer.aspectTransform(String, ClassLoader, Class<?>, ProtectionDomain, byte[]) line: 85 AOPTransformer.transform(ClassLoader, String, Class<?>, ProtectionDomain, byte[]) line: 73 TransformerManager.transform(ClassLoader, String, Class, ProtectionDomain, byte[]) line: 122 InstrumentationImpl.transform(ClassLoader, String, Class, ProtectionDomain, byte[]) line: 155 ClassLoader.defineClass1(String, byte[], int, int, ProtectionDomain, String) line: not available [native method] UnifiedClassLoader3(ClassLoader).defineClass(String, byte[], int, int, ProtectionDomain) line: 620 UnifiedClassLoader3(SecureClassLoader).defineClass(String, byte[], int, int, CodeSource) line: 124 UnifiedClassLoader3(URLClassLoader).defineClass(String, Resource) line: 260 URLClassLoader.access$100(URLClassLoader, String, Resource) line: 56 URLClassLoader$1.run() line: 195 AccessController.doPrivileged(PrivilegedExceptionAction<T>, AccessControlContext) line: not available [native method] UnifiedClassLoader3(URLClassLoader).findClass(String) line: 188 UnifiedClassLoader3(RepositoryClassLoader).findClassLocally(String) line: 682 UnifiedClassLoader3(RepositoryClassLoader).findClass(String) line: 662 UnifiedClassLoader3(ClassLoader).loadClass(String, boolean) line: 306 UnifiedClassLoader3(RepositoryClassLoader).loadClassLocally(String, boolean) line: 200 ClassLoadingTask$ThreadTask.run() line: 131 LoadMgr3.nextTask(Thread, ClassLoadingTask, UnifiedLoaderRepository3) line: 399 UnifiedClassLoader3(RepositoryClassLoader).loadClassImpl(String, boolean, int) line: 527 UnifiedClassLoader3(RepositoryClassLoader).loadClass(String, boolean) line: 415 UnifiedClassLoader3(ClassLoader).loadClass(String) line: 251 UnifiedClassLoader3(ClassLoader).loadClassInternal(String) line: 319 Class<T>.getDeclaredFields0(boolean) line: not available [native method] Class<T>.privateGetDeclaredFields(boolean) line: 2259 Class<T>.getDeclaredField(String) line: 1852 FieldInfo.doGet(Class<?>, String) line: 181 FieldInfo.access$000(FieldInfo, Class, String) line: 42 FieldInfo$GetDeclaredFieldAction$2.get(FieldInfo, Class<?>, String) line: 232 FieldInfo.<init>(Class<?>, int, String, long, Advisor, boolean) line: 67 Domain$DomainAdvisor.initialiseFieldReads(Collection) line: not available GeneratedClassAdvisor$ClassAdvisorStrategy.initialise(Class<?>, AspectManager) line: 1267 Domain$DomainAdvisor(GeneratedClassAdvisor).initialise(Class<?>, AspectManager) line: 141 Domain$DomainAdvisor.initialise(AspectManager, boolean) line: not available Domain$DomainAdvisor.<init>() line: not available Domain.<clinit>() line: not available Class<T>.forName0(String, boolean, ClassLoader) line: not available [native method] Class<T>.forName(String) line: 164 $Proxy79.<clinit>() line: not available NativeConstructorAccessorImpl.newInstance0(Constructor, Object[]) line: not available [native method] NativeConstructorAccessorImpl.newInstance(Object[]) line: 39 DelegatingConstructorAccessorImpl.newInstance(Object[]) line: 27 Constructor<T>.newInstance(Object...) line: 494 Proxy.newProxyInstance(ClassLoader, Class<?>[], InvocationHandler) line: 588 FabricDelegateProvider.getFabricDelegate() line: 49 FabricDelegateProvider.<clinit>() line: 30 ActiveDirectoryRefresh.refreshCache() line: 150 ActiveDirectoryRefresh.run() line: 115