IllegalStateException: ClassLoadingTask.loadedTask is null

[youngkin]

I'm having intermittent problems during initialization of an application deployed as a .sar. I'm getting the error mentioned in the subject line. When the error occurs it usually moves around - i.e,. the stack trace location and offending class changes from occurrence to occurrence.

The context:

1. JBoss 4.0.3 SP1
2. There are several other applications deployed in the same JBoss instance - other .sars as well as an .ear that includes a .war.
3. JMX is used by the .sar to start other internal services.

Attached below is a partial stack trace. Thanks in advance.

Rich

java.lang.IllegalStateException: ClassLoadingTask.loadedTask is null, name: javax.xml.rpc.handler.HandlerRegistry
 at org.jboss.mx.loading.RepositoryClassLoader.loadClassImpl(RepositoryClassLoader.java:516)
 at org.jboss.mx.loading.RepositoryClassLoader.loadClass(RepositoryClassLoader.java:377)
 at java.lang.ClassLoader.loadClass(Unknown Source)
 at java.lang.ClassLoader.loadClassInternal(Unknown Source)
 at example.MyClass.foo(MyClass.java:72)
 at org.jboss.mx.server.RawDynamicInvoker.invoke(RawDynamicInvoker.java:150)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
 at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:225)
 at com.avaya.ccr.jmx.JmxUtil.invoke(JmxUtil.java:69)
 at example.MyService.start(MyService.java:48)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:245)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
 at java.util.concurrent.FutureTask.run(Unknown Source)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)

[youngkin]

Hi,

I'm still trying to figure this one out. Here is some additional information.

I turned on UCL debugging and here's the pertinent log output

[85895,LoadMgr3,pool-1-thread-1] scheduleTask(1), created subtask: {t=Thread[main,5,jboss], ucl=org.jboss.mx.loading.UnifiedClassLoader3@55bb93{ url=file:/opt/coreservices/jboss-4.0.3SP1/server/ccr3/tmp/deploy/tmp29316jboss-service.xml ,addedOrder=2}, name=com.avaya.ccr.common.jmxtools.JmxUtil, requestingThread=Thread[pool-1-thread-1,5,jboss], order=2, releaseInNextTask=false}
[85895,LoadMgr3,pool-1-thread-1] End beginLoadTask, task=org.jboss.mx.loading.ClassLoadingTask@3c33d3{classname: com.avaya.ccr.common.jmxtools.JmxUtil, requestingThread: Thread[pool-1-thread-1,5,jboss], requestingClassLoader: org.jboss.mx.loading.UnifiedClassLoader3@381a53{ url=file:/opt/coreservices/jboss-4.0.3SP1/server/ccr3/tmp/deploy/tmp29328CCR_RtReportExecSvc.sar ,addedOrder=8}, loadedClass: nullnull, loadOrder: 2147483647, loadException: null, threadTaskCount: 1, state: 1, #CCE: 0}
[85895,LoadMgr3,pool-1-thread-1] Begin nextTask(WAIT_ON_EVENT), task=org.jboss.mx.loading.ClassLoadingTask@3c33d3{classname: com.avaya.ccr.common.jmxtools.JmxUtil, requestingThread: Thread[pool-1-thread-1,5,jboss], requestingClassLoader: org.jboss.mx.loading.UnifiedClassLoader3@381a53{ url=file:/opt/coreservices/jboss-4.0.3SP1/server/ccr3/tmp/deploy/tmp29328CCR_RtReportExecSvc.sar ,addedOrder=8}, loadedClass: nullnull, loadOrder: 2147483647, loadException: null, threadTaskCount: 1, state: 1, #CCE: 0}
[85895,LoadMgr3,pool-1-thread-1] nextTask(WAIT_ON_EVENT), interrupted, task=org.jboss.mx.loading.ClassLoadingTask@3c33d3{classname: com.avaya.ccr.common.jmxtools.JmxUtil, requestingThread: Thread[pool-1-thread-1,5,jboss], requestingClassLoader: org.jboss.mx.loading.UnifiedClassLoader3@381a53{ url=file:/opt/coreservices/jboss-4.0.3SP1/server/ccr3/tmp/deploy/tmp29328CCR_RtReportExecSvc.sar ,addedOrder=8}, loadedClass: nullnull, loadOrder: 2147483647, loadException: null, threadTaskCount: 1, state: 3, #CCE: 0}
java.lang.InterruptedException
 at java.lang.Object.wait(Native Method)
 at java.lang.Object.wait(Unknown Source)
 at org.jboss.mx.loading.LoadMgr3.nextTask(LoadMgr3.java:337)
 at org.jboss.mx.loading.RepositoryClassLoader.loadClassImpl(RepositoryClassLoader.java:481)
 at org.jboss.mx.loading.RepositoryClassLoader.loadClass(RepositoryClassLoader.java:377)
 at java.lang.ClassLoader.loadClass(Unknown Source)
 at java.lang.ClassLoader.loadClassInternal(Unknown Source)
 at com.avaya.ccr.reportingtools.realtime.server.ReportingToolsServiceManager.start(ReportingToolsServiceManager.java:173)
 at com.avaya.ccr.reportingtools.realtime.server.ReportingToolsServiceManagerMBean.invoke(ReportingToolsServiceManagerMBean.java:150)
 at org.jboss.mx.server.RawDynamicInvoker.invoke(RawDynamicInvoker.java:150)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
 at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:225)
 at com.avaya.ccr.jmx.JmxUtil.invoke(JmxUtil.java:69)
 at com.avaya.ccr.reportingtools.realtime.server.ReportingToolsServiceManagerFactory.startPE(ReportingToolsServiceManagerFactory.java:48)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:245)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
 at org.jboss.system.server.jmx.LazyMBeanServer.invoke(LazyMBeanServer.java:225)
 at com.avaya.ccr.reportingtools.realtime.server.ReportingToolsServiceManagerLCM.invoke(ReportingToolsServiceManagerLCM.java:240)
 at com.avaya.ccr.reportingtools.realtime.server.ReportingToolsServiceManagerLCM.start(ReportingToolsServiceManagerLCM.java:63)
 at com.avaya.coreservice.admin.pe.PEGenericJmx.start(PEGenericJmx.java:119)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
 at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
 at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
 at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:245)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
 at com.avaya.coreservice.lifecycle.jmx.InvokeTask.call(InvokeTask.java:66)
 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
 at java.util.concurrent.FutureTask.run(Unknown Source)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)

Prior to this exception, there are several other instances of this same class (JmxUtil) being loaded successfully. Immediately prior to this attempt to load the class I see an attempt to load javax.activation.SecuritySupport, which doesn't finish until after the JmxUtil class loading attempt fails. Not sure if this is significant or not.

I did find a bug report on source forge that seems similar, but that thread makes it sound like this was fixed a while ago (2003) - http://www.mail-archive.com/[e-mail]/msg33888.html

Even though I have more info, I'm still no closer to figuring out what may be going wrong. Any help is greatly appreciated.

Thanks,
Rich

[youngkin]

In the words of the immortal Frank Zappa - "Hi, it's me again. I'm back" :>

More investigation indicates that one cause of this problem is that a deadlock was detected between instances of the JBoss classloader or class loading tasks. This is based the JUnit test implemented in org.jboss.test.classloader.circularity.test.DeadlockTests3.java.

I'm curious if anyone knows the answer to the following questions:
- How is the deadlock detected (e.g., timeout, cycle detection)?
- Are there causes other than deadlock (e.g., if deadlock is detected by a
timeout, maybe it's possible that the timeout could be triggered by a slow
system)?

Thanks again,
Rich