-
15. Re: Classloader problem with NamingRestartUnitTestCase
kabirkhan Feb 6, 2008 6:28 AM (in response to brian.stansberry)if (result != null) 220 { 221 // Has this classloader been undeployed? 222 ClassLoader otherClassLoader = getClassLoader(result); 223 if (otherClassLoader != null && otherClassLoader != this && otherClassLoader instanceof RealClassLoader) 224 { 225 RealClassLoader rcl = (RealClassLoader) otherClassLoader; 226 // Ignore when undeployed 227 if (rcl.isValid() == false) 228 { 229 if (trace) 230 log.trace(this + " ignoring already loaded class from undeployed classloader " + ClassLoaderUtils.classToString(result)); 231 result = null; 232 } 233 } 234 }
The call on 227 ends up in BaseClassLoaderpublic boolean isValid() { BaseClassLoaderPolicy basePolicy = policy; return basePolicy.getClassLoader() != null; }
which in turn ends up in BaseClassLoaderPolicysynchronized BaseClassLoader getClassLoader() { if (classLoader == null) throw new IllegalStateException("No classloader associated with policy therefore it is no longer registered " + toLongString()); return classLoader; }
So the isValid() check will throw an IllegalStateException rather than return false -
16. Re: Classloader problem with NamingRestartUnitTestCase
kabirkhan Feb 6, 2008 6:31 AM (in response to brian.stansberry)Doh! Ignore my last post, I see the impl of isValid is now
public boolean isValid() { BaseClassLoaderPolicy basePolicy = policy; return basePolicy.getClassLoaderUnchecked() != null; }
So false will be returned -
17. Re: Classloader problem with NamingRestartUnitTestCase
kabirkhan Feb 6, 2008 8:55 AM (in response to brian.stansberry)"bstansberry@jboss.com" wrote:
Is this something we want to do for Beta4?
I have added Adrian's suggested fix to trunk. This was also needed for the MicrocontainerJMXUnitTestCase which also does a redeploy. http://jira.jboss.com/jira/browse/JBAS-5215 -
18. Re: Classloader problem with NamingRestartUnitTestCase
dimitris Feb 7, 2008 12:27 PM (in response to brian.stansberry)After the change was applied, I now see iiop tests failing:
http://hudson.qa.jboss.com/hudson/view/JBoss%20AS/job/JBoss-AS-5.0.x-TestSuite-sun15/366/
http://hudson.qa.jboss.com/hudson/view/JBoss%20AS/job/JBoss-AS-5.0.x-TestSuite-sun15/366/testReport/
Again the weird thing is they pass locally. -
19. Re: Classloader problem with NamingRestartUnitTestCase
dimitris Feb 7, 2008 12:29 PM (in response to brian.stansberry)The server log:
11:41:17,992 INFO [bank-iiop/Customer] Home IOR for Customer bound to bank-iiop/Customer in CORBA naming service 11:41:18,870 ERROR [WebCL] failed finding class org.jboss.test.bankiiop.interfaces._AccountHome_Stub org.jboss.util.NestedRuntimeException: - nested throwable: (java.lang.ExceptionInInitializerError) at org.jboss.iiop.WebCL.findClass(WebCL.java:124) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at org.jboss.classloading.spi.DelegatingClassLoader.loadClass(DelegatingClassLoader.java:81) at java.lang.ClassLoader.loadClass(ClassLoader.java:251) at org.jboss.web.WebServer.run(WebServer.java:379) at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) at java.lang.Thread.run(Thread.java:595) Caused by: java.lang.ExceptionInInitializerError at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)11:41:17,981 INFO [bank-iiop/Customer] Home IOR for Customer bound to iiop/bank-iiop/Customer in JNP naming service at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:494) at java.lang.Class.newInstance0(Class.java:350) at java.lang.Class.newInstance(Class.java:303) at org.jboss.iiop.WebCL.findClass(WebCL.java:120) ... 8 more Caused by: java.lang.IllegalStateException: BaseClassLoader@a8c265{vfsfile:/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15/trunk/testsuite/output/lib/naming-restart.sar} classLoader is not connected to a domain (probably undeployed?) for class javax.ejb.EJBMetaData at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:579) at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:234) at java.lang.ClassLoader.loadClass(ClassLoader.java:251) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:242) at org.jboss.iiop.rmi.marshal.CDRStream.readerFor(CDRStream.java:183) at org.jboss.iiop.rmi.marshal.strategy.StubStrategy.<init>(StubStrategy.java:175) at org.jboss.iiop.rmi.marshal.strategy.StubStrategy.forMethod(StubStrategy.java:115) at org.jboss.test.bankiiop.interfaces._AccountHome_Stub.$i1(Unknown Source) at org.jboss.test.bankiiop.interfaces._AccountHome_Stub.<clinit>(Unknown Source) ... 15 more 11:41:19,233 INFO [EjbModule] Undeployed Customer 11:41:19,234 INFO [EjbModule] Undeployed Teller
-
20. Re: Classloader problem with NamingRestartUnitTestCase
brian.stansberry Feb 7, 2008 1:17 PM (in response to brian.stansberry)"dimitris@jboss.org" wrote:
Again the weird thing is they pass locally.
When they pass locally, do you mean when you just run the one test, e.g. via one-test?
These problems relate to classes from undeployed classloaders being picked up when the same artifact is deployed again. I'm guessing what artifact this test deploys (or at least the classes in it) has been deployed/undeployed by an earlier test. If you just run the BankStressTestCase by itself you won't see the problem. -
21. Re: Classloader problem with NamingRestartUnitTestCase
dimitris Feb 7, 2008 1:40 PM (in response to brian.stansberry)Yes, they pass, e.g. with 'build tests-iiop', even if I run them multiple times.
But it fails on the full testsuite run on Hudson.
If you look at the stacktrace, its asking the wrong classloader that points to 'naming-restart.sar' which is unreleated with the test. -
22. Re: Classloader problem with NamingRestartUnitTestCase
dimitris Feb 7, 2008 1:45 PM (in response to brian.stansberry)So if I run in between:
build one-test -Dtest=org.jboss.test.naming.test.NamingRestartUnitTestCase
...the iiop-tests fail afterwards :-) -
23. Re: Classloader problem with NamingRestartUnitTestCase
brian.stansberry Feb 7, 2008 3:14 PM (in response to brian.stansberry)I need to understand why native method Class.forName0() ends up calling into the old BaseClassLoader for the naming-restart sar. First thing I want to do is see what the TCCL is when WebCL is trying to create the instance.
-
24. Re: Classloader problem with NamingRestartUnitTestCase
brian.stansberry Feb 7, 2008 6:47 PM (in response to brian.stansberry)Just an update since I've been silent for so long.
Something is setting the TCCL on some of the JBoss System thread pool threads to the naming-restart.sar CL. Trying to figure out who. Needle in haystack. -
25. Re: Classloader problem with NamingRestartUnitTestCase
brian.stansberry Feb 7, 2008 8:27 PM (in response to brian.stansberry)Needle found after sifting a lot of hay. http://jira.jboss.org/jira/browse/JBAS-5222
-
26. Re: Classloader problem with NamingRestartUnitTestCase
starksm64 Feb 7, 2008 8:50 PM (in response to brian.stansberry)but it should be the Runnable of the thread that sets/clears the TCCL. What is using the pool and not resetting the TCCL? It might be good to limit the duration of the TCCL in the pool, but the task that is using the new thread can't have the right TCCL if I understand correctly.
-
27. Re: Classloader problem with NamingRestartUnitTestCase
brian.stansberry Feb 7, 2008 9:22 PM (in response to brian.stansberry)It's org.jnp.server.Main and org.jboss.ha.jndi.DetachedHANamingService. Both take a couple threads from the pool and use them to handle incoming connections. They do nothing with the TCCL. That doesn't cause a problem with the real services, because 1) they are started by the main thread, so the TCCL they "leak" is (I think) the conf/jboss-service.xml classloader and 2) they hold onto their pool threads as long as they are running, i.e. pretty much the life of the server.
The problem appears with NamingRestartUnitTestCase which deploys parallel naming services and restarts them numerous times.
The testsuite problem should be fixed. I changed the naming-restart-beans.xml so it uses its own thread pool. It no longer leaks its classloader to the regular pool.the task that is using the new thread can't have the right TCCL if I understand correctly.
Correct; it's indeterminate. The ThreadPoolExecutor creates a new thread when it needs one. The new thread picks up the TCCL of the thread that called BasicThreadPool.run(). That's probably the right one for that first task, but thereafter it isn't. So, yeah, a well written task will set the right TCCL and restore at the end. The naming services are not well written in this respect. But restoring the TCCL of a random first task does little good. Only the thread pool itself has the knowledge to restore a meaningful default TCCL (i.e. null or its own classloader). -
28. Re: Classloader problem with NamingRestartUnitTestCase
brian.stansberry Feb 7, 2008 9:41 PM (in response to brian.stansberry)org.jboss.web.WebServer seems broken in this regard as well. It's run() method just accepts whatever TCCL is associated with the thread pool thread. That's how the leaked naming-restart CL was able to break the iiop tests.
-
29. Re: Classloader problem with NamingRestartUnitTestCase
starksm64 Feb 7, 2008 9:58 PM (in response to brian.stansberry)Its not really the WebServer's fault though as it only uses the TCCL if it could not find a ClassLoader registered based on the incoming request information that should tie back to a previous addClassLoader call. The ejb containers use this call to get the codebase for remote class loading. This should only be done for downloadServerClasses=true, which really should not be the default setting. I have changed this in the conf/jboss-service.xml. Now, the attempt to load the class will just fail if the TCCL would be used.