1 2 3 Previous Next 32 Replies Latest reply on Feb 8, 2008 10:28 AM by starksm64 Go to original post
      • 15. Re: Classloader problem with NamingRestartUnitTestCase
        kabirkhan

        Looking at
        http://viewvc.jboss.org/cgi-bin/viewvc.cgi/jbossas/projects/microcontainer/trunk/classloader/src/main/org/jboss/classloader/spi/base/BaseClassLoader.java/?revision=69628&r1=69628&r2=69627

         if (result != null)
        220 {
        221 // Has this classloader been undeployed?
        222 ClassLoader otherClassLoader = getClassLoader(result);
        223 if (otherClassLoader != null && otherClassLoader != this && otherClassLoader instanceof RealClassLoader)
        224 {
        225 RealClassLoader rcl = (RealClassLoader) otherClassLoader;
        226 // Ignore when undeployed
        227 if (rcl.isValid() == false)
        228 {
        229 if (trace)
        230 log.trace(this + " ignoring already loaded class from undeployed classloader " + ClassLoaderUtils.classToString(result));
        231 result = null;
        232 }
        233 }
        234 }
        


        The call on 227 ends up in BaseClassLoader
         public boolean isValid()
         {
         BaseClassLoaderPolicy basePolicy = policy;
         return basePolicy.getClassLoader() != null;
         }
        


        which in turn ends up in BaseClassLoaderPolicy
         synchronized BaseClassLoader getClassLoader()
         {
         if (classLoader == null)
         throw new IllegalStateException("No classloader associated with policy therefore it is no longer registered " + toLongString());
         return classLoader;
         }
        


        So the isValid() check will throw an IllegalStateException rather than return false

        • 16. Re: Classloader problem with NamingRestartUnitTestCase
          kabirkhan

          Doh! Ignore my last post, I see the impl of isValid is now

           public boolean isValid()
           {
           BaseClassLoaderPolicy basePolicy = policy;
           return basePolicy.getClassLoaderUnchecked() != null;
           }
          

          So false will be returned

          • 17. Re: Classloader problem with NamingRestartUnitTestCase
            kabirkhan

             

            "bstansberry@jboss.com" wrote:
            Is this something we want to do for Beta4?


            I have added Adrian's suggested fix to trunk. This was also needed for the MicrocontainerJMXUnitTestCase which also does a redeploy. http://jira.jboss.com/jira/browse/JBAS-5215


            • 18. Re: Classloader problem with NamingRestartUnitTestCase
              dimitris
              • 19. Re: Classloader problem with NamingRestartUnitTestCase
                dimitris

                The server log:

                
                11:41:17,992 INFO [bank-iiop/Customer] Home IOR for Customer bound to bank-iiop/Customer in CORBA naming service
                11:41:18,870 ERROR [WebCL] failed finding class org.jboss.test.bankiiop.interfaces._AccountHome_Stub
                org.jboss.util.NestedRuntimeException: - nested throwable: (java.lang.ExceptionInInitializerError)
                 at org.jboss.iiop.WebCL.findClass(WebCL.java:124)
                 at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
                 at org.jboss.classloading.spi.DelegatingClassLoader.loadClass(DelegatingClassLoader.java:81)
                 at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
                 at org.jboss.web.WebServer.run(WebServer.java:379)
                 at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
                 at java.lang.Thread.run(Thread.java:595)
                Caused by: java.lang.ExceptionInInitializerError
                 at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)11:41:17,981 INFO [bank-iiop/Customer] Home IOR for Customer bound to iiop/bank-iiop/Customer in JNP naming service
                 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
                 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
                 at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
                 at java.lang.Class.newInstance0(Class.java:350)
                 at java.lang.Class.newInstance(Class.java:303)
                 at org.jboss.iiop.WebCL.findClass(WebCL.java:120)
                 ... 8 more
                Caused by: java.lang.IllegalStateException: BaseClassLoader@a8c265{vfsfile:/qa/services/hudson/hudson_workspace/workspace/JBoss-AS-5.0.x-TestSuite-sun15/trunk/testsuite/output/lib/naming-restart.sar} classLoader is not connected to a domain (probably undeployed?) for class javax.ejb.EJBMetaData
                 at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:579)
                 at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:234)
                 at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
                 at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
                 at java.lang.Class.forName0(Native Method)
                 at java.lang.Class.forName(Class.java:242)
                 at org.jboss.iiop.rmi.marshal.CDRStream.readerFor(CDRStream.java:183)
                 at org.jboss.iiop.rmi.marshal.strategy.StubStrategy.<init>(StubStrategy.java:175)
                 at org.jboss.iiop.rmi.marshal.strategy.StubStrategy.forMethod(StubStrategy.java:115)
                 at org.jboss.test.bankiiop.interfaces._AccountHome_Stub.$i1(Unknown Source)
                 at org.jboss.test.bankiiop.interfaces._AccountHome_Stub.<clinit>(Unknown Source)
                 ... 15 more
                11:41:19,233 INFO [EjbModule] Undeployed Customer
                11:41:19,234 INFO [EjbModule] Undeployed Teller
                


                • 20. Re: Classloader problem with NamingRestartUnitTestCase
                  brian.stansberry

                   

                  "dimitris@jboss.org" wrote:
                  Again the weird thing is they pass locally.


                  When they pass locally, do you mean when you just run the one test, e.g. via one-test?

                  These problems relate to classes from undeployed classloaders being picked up when the same artifact is deployed again. I'm guessing what artifact this test deploys (or at least the classes in it) has been deployed/undeployed by an earlier test. If you just run the BankStressTestCase by itself you won't see the problem.

                  • 21. Re: Classloader problem with NamingRestartUnitTestCase
                    dimitris

                    Yes, they pass, e.g. with 'build tests-iiop', even if I run them multiple times.

                    But it fails on the full testsuite run on Hudson.

                    If you look at the stacktrace, its asking the wrong classloader that points to 'naming-restart.sar' which is unreleated with the test.

                    • 22. Re: Classloader problem with NamingRestartUnitTestCase
                      dimitris

                      So if I run in between:

                      build one-test -Dtest=org.jboss.test.naming.test.NamingRestartUnitTestCase
                      

                      ...the iiop-tests fail afterwards :-)

                      • 23. Re: Classloader problem with NamingRestartUnitTestCase
                        brian.stansberry

                        I need to understand why native method Class.forName0() ends up calling into the old BaseClassLoader for the naming-restart sar. First thing I want to do is see what the TCCL is when WebCL is trying to create the instance.

                        • 24. Re: Classloader problem with NamingRestartUnitTestCase
                          brian.stansberry

                          Just an update since I've been silent for so long.

                          Something is setting the TCCL on some of the JBoss System thread pool threads to the naming-restart.sar CL. Trying to figure out who. Needle in haystack.

                          • 25. Re: Classloader problem with NamingRestartUnitTestCase
                            brian.stansberry

                            Needle found after sifting a lot of hay. http://jira.jboss.org/jira/browse/JBAS-5222

                            • 26. Re: Classloader problem with NamingRestartUnitTestCase
                              starksm64

                              but it should be the Runnable of the thread that sets/clears the TCCL. What is using the pool and not resetting the TCCL? It might be good to limit the duration of the TCCL in the pool, but the task that is using the new thread can't have the right TCCL if I understand correctly.

                              • 27. Re: Classloader problem with NamingRestartUnitTestCase
                                brian.stansberry

                                It's org.jnp.server.Main and org.jboss.ha.jndi.DetachedHANamingService. Both take a couple threads from the pool and use them to handle incoming connections. They do nothing with the TCCL. That doesn't cause a problem with the real services, because 1) they are started by the main thread, so the TCCL they "leak" is (I think) the conf/jboss-service.xml classloader and 2) they hold onto their pool threads as long as they are running, i.e. pretty much the life of the server.

                                The problem appears with NamingRestartUnitTestCase which deploys parallel naming services and restarts them numerous times.

                                The testsuite problem should be fixed. I changed the naming-restart-beans.xml so it uses its own thread pool. It no longer leaks its classloader to the regular pool.

                                the task that is using the new thread can't have the right TCCL if I understand correctly.


                                Correct; it's indeterminate. The ThreadPoolExecutor creates a new thread when it needs one. The new thread picks up the TCCL of the thread that called BasicThreadPool.run(). That's probably the right one for that first task, but thereafter it isn't. So, yeah, a well written task will set the right TCCL and restore at the end. The naming services are not well written in this respect. But restoring the TCCL of a random first task does little good. Only the thread pool itself has the knowledge to restore a meaningful default TCCL (i.e. null or its own classloader).

                                • 28. Re: Classloader problem with NamingRestartUnitTestCase
                                  brian.stansberry

                                  org.jboss.web.WebServer seems broken in this regard as well. It's run() method just accepts whatever TCCL is associated with the thread pool thread. That's how the leaked naming-restart CL was able to break the iiop tests.

                                  • 29. Re: Classloader problem with NamingRestartUnitTestCase
                                    starksm64

                                    Its not really the WebServer's fault though as it only uses the TCCL if it could not find a ClassLoader registered based on the incoming request information that should tie back to a previous addClassLoader call. The ejb containers use this call to get the codebase for remote class loading. This should only be done for downloadServerClasses=true, which really should not be the default setting. I have changed this in the conf/jboss-service.xml. Now, the attempt to load the class will just fail if the TCCL would be used.