1 2 Previous Next 18 Replies Latest reply on May 2, 2008 2:15 PM by brian.stansberry

    Classloading problems with ClassProxyFactory

    brian.stansberry

      I'm seeing intermittent failures in the AS clustering testsuite tests of replicating aspectized web session attributes (aka FIELD granularity session replication.) These are tests where classes are prepared with aopc for use with PojoCache, and then inserted into PojoCache via a web session.

      Problem is one of stale (i.e. undeployed) classloaders being used by ClassProxyFactory to create proxies. Specifically:

      1) Test class A deploys http-scoped-field.war, classloader BaseClassLoader@17eb527{vfsfile:/home/bes/dev/jboss/trunk/testsuite/output/lib/http-scoped-field.war} is created.
      2) During test, PojoCache uses ClassProxyFactory to create a proxy of java.util.ArrayList. TCCL in effect is the "http-scoped-field.war" classloader, so that is the classloader used to define the proxy class.
      3) Test class A finishes, http-scoped-field.war undeploys; all refs to "http-scoped-field.war" classloader should be released.

      4) Test class B deploys different war http-field.war".
      5) Same as step 2, during test PojoCache uses ClassProxyFactory to create a proxy of java.util.ArrayList. Now TCCL in effect is the "http-field.war" classloader,

      Step 5 fails, as ClassProxyFactory finds the cached proxy class from step 2 and tries to create an instance of it:

      2008-04-24 16:19:42,799 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/http-field].[jsp]] (http-besdev%2F192.168.1.145-8080-2) Servlet.service() for servlet jsp threw exception
      java.lang.IllegalStateException: BaseClassLoader@17eb527{vfsfile:/home/bes/dev/jboss/trunk/testsuite/output/lib/http-scoped-field.war} classLoader is not connected to a domain (probably undeployed?) for class sun.reflect.ConstructorAccessorImpl
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:723)
       at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:372)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:299)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
       at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
       at sun.misc.Unsafe.defineClass(Native Method)
       at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:45)
       at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
       at java.security.AccessController.doPrivileged(Native Method)
       at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377)
       at sun.reflect.MethodAccessorGenerator.generateConstructor(MethodAccessorGenerator.java:76)
       at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:30)
       at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
       at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
       at java.lang.Class.newInstance0(Class.java:350)
       at java.lang.Class.newInstance(Class.java:303)
       at org.jboss.aop.proxy.ClassProxyFactory.newInstance(ClassProxyFactory.java:103)
       at org.jboss.aop.proxy.ClassProxyFactory.newInstance(ClassProxyFactory.java:71)
       at org.jboss.aop.proxy.ClassProxyFactory.newInstance(ClassProxyFactory.java:66)
       at org.jboss.cache.pojo.collection.CollectionInterceptorUtil.createProxy(CollectionInterceptorUtil.java:50)
       at org.jboss.cache.pojo.collection.CollectionInterceptorUtil.createListProxy(CollectionInterceptorUtil.java:97)
       at org.jboss.cache.pojo.impl.CollectionClassHandler.get(CollectionClassHandler.java:63)
       at org.jboss.cache.pojo.impl.PojoCacheDelegate.getObjectInternal(PojoCacheDelegate.java:354)
       at org.jboss.cache.pojo.impl.PojoCacheDelegate.getObject(PojoCacheDelegate.java:102)
       at org.jboss.cache.pojo.impl.PojoCacheImpl.getObject(PojoCacheImpl.java:209)
       at org.jboss.cache.pojo.impl.PojoCacheImpl.org$jboss$cache$pojo$impl$PojoCacheImpl$detach$aop(PojoCacheImpl.java:151)
       at org.jboss.cache.pojo.impl.PojoCacheImpl$PojoCacheImplAdvisor.detach_N_6302035201148273652(PojoCacheImpl$PojoCacheImplAdvisor.java)
       at org.jboss.cache.pojo.impl.PojoCacheImpl.detach(PojoCacheImpl.java)
       at org.jboss.cache.pojo.impl.AdvisedPojoHandler.remove(AdvisedPojoHandler.java:182)
       at org.jboss.cache.pojo.impl.PojoCacheDelegate.removeObject(PojoCacheDelegate.java:276)
       at org.jboss.cache.pojo.impl.PojoCacheImpl.removeObject(PojoCacheImpl.java:171)
       at org.jboss.cache.pojo.impl.PojoCacheImpl.org$jboss$cache$pojo$impl$PojoCacheImpl$detach$aop(PojoCacheImpl.java:154)
       at org.jboss.cache.pojo.impl.PojoCacheImpl$PojoCacheImplAdvisor.detach_N_6302035201148273652(PojoCacheImpl$PojoCacheImplAdvisor.java)
       at org.jboss.cache.pojo.impl.PojoCacheImpl.detach(PojoCacheImpl.java)
       at org.jboss.cache.pojo.impl.PojoCacheImpl.detach(PojoCacheImpl.java:165)
       at org.jboss.cache.pojo.impl.PojoCacheImpl.detach(PojoCacheImpl.java:143)
       at org.jboss.web.tomcat.service.session.FieldBasedJBossCacheService.removePojo(FieldBasedJBossCacheService.java:197)
       at org.jboss.web.tomcat.service.session.FieldBasedClusteredSession.removeJBossInternalAttribute(FieldBasedClusteredSession.java:285)
       at org.jboss.web.tomcat.service.session.JBossCacheClusteredSession.removeAttributeInternal(JBossCacheClusteredSession.java:146)
       at org.jboss.web.tomcat.service.session.ClusteredSession.removeAttributeInternal(ClusteredSession.java:1348)
       at org.jboss.web.tomcat.service.session.ClusteredSession.removeAttributeInternal(ClusteredSession.java:1325)
       at org.apache.catalina.session.StandardSession.removeAttribute(StandardSession.java:1207)
       at org.apache.catalina.session.StandardSession.removeAttribute(StandardSession.java:1179)
       at org.apache.catalina.session.StandardSessionFacade.removeAttribute(StandardSessionFacade.java:140)
       at org.apache.jsp.removeAttribute_jsp._jspService(removeAttribute_jsp.java:60)
       at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
       at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:369)
       at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
       at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
       at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:189)
       at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:89)
       at org.jboss.web.tomcat.service.session.BatchReplicationClusteredSessionValve.invoke(BatchReplicationClusteredSessionValve.java:102)
       at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
       at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:90)
       at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:96)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
       at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
       at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
       at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:310)
       at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
       at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:601)
       at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
       at java.lang.Thread.run(Thread.java:595)
      


      I've been searching high and low to try and figure out why the old proxy class is being used, since AOP is using a weak ref to cache it. I don't completely understand this (see below) but here's one thing I've seen:

      MethodHashesByClass.methodHashesByClass
      + WeakHashMap.table
      ++ WeakHashMap$Entry.value
      +++ HashMap.table
      ++++ HashMap$Entry.value
      +++++ MethodPersistentReference.referencedObject
      ++++++ SoftReference.referent
      +++++++ java.lang.reflect.Method.clazz
      ++++++++ class Person.classloader
      +++++++++ "http-scoped-field.war" classloader classes
      ++++++++++ proxy to ArrayList

      A chain from the proxy class running through a SoftReference. Basically, holding a SoftReference to a Method effectively converts the ClassProxyFactory.proxyCache's weak ref to the proxy class into a SoftReference.

      This would help explain the intermittent nature of these failures; if the SoftReference happens to get released, the proxy class is gone and the test passes.


      Now, the caveat: while trying to debug this I've developed some tests that load and instantiate AOP-prepared (via aopc) classes in a war, then undeploy the war, fill memory to force all soft references to be cleared, and see then test if the classloader was collected. These are variants on the tests discussed at http://wiki.jboss.org/auth/wiki/ClassloaderLeakUnitTestCase . These tests show the classloader not being released, but an analysis of the heap shows all refs to the classloader going through weak references -- i.e. the classloader should have been collected.

      If I run the same test with the same classes, except I haven't run aopc first to prepare the classes, the classloader is released. Been tearing my hair out trying to understand this; using different profiling tools to examing heap, etc. So, there's still some mystery here. :(

        • 1. Re: Classloading problems with ClassProxyFactory
          flavia.rainone

          Hi, Brian!

          Are you using Linux? If yes, are you using the Sun's jvm?
          If you have answered yes to both questions, you might be facing a situation similar to the one we are facing with our memory leak tests.
          Some of them fail only on Linux when using Sun's jvm. If we try running them on Windows XP, or on a different jvm (say, JRockit), they don't fail. The situation is very similar to yours: the class loader is never collected and the references to classes are held through weak references only. I haven't managed to figure out why yet.

          • 2. Re: Classloading problems with ClassProxyFactory
            brian.stansberry

            Yep; Fedora 7 and Sun JVM.

            I've got my tests 98% down to a usable-by-others form. I'm going to grab a bite to eat, get them to 100%, check them in, try them on Windows XP and post back.

            • 3. Re: Classloading problems with ClassProxyFactory
              brian.stansberry

              My test fails on Windows XP as well. Not every time, but most.

              FYI, here's how to run that test

              1) Build the AS and its testsuite from trunk.

              2) In on window start the AS with the all config

              3) From the root of the testsuite module:

              ./run.sh -Dtest=org.jboss.test.cluster.classloader.leak.test.ReplicableClassesClassloaderLeakUnitTestCase one-test


              There's 2 methods, one uses aopc-prepared classes, the other doesn't. The former fails. But when I analyze the heap, all I see are chains leading to weak references.

              • 4. Re: Classloading problems with ClassProxyFactory
                flavia.rainone

                Hm... interesting.
                I'll try to run the test locally and see if I can get any further.

                • 5. Re: Classloading problems with ClassProxyFactory
                  kabirkhan

                  Brian,

                  If the


                  MethodHashesByClass.methodHashesByClass
                  + WeakHashMap.table
                  ++ WeakHashMap$Entry.value
                  +++ HashMap.table
                  ++++ HashMap$Entry.value
                  +++++ MethodPersistentReference.referencedObject
                  ++++++ SoftReference.referent
                  +++++++ java.lang.reflect.Method.clazz
                  ++++++++ class Person.classloader
                  +++++++++ "http-scoped-field.war" classloader classes
                  ++++++++++ proxy to ArrayList


                  If MethodHashesByClass refers to org.jboss.aop.util.MethodHashing. Changing the underlying SoftReference to a WeakReference should fix the problem you are seeing? Please confirm and we'll make the change for you

                  • 6. Re: Classloading problems with ClassProxyFactory
                    brian.stansberry

                    Maybe it will fix it. :(

                    When I was analyzing the regular field granularity session replication tests failure shown in my first post, I analyzed the heap and saw that reference chain. So as long as the SoftReference is around, for sure the proxy class won't be released and the tests will fail.

                    But, Flavia says that in some tests you guys are seeing classloaders not being released even though all that's held are weak references. I see the same thing in the classloader leak test described in the last post. If that happens, and the classloader has a ref to the proxy class, then the problem will still exist.

                    • 7. Re: Classloading problems with ClassProxyFactory
                      flavia.rainone

                       

                      But, Flavia says that in some tests you guys are seeing classloaders not being released even though all that's held are weak references. I see the same thing in the classloader leak test described in the last post. If that happens, and the classloader has a ref to the proxy class, then the problem will still exist.


                      I'm hoping this is not the case with your tests, Brian. I have already teared all my hair out with those misterious memory leaks we've been seen, and it would be great if your tests don't make to the list.

                      Anyway, the fact that your tests also fail on Windows is a strong indicator that the problem is elsewhere. So I think Kabir is probably right regarding the soft references.

                      • 8. Re: Classloading problems with ClassProxyFactory
                        kabirkhan

                        I will try a few things today

                        • 9. Re: Classloading problems with ClassProxyFactory
                          kabirkhan

                          I cannot run the test, it freezes, but having looked more at the description, it seems to me that the problem is that the ContainerProxyFactory caches the proxies by the class.

                          In this case the key is java.utl.ArrayList which will never be removed from the cache, but the proxy stored is created using the TCL, and thus never cleared out. I will create a simple test for this under aop and try generating the proxies with the classloader of the class being proxied instead.

                          • 10. Re: Classloading problems with ClassProxyFactory
                            brian.stansberry

                            Cool. That seems a better solution. :) If you want, let me know and I can update my AOP checkout and run the FIELD granularity tests.

                            • 11. Re: Classloading problems with ClassProxyFactory
                              kabirkhan

                              Digging into this I deiscovered that if you create a proxy for something loaded up by a bootstrap classloader (e.g. java.util.ArrayList) the associated advisor is cleared due to JBoss5Integration.isValidClassLoader() returning false .

                              http://jira.jboss.com/jira/browse/JBAS-5495

                              • 12. Re: Classloading problems with ClassProxyFactory
                                kabirkhan

                                My previous comment relates to the fact that once a ClassProxyContainer is created for an ArrayList with no bindings, the ClassProxyContainer would never be updated when new bindings which should affect it are added

                                • 13. Re: Classloading problems with ClassProxyFactory
                                  jason.greene

                                   

                                  "kabir.khan@jboss.com" wrote:

                                  In this case the key is java.utl.ArrayList which will never be removed from the cache, but the proxy stored is created using the TCL, and thus never cleared out. I will create a simple test for this under aop and try generating the proxies with the classloader of the class being proxied instead.


                                  I think this approach of using the source class CL is a good solution. This of course requires that the CL associated with the class can be modified (a common problem with custom CLs). An alternative approach would be to create a small child CL that is associated with the source class CL. This child CL would then only be used to generate proxies. This solves the isolation issues, and yet still allows for the generated proxies to be easily collected, and does not pollute the source CL.

                                  I responded to your jboss-dev post about the bootstrap CL (basically saying its not possible). This can also be solved by using a special proxy CL, which is dedicated to source classes that are on the bootstrap classpath.

                                  • 14. Re: Classloading problems with ClassProxyFactory
                                    kabirkhan

                                    I now have a simple test working that reliably fails, so I can start looking at the fix.

                                    Locally I am getting ok results with the
                                    ClassLoader.getSystemClassLoader().getParent() stuff, although I have not tried calling defineClass() yet on that. Javassist always calls ClassLoader.defineClass() for new classes, no matter what the CL is, and it has worked fine so far. I'll take a look.

                                    Can you please elaborate a bit on the custom Proxy CL stuff? I see it a a subclass of java.lang.ClassLoader that delegates to the parent classloader, but I am not sure how it would get cleaned up?

                                    1 2 Previous Next