[kabirkhan]

If deploying an ear, I need to find the war files. I think the precense of a JBossWebMetaData attachment is the best way to determine if we are a war?

Yes. What is special about wars for the aspect deployer? I'm still trying to understand how the JBossWebMetaData is coming into the picture and causing problems now, but not before the war class loader change. Note that you can ask about whether an attachment is present using its name as DeploymentUnit.isAttachmentPresent(Class type) is just a convenience method for isAttachmentPresent(type.getName()).

Basically, it needs to be able to understand that the war entries should be part of a sub domain with their own class pool. I only started using JBossWebMetaData locally today to figure out whether the sub unit is a jar or not, so please ignore the talk about that. isAttachmentPresent(String name) should fix that for me.

Next, I need to check if the war classloader delegates to the parent or not. I will try looking for that info in the in ClassLoadingMetaData of the war's sub deployment.

Ok, this may not be in synch with the new org.jboss.deployers.structure.spi.classloading.ClassLoaderMetaData attachment. I'll have to check that.

What is the code your working on here so I can understand what type of class loader processing your doing?

I want to know if we are using j2seClassLoadingCompilance or not, to be able to construct the new ScopedClassPools.

[starksm64]

Basically, it needs to be able to understand that the war entries should be part of a sub domain with their own class pool. I only started using JBossWebMetaData locally today to figure out whether the sub unit is a jar or not, so please ignore the talk about that. isAttachmentPresent(String name) should fix that for me.
...
I want to know if we are using j2seClassLoadingCompilance or not, to be able to construct the new ScopedClassPools.

Ok, this should be defined in the org.jboss.deployers.structure.spi.classloading.ClassLoaderMetaData, but I need to check that its correct. There is a bit of a disconnect between the war class loading deployer still having these settings, a jboss-web.xml having these settings for the legacy LoaderRepository, and the war deployers also trying to use their defaults.

Once we have the org.jboss.deployers.structure.spi.classloading.ClassLoaderMetaData consistently defining the class loader for a deployment unit, it seems that you should not care about wars specifically. The ScopedClassPools should be driven off the unit ClassLoaderMetaData.isJ2seClassLoadingCompliance() setting regardless of the deployment unit type.

[starksm64]

The war ClassLoaderMetaData.isJ2seClassLoadingCompliance is false by default which is correct for wars, so in between the isAttachmentPresent call and checking ClassLoaderMetaData.isJ2seClassLoadingCompliance you should be correct for the standard wars.

Is it the VFSClassLoaderScopingPolicy your working on?

[kabirkhan]

Is it the VFSClassLoaderScopingPolicy your working on?

Yes

[brian.stansberry]

The recent changes fixed a bunch of the FIELD tests. Still a lot of failures though. I'm researching, but FYI here's what I see.

The setSession.jsp call fails with this:

2008-02-06 11:07:38,354 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/http-field].[jsp]] Servlet.service() for servlet jsp threw exception
java.lang.NullPointerException
 at org.jboss.test.cluster.web.aop.Person.name_w_$aop(Person.java)
 at org.jboss.test.cluster.web.aop.Person.<init>(Person.java:38)
 at org.apache.jsp.setSession_jsp._jspService(setSession_jsp.java:65)
 at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
 at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:369)
 at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:337)
 at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:266)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
 at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
 at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
 at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
 at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
 at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:189)
 at org.jboss.web.tomcat.service.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:89)
 at org.jboss.web.tomcat.service.session.BatchReplicationClusteredSessionValve.invoke(BatchReplicationClusteredSessionValve.java:102)
 at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:433)
 at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:90)
 at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:96)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
 at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:309)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:601)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:595)

I'm pretty sure this has something to do with the fact that another test has run earlier, org.jboss.test.cluster.defaultcfg.web.field.test.ScopedFieldBasedTestCase. That test basically does the same thing, but includes a loader-repository element in jboss-web.xml. The Person, etc. types used in both tests have the same FQCN.

I'm going to experiment with running one of the failing tests by itself; if it passes but then fails in combination with another, that confirms my theory. I'll also decompile the bytecode enhanced classes to try and get a feel for the cause of the NPE.

[brian.stansberry]

It's definitely related to multiple tests. Just running FieldBasedTestCase once, it passes. But run it after the FieldBasedSessionPassivationTestCase w/o restarting the servers, and problems appear. The testsuite runs ScopedFieldBasedTestCase in between; that doesn't cause any issue.

[brian.stansberry]

Problem seems to arise from the FieldBasedSessionPassivationTestCase.testRedeploy() test, which deploys a war, undeploys it and deploys it again. Trying to store a Person object in PojoCache fails after the redeploy:

2008-02-06 14:07:02,175 ERROR [org.apache.catalina.connector.CoyoteAdapter] An exception or error occurred in the container during the request processing
java.lang.NullPointerException
 at org.jboss.test.cluster.web.aop.Person._getInstanceAdvisor(Person.java)
 at org.jboss.cache.pojo.impl.AdvisedPojoHandler.get(AdvisedPojoHandler.java:60)
 at org.jboss.cache.pojo.impl.PojoCacheDelegate.getObjectInternal(PojoCacheDelegate.java:350)
 at org.jboss.cache.pojo.impl.PojoCacheDelegate.getObject(PojoCacheDelegate.java:102)
 at org.jboss.cache.pojo.impl.PojoCacheImpl.getObject(PojoCacheImpl.java:207)
 at org.jboss.cache.pojo.impl.PojoCacheImpl.getObject(PojoCacheImpl.java:202)
 at org.jboss.cache.pojo.impl.PojoCacheImpl.org$jboss$cache$pojo$impl$PojoCacheImpl$find$aop(PojoCacheImpl.java:192)
 at org.jboss.cache.pojo.impl.PojoCacheImpl$PojoCacheImplAdvisor.find_N_7063709169143275953(PojoCacheImpl$PojoCacheImplAdvisor.java)
 at org.jboss.cache.pojo.impl.PojoCacheImpl.find(PojoCacheImpl.java)
 at org.jboss.cache.pojo.impl.PojoCacheImpl.find(PojoCacheImpl.java:184)
 at org.jboss.web.tomcat.service.session.JBossCacheService.getPojo(JBossCacheService.java:760)
 at org.jboss.web.tomcat.service.session.FieldBasedClusteredSession.populateAttributes(FieldBasedClusteredSession.java:179)
 at org.jboss.web.tomcat.service.session.JBossCacheClusteredSession.initAfterLoad(JBossCacheClusteredSession.java:67)
 at org.jboss.web.tomcat.service.session.JBossCacheManager.loadSession(JBossCacheManager.java:1224)
 at org.jboss.web.tomcat.service.session.JBossCacheManager.findSession(JBossCacheManager.java:933)
 at org.apache.catalina.connector.Request.doGetSession(Request.java:2306)
 at org.apache.catalina.connector.Request.getSessionInternal(Request.java:2225)
 at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:394)
 at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:90)
 at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:96)
 at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
 at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
 at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
 at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
 at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:309)
 at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
 at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:601)
 at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
 at java.lang.Thread.run(Thread.java:595)

Trying to track down what's happening here in a debugger, I see that the Person class being manipulated is the old class from before the redeploy.

[brian.stansberry]

Enjoying talking to myself. ;)

This looks like another variant on the problem discussed at
http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4126757#4126757 .

FieldBasedSessionPassivationTestCase.testRedeploy() tests redeploying a war and still being able to access sessions. The sessions are passivated to disk on undeploy and after redeploy can be activated.

Problem is JBC isn't handling the classloader very well during activation. It has logic to determine the correct classloader to use for deserialization based on the FQN of the node being deserialized. But that is broken in the case of deserializing passivated nodes. The effect is that JBC is using the cache's classloader to deserialize rather than the TCCL. This results in hitting the internal JVM caching problem Adrian was talking about on the other thread.

I don't really want to support retrieving sessions from disk following a cluster restart anyway, so this test is testing something that won't exist in CR1. But I don't want to disable the test now since it has highlighted an important problem. I'll see if I can rearrange how test execution happens so this failure doesn't cause 30+ other tests to fail -- one failure is enough.

I'll add a test to the JBC testsuite for the underlying issue, and for CR1 will add a couple general classloading tests to the AS to test what happens when the first usage of a type following a redeploy is via deserialization.

[brian.stansberry]

The JBC issue s/b fixed in JBC's trunk, but it turns out it's not the core issue. It causes one failure, but not the others.

Core issue is classloaders are not getting cleaned up on undeploy. The old classloader is still registered in BaseClassLoaderDomain; when the same types are deployed in another tests requests come in to load them, the old classloader is picked to load them.

Only place I see classloaders being cleaned up in BaseClassLoaderDomain is in unregisterClassLoader(BaseClassLoader). I put a breakpoint in there and undeployed a war; the method was never called.

[brian.stansberry]

I found why BaseClassLoaderDomain.unregisterClassLoader() is never called. It's because WarClassLoaderDeployer overrides the superclass removeClassLoader(DeploymentContext) and turns it into a no-op.

Tried commenting out the override; i.e. just using the superclass behavior. No joy. Didn't seem to break war undeployments in any way, and in a debugger I can see the BaseClassLoaderDomain.unregisterClassLoader() stuff happening nicely. But if I redeploy the war and try to use it, the old invalid classloader still gets used:

2008-02-08 17:00:35,073 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/http-scoped-field].[jsp]] Servlet.service() for servlet jsp threw exception
java.lang.IllegalStateException: BaseClassLoader@17ebc6c{vfsfile:/home/bes/dev/jboss/trunk/testsuite/output/lib/http-field-pass.war} classLoader is not connected to a domain (probably undeployed?) for class sun.reflect.ConstructorAccessorImpl
 at org.jboss.classloader.spi.base.BaseClassLoader.loadClassFromDomain(BaseClassLoader.java:579)
 at org.jboss.classloader.spi.base.BaseClassLoader.loadClass(BaseClassLoader.java:234)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:299)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
 at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
 at sun.misc.Unsafe.defineClass(Native Method)
 at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:45)
 at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377)
 at sun.reflect.MethodAccessorGenerator.generateConstructor(MethodAccessorGenerator.java:76)
 at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:30)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
 at java.lang.Class.newInstance0(Class.java:350)
 at java.lang.Class.newInstance(Class.java:303)
 at org.jboss.aop.proxy.ClassProxyFactory.newInstance(ClassProxyFactory.java:103)
 at org.jboss.aop.proxy.ClassProxyFactory.newInstance(ClassProxyFactory.java:71)
 at org.jboss.aop.proxy.ClassProxyFactory.newInstance(ClassProxyFactory.java:66)
 at org.jboss.cache.pojo.collection.CollectionInterceptorUtil.createProxy(CollectionInterceptorUtil.java:50)
....