EJB3 Classloader Leak (EJBTHREE-1442)

[alrubinger]

I've moved this topic away from http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4175163 as the culprit is no longer the proxy factories.

I've found that the remaining transient failures may be linked to Ejb3JBoss5Deployment instances that are not cleaned up. Profiler image tracking these instances is attached to https://jira.jboss.org/jira/browse/EJBTHREE-1442

What's revealed is that 2 instances of Ejb3JBoss5Deployment (with references to the CL) remain upon test completion, and are GC'd if requested by the Profiler.

So the fun part will be to find out why.

S,
ALR

[skajotde]

Sorry for unrelated qustion: Which soft did you use to that monitoring https://jira.jboss.org/jira/secure/attachment/12322115/Ejb3JBoss5Deployment-allocation.png in screenshot in EJBTHREE-1442 ? ;>

[alrubinger]

This is *kind of* a dev question. :)

I'll answer this because it promotes some great software generously provided to us for free via an open source license:

JProfiler 5.1.4 from ej-technologies, http://www.jprofiler.com

Note: I added all the red text afterwards.

S,
ALR

[alrubinger]

I'm also seeing that the delta of instance allocations to AbstractVFSDeploymentUnit is non-0 after running the tests..

[alrubinger]

Changing directions, this seems a bit more likely:

java.lang.Thread has a field "contextClassLoader". Remoting ServerThread inherits this, and the ServerThread hangs around after invocations (until timeout).

So if the BaseClassLoader instance in there is the one we're leaking, that could be it. Continuing to verify.

S,
ALR

[alrubinger]

Some more circumstantial evidence that the leak stems from Remoting ServerThread:

* When commenting out the execution of "testSimpleEjb", which fails, the next test in line, "testSimpleEar", fails where it previously passed.
* Subsequent invocations of the tests all pass after the first (failed) run.

My working theory is that the EJB3 CL is the TCL when the ServerThread is constructed, and therefore remains throughout the life of the ServerThread.

Will be posting this to jboss-dev for some wider opinions on proper resolution.

S,
ALR

[anil.saldhana]

Maybe better to just get hold of the Remoting sources, make the changes you wish and test if the leak persists.

[alrubinger]

Yeah, I'm doing that now and trying to get a version compatible. Currently it's building me "3.0.0.Alpha1", even from https://svn.jboss.org/repos/jbossremoting/remoting2/trunk

S,
ALR

[brian.stansberry]

From a JBoss Profiler analysis of this:

!--org.jboss.classloader.spi.base.BaseClassLoader@166889468(BaseClassLoader@9f287fc{vfszip:/home/bes/dev/jboss/clean/trunk/testsuite/output/lib/classloader-leak-ejb3.jar})
!--!--ClassLoaderReference @ java.lang.Class@161343925(interface org.jboss.test.classloader.leak.ejb3.Ejb3StatelessSession)
!--!--![e-mail]@370564132(class $Proxy177)
!--!--!--!--arrayRef [Ljava.lang.Object;[0] id=@1684924387
!--!--!--!--!--FieldReference protected java.lang.Object[] java.util.Vector.elementData=java.util.Vector@1804802967([class $Proxy177, class $Proxy178])
!--!--!--!--!--!--FieldReference private java.util.Vector java.lang.ClassLoader.classes=org.jboss.classloader.spi.base.BaseClassLoader@1648466027(BaseClassLoader@6241986b{vfsfile:/home/bes/dev/jboss/clean/trunk/build/output/jboss-5.0.0.CR2/server/all/deploy/ejb3-connectors-jboss-beans.xml})
!--!--!--!--!--!--!--FieldReference private java.lang.ClassLoader org.jboss.classloading.spi.dependency.policy.ClassLoaderPolicyModule.classLoader=org.jboss.deployers.vfs.plugins.classloader.VFSDeploymentClassLoaderPolicyModule@116952494(VFSDeploymentClassLoaderPolicyModule ejb3-connectors-jboss-beans.xml:0.0.0)

... carries on into the default classloading domain.


Looks like a proxy class loaded by the ejb3-connectors-jboss-beans.xml classloader is holding a ref to the org.jboss.test.classloader.leak.ejb3.Ejb3StatelessSession loaded from the ejb deployment. I think this is a manifestation of the problem in SessionProxyFactoryBase discussed at http://www.jboss.com/index.html?module=bb&op=viewtopic&t=141165. I bet if you limit the creation of an alternative proxy to cases where the caller classloader's version of the business interface is distinct from the standard one, this leak will go away.

[alrubinger]

Thanks for your continued help, Brian.

I'd tried (unsuccessfully) to limit the alternate proxy creation before, which resulted in errors in web injection. But probably using the TCCL ain't great either; I'll give it another spin.

S,
ALR

[alrubinger]

Which brings up another question:

Why would one of the Proxies still be hanging around?

S,
ALR

[dmlloyd]

"ALRubinger" wrote:
Yeah, I'm doing that now and trying to get a version compatible. Currently it's building me "3.0.0.Alpha1", even from https://svn.jboss.org/repos/jbossremoting/remoting2/trunk

S,
ALR

Yeah the Remoting 2 trunk is defunct. You should use the https://svn.jboss.org/repos/jbossremoting/remoting2/branches/2.x branch.

[brian.stansberry]

"ALRubinger" wrote:
Which brings up another question:

Why would one of the Proxies still be hanging around?

It's not the proxy hanging around, it's the proxy's class. A classloader maintains a ref to all classes it has defined; the ejb3-connectors-jboss-beans.xml classloader defined that proxy class. But the business interface came from the deployment CL, since classes loaded by the deployment CL are visible to the ejb3-connectors-jboss-beans.xml classloader. The proxy's class has a ref to the business interface class.

[alrubinger]

So:

ejb3-connectors-jboss-beans.xml CL > Proxy Class > Business Interface Class > Deployment CL

?

...and that's how one CL creates a chain of linked references to the EJB_SLSB CL?

S,
ALR

[alrubinger]

Regarding the Remoting ServerThread I mentioned earlier:

It has a reference to the TCL that defined it, but it's the CL of ejb3-connectors-jboss-beans.xml.

S,
ALR