Class Leakage Tool

[clebert.suconic]

Last week, as I was involved on finding memory leaks on redeployments, I created some features on the JVMTIWrapper (org.jboss.profiler.jvmti.JVMTIInterface) useful to find leaks specific to redeployment.

Instead of explaining the nature of how a ClassDeployment would leak, I would like to talk about what I've added, and point this thread as a reference on ClassLeakage:

http://www.jboss.com/index.html?module=bb&op=viewtopic&t=78975

I've added two methods on JVMTIInterfaceMBean:

I - listClassesHTMLReport() what will show every class loaded on the current JVM. On the beggining of the output you will see every class loaded in more than one class loader. When you redeploy packages and they are not GCed you will see duplications.

II - exploreClassReferences

 /**
 * @param className The name of the class to explore
 * @param maxLevel The number of levels to explode. Be careful as if you put this number too high, you migh endup in a forever loop, specially if your object is referencing something too generic
 * @param solveReferencesOnClass Will expose the tree on the class
 * @param solveReferencesOnClassLoader Will expode the tree on the classLoader (I mostly recommend to only look for classLoader's references)
 * @param useToString If true, will use toString when an object is printed. If False will use className@<System.identityHashCode(object)>
 * @param weakAndSoft If false, won't detail references on Weak and Soft References
 * @param printObject If true, Will print (with toString) every single instance of the object passed as parameter
 */
 public String exploreClassReferences(String className, int maxLevel, boolean solveReferencesOnClasses, boolean solveReferencesOnClassLoaders, boolean useToString, boolean weakAndSoft, boolean printObjects)

I have also added a new menu item on jboss-profiler.war

- Class Loader Analysis

With this new Tool, you will be able to naviage on references for any Class/ClassLoader inside the JVM where the application is deployed.

Notice that you need to provide extra memory, as this feature will download a hashMap with a matrix between referenceHolders and referencedObjects from JVMTI. I have executed an all instance of JBoss with 400M of memory for this sucessfully. If I've used 128M GC would be so crazy that you would even have problems killing the process.


I wanted at first, to take the ClassLeakage tool outside of JBossProfiler.war, but since this is part of the project, I guess it should be okay to be part of the package.


also, it's important to understand that this tool can only be used on a JVM 5. (needs JVMTI support)


Clebert


PS: As usual my HTML design skills are terrible. I could provide the information, but if someone want to contribute with any fancy HTML, it would be really nice.

[ngolovin]

Hello Clebert, thank you for the great tool.

I have a question -- the number of references is overwhelming,
even with the display limit of 200, so it's hard to find this very bad reference which prevents classloader to be gc-ed. Most of the references to the classloader however comes from its own classes(i.e. loaded by this classloader and its children), and should not be the problem. Is it possible to somehow filter references from the own classes and find that bad references from the container?
Or am I generally misunderstanding something?

[clebert.suconic]

On JBossProfiler.war there is a WebInterface that will automate the navigation on JVMTIInterface.


Go to http://localhost:8080/jboss-profiler
Look at the third option on the Menu.

That will intercept class references in a Tomcat/JBoss instance.

You will need JVMTIInterface (the MBean) deployed as well.

You can redeploy your application couple times, forcing classes being loaded twice, and look for duplications.

You can play with Clean Session and get duplications again every time you execute a redeployment. The application will "download" a matrix from C++layer into Java Layer. If you redeploy the matrix will be out dated.

[ngolovin]

Hi Clebert, either I have expressed myself way too vaguely, or you answer was for another thread.

What you are suggesting is exactly what I did. I saw the matrix and the duplicates. Then I clicked on loader references and saw there approx. 2000 references (I have removed the 200-reference limit in my copy of the references.jsp)
My problem is, I am looking for the reference, which prevents the my ClassLoader from being collected by the GC.
Do you have any suggestions on how to find this reference? As far as I understand, this reference will not be coming from the classes, which were loaded by that ClassLoader, but should come from classes loaded by parent classloaders. Is it possible to put a filter on references to show only references from classes, loaded by parent classloaders? If you have an ideea how to implement it in Java, I could implement it myself. I don't even have a clue, how to find out which classloader loaded a particular class.

[clebert.suconic]

So, you will have to drill down your references until you find the root cause.

Please, if you find anything related to JBoss, we would like to know. (open a thread on the proper forum. For example if you found something on EJB, open a thread on EJB forum).

Notice that we are aware of EJB3 leaking redeployments at this point. (There is a JIRA opened)

[clebert.suconic]

"ngolovin" wrote:

What you are suggesting is exactly what I did. I saw the matrix and the duplicates. Then I clicked on loader references and saw there approx. 2000 references (I have removed the 200-reference limit in my copy of the references.jsp)

What I do is...
I - I always look for the classLoader reference.

It's hard to create a filter for this. I couldn't come up with a logic, as sometimes a third level is what's causing a leakage. For example if you keep a reflection reference, you will have:

reflectionObject->class->ClassLoader


And this is really hard. If you have a classLoader loading 2000 classes, and only one class is kept referenced, all the 2000 classes will still be in the memory because of the ArrayList on ClassLoader. (read source code from JDK you will know what I'm talking about).

Try looking at field references first. Than move to another level.



Also.. when you said about parent classLoaders, I didn't understand what you meant. But what usually happens is a class or an instance of that class, loaded by any different classLoader keeping directly or indirectly a reference to your classLoader.

an example is Commons Logging. (if you're not using our patched version)

CommonsLogging will keep a HashMap of Loggers using Class as the key.

Class has the reference to the classLoader and CommonsLogging is loaded from System ClassLoader.


I don't know if this helps...
But... as I said before.. .chasing a redeployment leakage is never fun, even if you were using the most expensive tool.


Clebert Suconic

[ngolovin]

Thank you for your hints. I am using JBoss Seam, which in turn uses EJB3.
So that might be that leak in EJB3, however I do not see any references from org.jboss.ejb3.StatelessContainer which you mentioned in JIRA or any references from package org.jboss.ejb3 at all.

But what usually happens is a class or an instance of that class, loaded by any different classLoader keeping directly or indirectly a reference to your classLoader.

Yes, that's what I meant too. But how do you tell when looking at the references, which classloader was used to load the class? Is there a way to display this information for each class?

[clebert.suconic]

which classloader was used to load the class?

I don't know if this is what you meant.. but I never cared about the classLoader holding the class.

Every time I had a class leakage was one of these three scenarios:

- static field holding reference to class/classLoader/reflection
- Class instance/ClassLoader instance being held in a Map
- Class instance being held in a WeakHashMap value. (Values are not weak. Any circular reference between the key and a value will keep a leak)
- Instance of an object referenced.
- Stack Reference to object/class/classLoaders


I never had any case where a classLoader instance is holding a reference to its child classLoaders.

I never cared about this, because if all references are inside the same classLoader, the reference would be gone before getting to the duplicates.


ah... and about EJB3 I guess Bill Burke fixed some of those references, but there might be still some leakage he needs to work on.

[ngolovin]

I have checked the Jboss Seam standard "booking sample" application with JBoss profiler, and it has the redeployment leak as well. So it looks like JBoss Seam also has class leakage (or maybe the problem is in ear deployer?). There are however so many references, that I am unable to find where the problem is. To my inexperienced eye, it seems that the problem is in Seam. Do you mind if I attach the list of classloader references here and you take a short look at it? Then we can make a post in the Seam forum about the problem.

[clebert.suconic]

Open a new thread post, even on JBossProfiler forum if you want... something like...

"Is JBos SEAM leaking redeployment?"

Just to make it easier for future references. It will be easier for people googling for SEAM and leak in the future.

Besides, If we are sure about the leakage later I could even move the thread to that forum. (as I have admin privileges on this forum)



Clebert