1 2 3 Previous Next 43 Replies Latest reply on Apr 24, 2006 5:27 PM by kabirkhan

JBAS-3016 - Memory Leaks on redeployments

clebert.suconic Apr 3, 2006 4:31 PM

I just made dcl a WeakReference on AOPClassPool.
Had to make dcl private, and make it a WeakReference.

but that didn't solve the leak. There are still StrongReferences through reflection.

Look at org.jboss.aop.MethodJoinPoint sourceCode for instance.

/**
 * This class is here to eliminate a hash lookup in invokeMethod
 */
public class MethodJoinPoint extends JoinPoint
{
 public Method advisedMethod;
 public Method unadvisedMethod;

also, something caught my attention.

After running AOPTestCase (from jboss-head/testsuite)

org.jboss.test.aop.bean.POJOWildCardConstructorInterceptor had an impressive number of InstanceOf references.

So, It looks this is not just leaking Classes references, but it might also have some Object leaks.

org.jboss.mx.loading.UnifiedClassLoader3@4373267
!--- class org.jboss.aop.advice.org.jboss.test.aop.bean.Aspect15
!--- !--- [Ljava.lang.Object;@14123321
!--- !--- !--- java.util.Vector@22053093
!--- !--- !--- !--- org.jboss.mx.loading.UnifiedClassLoader3@4373267
!--- !--- !--- !--- !--- class org.jboss.test.aop.bean.POJOWildCardConstructorInterceptor
!--- !--- !--- !--- !--- !--- class java.lang.Class
!---!---!---!---!---!---!---Root Detail
!---!---!---!---!---!---!---Root Detail
!---!---!---!---!---!---!---Root Detail
!---!---!---!---!---!---!---Root Detail Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@7262684 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@26471365 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@6021288 Detail
!---!---!---!---!---!---!--- FieldReference private final java.lang.Class java.io.ObjectStreamField.type=java.io.ObjectStreamField@24426889 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@4256497 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@907380 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@16012915 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@22527606 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@26832405 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@678620 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@30625452 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@30541393 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@27061842 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@31617094 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@20519468 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@31805111 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@26411145 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@549050 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@419018 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@29570270 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@16737727 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@24188282 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@20516810 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@20687062 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@29996737 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@33386806 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@1293396 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@28404046 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@7911461 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@25268918 Detail
!---!---!---!---!---!---!---InstanceOfReference:ToString=[Ljava.lang.Class;@29827550 Detail


... there are more... I had to cut the report here, or it would be too big otherwise

1. Re: JBAS-3016 - Memory Leaks on redeployments

clebert.suconic Apr 3, 2006 6:23 PM (in response to clebert.suconic)

I was talking to Kabir today over MSN, and I advised him to take a look on package org.jboss.serial.references at JBossSerialization.

There are three useful classes on that package:

- ConstructorPersistentReference
- FieldPersistentReference
- MethodPersistentReference

All subClasses of PersistentReference.

BTW: Maybe we should make this package more generic, in case AOP and JBossSerialization are both using it.

A PersistentReference (this is how I called) will keep the reflection object under Weak or Soft reference (depending on the parameter). In case the reference is gone the super class (PersistentReference) will transparently recreate the reflection object using reflection operations.

I had to do this way on JBossSerialization, because:

I - Reflection objects are always cloned, so you can't keep then under a regular WeakReference. (There is nothing on Java API keeping a reference what would trigger WeakReferences correctly)

II - Keeping reflection objects won't allow WeakHashMap<ClassLoader,Whatever) to be released.

There are more details on this thread:

http://www.jboss.com/index.html?module=bb&op=viewtopic&t=78975

And

http://wiki.jboss.org/wiki/Wiki.jsp?page=ClassLeakage
Actions
2. Re: JBAS-3016 - Memory Leaks on redeployments

clebert.suconic Apr 5, 2006 12:39 AM (in response to clebert.suconic)
I just created a Testcase that will help identifying where are MemoryLeaks, and should prevent regressions on the future.

This is really cool, BTW.

To execute this, you should run:

ant main,memory-test

under /aop project. (at jboss-head of course)

You only need jbossAgent.dll or jbossAgent.so (update version from jboss-profiler/CVS, as it will be required).

Some considerations:

I - On org.jboss.test.aop.memoryleaks.MemoryLeakTester
There is a commented line that can be useful to this test:

private void checkUnload(String className) throws Exception { JVMTIInterface jvmti = new JVMTIInterface(); if (jvmti.isActive()) { //clearEverySingleFieldOnInstances("org.jboss.aop.AspectManager"); // This part is not intended to be commited. It could be used during debug, and you could use to release references on purpose, just to evaluate behavior

clearEverySingleField will clear any caches on AspectManager. Hence if this line is here, the test runs successfully what means the infra-structure here is working.

II - Undeploment is not working for some reason. I need some help (from Kabir/Bill B) on why this didn't work.

III - If references are still available, a report (leaks-report.html) will be created. Also a snapshot is also created if you want to use memory-profiler on jboss-profiler.
Actions
3. Re: JBAS-3016 - Memory Leaks on redeployments

clebert.suconic Apr 5, 2006 12:20 PM (in response to clebert.suconic)

"Clebert" wrote:
II - Undeployment is not working for some reason

Scratch that. undeployment is working fine. The test is complete already.
Actions
4. Re: JBAS-3016 - Memory Leaks on redeployments

kabirkhan Apr 6, 2006 9:45 AM (in response to clebert.suconic)
I've created a branch for this work, to make sure my changes don't affect anything else until I've found a proper solution.

$cvs co -r AOP_MEMORY_LEAK_INVESTIGATION jboss-head

I don't really understand the output of your "memory-test" unit test, but in the branch I have got the following working:

The AspectManager is what holds the references to the classes indexed by classloader, and then the ClassAdvisors are what holds all the reflection info for the classes. When weaving, we introduce a field into the woven classes as follows:
public class POJO{ org.jboss.aop.Advisor aop$classAdvisor = AspectManager.instance().getAdvisor(POJO.class); //original code }

I've gone through and made sure that everything holds a weak reference to the advisor, i.e. this field is the only hard reference. At undeployment, in AspectDeployer I now call AspectManager.unregisterClassLoader() which goes through and sets this field to null on all the advised objects.

I created a small really basic test case in the testsuite for this branch. You can run this by:

$build one-test -Dtest=org.jboss.test.aop.test.AOPMiniUnitTestCase

Initially this failed, and I was seeing stuff like
org.jboss.mx.loading.UnifiedClassLoader3@1396421 !--- class org.jboss.test.aop.mini.TestInterceptor !--- !--- java.lang.reflect.Method@28036446 !--- !--- !--- java.lang.ref.SoftReference@25756944 !--- !--- !--- !--- java.beans.PropertyDescriptor@9685265 !--- !--- !--- !--- !--- [Ljava.beans.PropertyDescriptor;@31407084 !--- !--- !--- !--- !--- !--- java.beans.GenericBeanInfo@10458977 !--- !--- !--- !--- !--- !--- !--- java.util.WeakHashMap$Entry@24645424 !--- !--- !--- !--- !--- !--- !--- !--- [Ljava.util.WeakHashMap$Entry;@8854988 !--- !--- !--- !--- !--- !--- !--- !--- !--- java.util.WeakHashMap@1524670 !---!---!---!---!---!---!---!---!---!--- FieldReference private java.util.Map java.util.Collections$SynchronizedMap.m=java.util.Collections$SynchronizedMap@5578467 Detail

We use property editors in GenericAspectFactory, to support things like:
<aop> <interceptor class="TestInterceptor"> <attribute name="Value">5</attribute> </interceptor>

when instantiated this will call TestInterceptor.setValue(5);

Once I comment out GenericAspectFactory.configureInstance(), which does the property editor stuff, I no longer get duplicate classes when running this test twice, so it seems that the SoftReferences they hold internally contributed to the problem.

I will keep what I have in the branch for now, and see if I can get the more advanced tests to pass. If so, I think I will need to find an alternative to property editors.

I have commited these changes to the branch if you want to take a look, and maybe see why the memory-test fails.
Actions
5. Re: JBAS-3016 - Memory Leaks on redeployments

clebert.suconic Apr 6, 2006 4:12 PM (in response to clebert.suconic)
A weaved class holding reflection information from itself shouldn't be a problem.

Like if a user class is holding a reflection object, a class reference, it shouldn't be a problem.

class POJO { Method methods[] = null; static { methods=POJO.class.getDelcaredMethod(); } }

So, Weaved classes would fall in the same category, as they will hold reflection information about itself.

the problem will be when some class holding metadata on a superior classLoader, like what happens with singletong AspectManager.

Hence if you clear every field on AspectManager, the test will pass.

Just uncomment this line from MemoryLeakTester, and you will see what I'm talking about:

//clearEverySingleFieldOnInstances("org.jboss.aop.AspectManager");
Actions
6. Re: JBAS-3016 - Memory Leaks on redeployments

clebert.suconic Apr 7, 2006 9:10 AM (in response to clebert.suconic)

Kabir,

If you're seeing only SoftReferences when you do the test, there is a method on JVMTIInterface (exposed throught its MBean on jmx-console) that will release every SoftReference without a secondary object holding a reference.

SoftReferences are only released when the memory is low, and that method only allocates memory until SoftReferences are gone.

I forgot what's the exact method's name. I guess is something on forceReleaseOnSoftReferences or something like that. Take a look JVMTIInterfaceMBean and call that method on JMX-Console before calling the report.

There is actually another method on the MBean (listHTMLClassesReport or something similar) that will only show if there are duplicates on classes.

Clebert Suconic
Actions
7. Re: JBAS-3016 - Memory Leaks on redeployments

kabirkhan Apr 7, 2006 9:18 AM (in response to clebert.suconic)

I've reworked your test to work as a wrapper for some of the standalone tests. It is in the branch, and can be run with

$ build memory-tests

Your original test can still be run with
$ build original-memory-test

I see that the memory test forces a release of soft refences, so the property editor stuff I mentioned isn't a "real" issue.

The "introduction" and "scope" tests still leak though, due to some other issue which I haven't isolated yet.
Actions
8. Re: JBAS-3016 - Memory Leaks on redeployments

kabirkhan Apr 7, 2006 9:24 AM (in response to clebert.suconic)

Did not see your post before posting :-) So yes, running the memory tests the property editors is fine since the memory tests force a release of soft references.

How do we force release of this when doing a redeploy in JBoss?
Actions
9. Re: JBAS-3016 - Memory Leaks on redeployments

kabirkhan Apr 7, 2006 9:26 AM (in response to clebert.suconic)

"kabir.khan@jboss.com" wrote:

How do we force release of this when doing a redeploy in JBoss?

I mean how do we release the soft references held, when running the AOPUnitTestCase twice in Jboss?
Actions
10. Re: JBAS-3016 - Memory Leaks on redeployments

clebert.suconic Apr 7, 2006 12:43 PM (in response to clebert.suconic)
JVMTIInterface is exposed through JMX.

If you have installed jboss-profiler-jvmti.sar (I know you do), you will see the proper MBean:

On JMX-Console jboss.profiler * mbean=JVMTIClass

If you hit this URL should get you there when you have JBoss running with JBossProfiler installed.

http://localhost:8080/jmx-console/HtmlAdaptor?action=inspectMBean&name=jboss.profiler%3Ambean%3DJVMTIClass
Actions

11. Re: JBAS-3016 - Memory Leaks on redeployments

kabirkhan Apr 7, 2006 12:54 PM (in response to clebert.suconic)

I have run the main aop tests in the testsuite checking for duplicates. I did it without the property editors enabled, since I only just found out the trick to clear all the soft references is to execute forceReleaseOnSoftReferences on the jboss.profiler:mbean=JVMTIClass. I will try with this next.

These seem ok:

 [junit] Running org.jboss.test.aop.test.AOPMiniUnitTestCase
 [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.859 sec
OK - no dupes

 [junit] Running org.jboss.test.aop.test.AOPUnitTestCase
 [junit] Tests run: 11, Failures: 0, Errors: 0, Time elapsed: 6.688 sec
OK - no dupes

 [junit] Running org.jboss.test.aop.test.JMXUnitTestCase
 [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.672 sec
OK - no dupes

 [junit] Running org.jboss.test.aop.test.ObservableUnitTestCase
 [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.875 sec
OK - no dupes

 [junit] Running org.jboss.test.aop.test.SecurityUnitTestCase
 [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 5.578 sec
OK - no dupes

 [junit] Running org.jboss.test.aop.test.TxLockUnitTestCase
 [junit] Tests run: 2, Failures: 0, Errors: 2, Time elapsed: 4.828 sec
OK - no dupes

 [junit] Running org.jboss.test.aop.test.TxUnitTestCase
 [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 3.516 sec
OK - no dupes

AOP remoting has a lot of maps hanging onto methods

 [junit] Running org.jboss.test.aop.test.RemotingUnitTestCase
 [junit] Tests run: 4, Failures: 0, Errors: 4, Time elapsed: 4.11 sec

org.jboss.mx.loading.UnifiedClassLoader3@28040030
!--- class org.jboss.test.aop.bean.POJO
!--- !--- java.lang.reflect.Method@1173995
!--- !--- !--- java.util.HashMap$Entry@29094721
!--- !--- !--- !--- [Ljava.util.HashMap$Entry;@15043005
!--- !--- !--- !--- !--- java.util.HashMap@21499286
!--- !--- !--- !--- !--- !--- java.util.WeakHashMap$Entry@23171383
!--- !--- !--- !--- !--- !--- !--- [Ljava.util.WeakHashMap$Entry;@21241280
!--- !--- !--- !--- !--- !--- !--- !--- java.util.WeakHashMap@5326199
!---!---!---!---!---!---!---!---!--- StaticFieldReference private static java.util.WeakHashMap org.jboss.aop.proxy.ClassProxyFactory.methodMapCache Detail

org.jboss.mx.loading.UnifiedClassLoader3@28040030
!--- class org.jboss.test.aop.bean.POJO
!--- !--- java.lang.reflect.Method@19711599
!--- !--- !--- java.util.HashMap$Entry@21414603
!--- !--- !--- !--- [Ljava.util.HashMap$Entry;@17917840
!--- !--- !--- !--- !--- java.util.HashMap@27901165
!---!---!---!---!---!--- StaticFieldReference private static java.util.Map AOPClassProxy$2.methodMap Detail

The versioned object tests gives duplicates, but I will need to look more at what this does since I am not familiar with it

 [junit] Running org.jboss.test.aop.test.VersionedObjectUnitTestCase
 [junit] Running org.jboss.test.aop.test.VersionedObjectUnitTestCase
 [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 4.109 sec

12. Re: JBAS-3016 - Memory Leaks on redeployments

kabirkhan Apr 7, 2006 1:02 PM (in response to clebert.suconic)
Finally, the testsuite test for intercepting an ejb gives duplicates, it looks like references are being kept.

org.jboss.test.aop.simpleejb.Simple is a remote ejb interface.

I don't know what a SoftCache is?
org.jboss.mx.loading.UnifiedClassLoader3@31988659 !--- interface org.jboss.test.aop.simpleejb.Simple !--- !--- java.util.HashMap$Entry@1547458 !--- !--- !--- [Ljava.util.HashMap$Entry;@23838842 !--- !--- !--- !--- java.util.HashMap@1260960 !---!---!---!---!--- FieldReference private java.util.Map sun.misc.SoftCache.hash=sun.misc.SoftCache@10765342 Detail

This looks like it comes from the server module (SessionMetaData) or remoting?
Actions

13. Re: JBAS-3016 - Memory Leaks on redeployments

kabirkhan Apr 7, 2006 1:03 PM (in response to clebert.suconic)

(Had my tags wrong)
....
This looks like it comes from the server module (SessionMetaData) or remoting?

org.jboss.mx.loading.UnifiedClassLoader3@31988659
!--- interface org.jboss.test.aop.simpleejb.Simple
!--- !--- java.lang.reflect.Method@9040273
!--- !--- !--- EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap$Entry@204692
!--- !--- !--- !--- [LEDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap$Entry;@9572696
!--- !--- !--- !--- !--- EDU.oswego.cs.dl.util.concurrent.ConcurrentReaderHashMap@6043841
!--- !--- !--- !--- !--- !--- org.jboss.metadata.SessionMetaData@8701723
!--- !--- !--- !--- !--- !--- !--- java.lang.ThreadLocal$ThreadLocalMap$Entry@18978380
!--- !--- !--- !--- !--- !--- !--- !--- [Ljava.lang.ThreadLocal$ThreadLocalMap$Entry;@17956480
!--- !--- !--- !--- !--- !--- !--- !--- !--- java.lang.ThreadLocal$ThreadLocalMap@28275996
!--- !--- !--- !--- !--- !--- !--- !--- !--- !--- org.jboss.remoting.transport.socket.ServerThread@31077072
!---!---!---!---!---!---!---!---!---!---!---Root Detail
!---!---!---!---!---!---!---!---!---!---!---Root Detail
!---!---!---!---!---!---!---!---!---!---!--- Reference inside a method - java.lang.Object::wait Detail
!---!---!---!---!---!---!---!---!---!---!--- Reference inside a method - org.jboss.remoting.transport.socket.ServerThread::run Detail
!---!---!---!---!---!---!---!---!---!---!--- Reference inside a method - org.jboss.remoting.transport.socket.ServerThread::run Detail
!---!---!---!---!---!---!---!---!---!---!--- arrayRef [Ljava.lang.Thread;[115] id=@870036 Detail
!---!---!---!---!---!---!---!---!---!---!--- FieldReference java.lang.Object java.util.LinkedList$Entry.element=java.util.LinkedList$Entry@874960 Detail
!---!---!---!---!---!---!---!---!---!---!--- FieldReference private java.lang.Object java.lang.ref.Reference.referent=java.util.WeakHashMap$Entry@8614136 Detail

14. Re: JBAS-3016 - Memory Leaks on redeployments

clebert.suconic Apr 7, 2006 1:10 PM (in response to clebert.suconic)

Ah....

I guess this is ThreadPool/Remoting not clearing Thread.currentClassLoader before the thread goes back to the pool.

Lets wait on Tom on what he says. I guess we will need to open a JIRA for MemoryLeaks on remoting, due to ClassLoader stored on ThreadPool.
Actions

1 2 3 Previous Next

Go to original post