Can't locate memory leak - possibly cache?

[mglowacki]

hi,

I am running JBoss AS 5.1 with some web applications. I run jboss using following jvm config:

view plaincopy to clipboardprint?

1.   set JAVA_OPTS=%JAVA_OPTS% -Xms4g -Xmx4g -XX:MaxPermSize=512m -XX:NewRatio=3 
2. set JAVA_OPTS=%JAVA_OPTS% -XX:+UseParallelGC -Dsun.rmi.dgc.client.gcInterval=180000 -Dsun.rmi.dgc.server.gcInterval=180000 -Dcom.sun.management.jmxremote   

After start my java.exe process occupies around 2GB of ram. After few hours it's becoming 4,7GB and is almost inaccessible...

I tried YourKit profiler, Eclipse Memory Analyzer with dump from jmap. Nothing, when my jboss java.exe process was 3,5GB, the analyze shown 2 possible memory leaks:

- 10 instances of org.apache.catalina.session.StandardManager, loaded by org.jboss.classloader.spi.base.BaseClassLoader @ 0xa9070438 occupy 276,022,464 (16.18%) bytes.

- 701 instances of org.jboss.virtual.plugins.context.zip.ZipEntryContext, loaded by org.jboss.bootstrap.NoAnnotationURLClassLoader @ 0x9fff7dc0 occupy 221,926,760 (13.01%) bytes.

This actually tells me nothing.

However YourKit shows 7500 instances of EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock

Also when I list class the biggest retained size belongs to ConcurrentHashMap, Also listing in jmx-console: jboss.cach.CacheJmxWrapper shows number of nodes growing, at 3,5GB it contains 3500 nodes, 5000 nodes at 4,5GB.

I'm stuck and don't know what to try next. Anybody has an idea?

[peterj]

Michal (or Franics), do you want me to answer you here or in the other forum where you posted this question?

Let's try here. Usually, I use two mechanisms to determine if there is a memory leak. First, I monitor the space left after a major garbage collection. After a decent wram up interval, I expect that the port major GC heap usage will have stabilized. If it does not and continues to increase, then I suspect a memory leak.

The other mechanism I use is to use VisualVM to take multiple memory snapshot and compare them. It will tell me which objects have increased in count, which then helps me to pinpoint the problem.

For suggestions on how to gather GC data and analyze it (which will show you the heap size after major GC), see these white papers:

Java Garbage Collection Statistical Analysis 101

Java Garbage Collection Performance Analysis 201

at http://www.cmg.org/cgi-bin/search.cgi?q=java+peter+johnson&x=30&y=10

[mglowacki]

hi Peter,

   this forum is fine Thanks for the tips, I will try and get back. From my recent tests I am bit confused why my OldGen is growing up to 3gb and is never collected. I will also look into suggested whitepapers.

Thanks for now,

Michal

[mglowacki]

looks like the problem is this ReadWriteLock, using another memory dump and then GC Roots looks like it,s created by:

state of org.jboss.jms.client.delegate.ClientConsumerDelegate [Stack Local]

<local variable> of java.lang.Thread [Stack Local, Thread]  "Connection Consumer for dest JBossTopic[Alerts] id=2"

When I have opened jmx-console I have above 1600 nondurable subscriptions! I don't know now if I misuse MDB or is it JBoss Messaging bug (I found some issues over the web).

[peterj]

Since this appears to be JBoss Messaging related you might want to raise the issue in the Messaging forum (and refer them back to this post).

http://community.jboss.org/en/jbossmessaging

[samuel.cai]

Michal, do you have same issue when running with JBoss 4?

If no, then you may check this: https://jira.jboss.org/browse/JBVFS-159

[mglowacki]

well, among with JBoss version upgrade I rewritten half my applications, so it's not that easy to say. With JBoss 4.2.3 was fine, also not that many vfs (ok, no vfs at all in jboss 4 :))/zipentries so it could be what you suggest. Thanks, I will try with this one as well.

For now system works fine, however java.exe process grows from 2GB at the beginning to 4,7GB in two days... And all objects retaining memory are Object[] and (Concurrent)HashMaps, what basically tells me nothing. Taking from GC Roots shows some FontLoader classes or ZipEntryHandlers...

[samuel.cai]

You can try on JBoss 4.