-
1. Re: Seam application crash under load
jonssonth Jun 15, 2010 4:20 AM (in response to mike82)How about memory consumption and used threads? Performance issues can be really hard to identify. Have your tried running jconsole or any other performance/monitor tool? Which JBoss version are you running?
-
2. Re: Seam application crash under load
mike82 Jun 15, 2010 4:50 AM (in response to mike82)yes, the memory usage growth up to 4,7GB... I used jconsole, but this gives me nothing but a graph and memory pools usage. I know it's also could be a memory leak connected, but can't identify the source with jconsole. I saw that IBM's JVM allows to see memory occupation at class level, I believe I need to switch then... Or maybe any other tools?
I use JBoss 5.1 and I start it with following JVM attributes:
set JAVA_OPTS=%JAVA_OPTS% -Xms4g -Xmx4g -XX:MaxPermSize=512m -XX:NewRatio=3
set JAVA_OPTS=%JAVA_OPTS% -XX:+UseParallelGC -Dsun.rmi.dgc.client.gcInterval=180000 -Dsun.rmi.dgc.server.gcInterval=180000 -Dcom.sun.management.jmxremote -
3. Re: Seam application crash under load
kapitanpetko Jun 15, 2010 5:09 AM (in response to mike82)What sort of load? May clients, DB load?
Slowness is most probably due to increased GC activity. JConsole gives you a graph of that too. If so, reduce the heap size, take a memory dump once it starts to get slow, analyze with Eclipse Memory Analyzer, find and fix the leak.
And do use code comments and post readable stack traces.
HTH
-
4. Re: Seam application crash under load
mike82 Jun 15, 2010 5:23 AM (in response to mike82)Many clients, db usage is average those moments
-
5. Re: Seam application crash under load
kapitanpetko Jun 15, 2010 5:27 AM (in response to mike82)
Francis Drake wrote on Jun 15, 2010 05:23:
Many clients, db usage is average those momentsThat probably means a lot stuff in session. Find the biggest session, then the biggest objects and the ones with the most instances and go from there. One of the usual culprits is the SMPC, since it lives in the session.
-
6. Re: Seam application crash under load
mike82 Jun 15, 2010 6:41 AM (in response to mike82)Sorry for basic question, where/how can I see sessions?
-
7. Re: Seam application crash under load
mike82 Jun 15, 2010 6:59 AM (in response to mike82)
Francis Drake wrote on Jun 15, 2010 06:41:
Sorry for basic question, where/how can I see sessions?And additional question, I also suspect it could be jboss cache which grows and grows (at crash time around 4000 nodes and 10000 attributes). However, using jmx-console and using
printCacheDetails
gives me around 50 nodes. Possible it could be 2nd level cache? Hibernate holding persisted objects? -
8. Re: Seam application crash under load
mike82 Jun 15, 2010 11:12 AM (in response to mike82)update, I tried YourKit profiler, Eclipse Memory Analyzer with dump from jmap. Nothing, when my jboss java.exe process was 3,5GB, the analyze shown 2 possible memory leaks:
- 10 instances of
org.apache.catalina.session.StandardManager
, loaded byorg.jboss.classloader.spi.base.BaseClassLoader @ 0xa9070438
occupy 276,022,464 (16.18%) bytes.- 701 instances of
org.jboss.virtual.plugins.context.zip.ZipEntryContext
, loaded byorg.jboss.bootstrap.NoAnnotationURLClassLoader @ 0x9fff7dc0
occupy 221,926,760 (13.01%) bytes.This actually tells me nothing.
However YourKit shows 7500 instances of EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock
I'm stuck and don't know what to try next. Anybody has an idea?
-
9. Re: Seam application crash under load
kapitanpetko Jun 15, 2010 9:23 PM (in response to mike82)The memory analyzer can generates other reports beside the 'leak suspects', check those out too. Also try to drill down: the StandardManager objects probably hold references to the actual session objects. You can probably check who instantiated the ReadWriteLocks in YK, that will give you an idea what is going wrong. And, as I said, you can make your life easier if decrease the heap size, then you'll get smaller dumps and faster analysis.