Seam application crash under load

[mike82]

hi,

   I am trying to find what is wrong with my application for few days but I am out of ideas. It works fine, until heavy load comes. Well, actually is dealing fine with that load for some time. Then it slows down and finally becomes almost not accessible.

I checked database connections, not the problem of any query. I suffer a lot because of jboss cache - when it crashes I can see in jmx-console over 10k attributes...

Below few examples from thread dump, also from moment of slowness:

(this one appears many times in the dump):

Thread: ajp-0.0.0.0-8009-25 : priority:5, demon:true, threadId:170, threadState:BLOCKED- waiting on <0x70e5a93e> (a java.util.WeakHashMap)
org.jboss.naming.ENCFactory.getObjectInstance(ENCFactory.java:116)javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304)javax.naming.spi.NamingManager.getContext(NamingManager.java:422)javax.naming.spi.ContinuationContext.getTargetContext(ContinuationContext.java:38)

Thread: Thread-52796 : priority:5, demon:true, threadId:56143, threadState:TIMED_WAITING- waiting on <0x1ecf9c96> (a java.util.concurrent.SynchronousQueue$TransferStack)
sun.misc.Unsafe.park(Native Method)java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)

Thanks for any tips,
Francis

[jonssonth]

How about memory consumption and used threads? Performance issues can be really hard to identify. Have your tried running jconsole or any other performance/monitor tool? Which JBoss version are you running?

[mike82]

yes, the memory usage growth up to 4,7GB... I used jconsole, but this gives me nothing but a graph and memory pools usage. I know it's also could be a memory leak connected, but can't identify the source with jconsole. I saw that IBM's JVM allows to see memory occupation at class level, I believe I need to switch then... Or maybe any other tools?

I use JBoss 5.1 and I start it with following JVM attributes:

set JAVA_OPTS=%JAVA_OPTS% -Xms4g -Xmx4g -XX:MaxPermSize=512m -XX:NewRatio=3
set JAVA_OPTS=%JAVA_OPTS% -XX:+UseParallelGC -Dsun.rmi.dgc.client.gcInterval=180000 -Dsun.rmi.dgc.server.gcInterval=180000 -Dcom.sun.management.jmxremote

[kapitanpetko]

What sort of load? May clients, DB load?

Slowness is most probably due to increased GC activity. JConsole gives you a graph of that too. If so, reduce the heap size, take a memory dump once it starts to get slow, analyze with Eclipse Memory Analyzer, find and fix the leak.

And do use code comments and post readable stack traces.

HTH

[mike82]

Many clients, db usage is average those moments

[kapitanpetko]

Francis Drake wrote on Jun 15, 2010 05:23:


Many clients, db usage is average those moments

That probably means a lot stuff in session. Find the biggest session, then the biggest objects and the ones with the most instances and go from there. One of the usual culprits is the SMPC, since it lives in the session.

[mike82]

Sorry for basic question, where/how can I see sessions?

[mike82]

Francis Drake wrote on Jun 15, 2010 06:41:


Sorry for basic question, where/how can I see sessions?

And additional question, I also suspect it could be jboss cache which grows and grows (at crash time around 4000 nodes and 10000 attributes). However, using jmx-console and using printCacheDetails gives me around 50 nodes. Possible it could be 2nd level cache? Hibernate holding persisted objects?

[mike82]

update, I tried YourKit profiler, Eclipse Memory Analyzer with dump from jmap. Nothing, when my jboss java.exe process was 3,5GB, the analyze shown 2 possible memory leaks:

- 10 instances of org.apache.catalina.session.StandardManager, loaded by org.jboss.classloader.spi.base.BaseClassLoader @ 0xa9070438 occupy 276,022,464 (16.18%) bytes.

- 701 instances of org.jboss.virtual.plugins.context.zip.ZipEntryContext, loaded by org.jboss.bootstrap.NoAnnotationURLClassLoader @ 0x9fff7dc0 occupy 221,926,760 (13.01%) bytes.

This actually tells me nothing.

However YourKit shows 7500 instances of EDU.oswego.cs.dl.util.concurrent.WriterPreferenceReadWriteLock

I'm stuck and don't know what to try next. Anybody has an idea?

[kapitanpetko]

The memory analyzer can generates other reports beside the 'leak suspects', check those out too. Also try to drill down: the StandardManager objects probably hold references to the actual session objects. You can probably check who instantiated the ReadWriteLocks in YK, that will give you an idea what is going wrong. And, as I said, you can make your life easier if decrease the heap size, then you'll get smaller dumps and faster analysis.