JBoss Application server core dumped

[rajasundaram]

Current thread (0x09307400):  VMThread [stack: 0x5109d000,0x5111e000] [id=5122]

Stack: [0x5109d000,0x5111e000],  sp=0x5111cba0,  free space=510k

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

V  [libjvm.so+0x60671b]

V  [libjvm.so+0x2d3a11]

V  [libjvm.so+0x19b715]

V  [libjvm.so+0x19b526]

V  [libjvm.so+0x55ace0]

V  [libjvm.so+0x6030df]

V  [libjvm.so+0x5ff61f]

V  [libjvm.so+0x1c60f7]

V  [libjvm.so+0x1c5d1b]

V  [libjvm.so+0x1c642b]

V  [libjvm.so+0x614907]

V  [libjvm.so+0x613f1c]

V  [libjvm.so+0x6140f1]

V  [libjvm.so+0x613ccf]

V  [libjvm.so+0x506929]

C  [libpthread.so.0+0x5832]

VM_Operation (0x4b8ad7c0): BulkRevokeBias, mode: safepoint, requested by thread 0x4dee9c00

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )

  0x1637b400 JavaThread "JBoss System Threads(1)-18435" daemon [_thread_blocked, id=31888, stack(0x3ca5e000,0x3caaf000)]

  0x1bdb6000 JavaThread "JBoss System Threads(1)-18434" daemon [_thread_blocked, id=31887, stack(0x444af000,0x44500000)]

  0x1a5aec00 JavaThread "JBoss System Threads(1)-18433" daemon [_thread_blocked, id=31886, stack(0x3ca0d000,0x3ca5e000)]

  0x0bfe5000 JavaThread "JBoss System Threads(1)-18432" daemon [_thread_blocked, id=31885, stack(0x4c20d000,0x4c25e000)]

  0x213fc400 JavaThread "JBoss System Threads(1)-18431" daemon [_thread_blocked, id=31884, stack(0x439bc000,0x43a0d000)]

  0x0a3a0000 JavaThread "JBoss System Threads(1)-18430" daemon [_thread_blocked, id=31883, stack(0x4e044000,0x4e095000)]

  0xbbdfd400 JavaThread "JBoss System Threads(1)-18429" daemon [_thread_blocked, id=31882, stack(0x3b00d000,0x3b05e000)]

  0x1a9a1c00 JavaThread "JBoss System Threads(1)-18428" daemon [_thread_blocked, id=31881, stack(0x3b45e000,0x3b4af000)]

  0x2079ec00 JavaThread "JBoss System Threads(1)-18427" daemon [_thread_blocked, id=31880, stack(0x4416b000,0x441bc000)]

  0x21d60c00 JavaThread "JBoss System Threads(1)-18426" daemon [_thread_blocked, id=31866, stack(0x4ddaf000,0x4de00000)]

  0x1d65b000 JavaThread "JBoss System Threads(1)-18425" daemon [_thread_blocked, id=31865, stack(0x4460d000,0x4465e000)]

  0x12e5cc00 JavaThread "JBoss System Threads(1)-18424" daemon [_thread_blocked, id=31841, stack(0x4afaf000,0x4b000000)]

  0x1a7dd400 JavaThread "JBoss System Threads(1)-18423" daemon [_thread_blocked, id=31834, stack(0x48daf000,0x48e00000)]

  0x1f9a5400 JavaThread "JBoss System Threads(1)-18422" daemon [_thread_blocked, id=31833, stack(0x4eb5e000,0x4ebaf000)]

  0x1cf34c00 JavaThread "JBoss System Threads(1)-18421" daemon [_thread_blocked, id=31831, stack(0x3ccaf000,0x3cd00000)]

                                                                                                           56,1           0%

=========================================================================

  JBoss Bootstrap Environment

  JBOSS_HOME: /usr/local/jboss

  JAVA: /usr/java/bin/java

  JAVA_OPTS: -Dprogram.name=run.sh -server  -Xms1024m -Xmx1024m -XX:PermSize=512M -XX:MaxPermSize=512M -XX:+DisableExplicitGC -XX:MaxNewSize=64m -XX:NewSize=64m -XX:SurvivorRatio=256 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:MaxTenuringThreshold=0 -XX:CMSInitiatingOccupancyFraction=60 -verbose:gc -XX:+PrintGCDetails -XX:+PrintTenuringDistribution -XX:+PrintGCTimeStamps -Xloggc:/opt/ion/logs/gc.log -Dcom.sun.management.jmxremote.port=8082 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/ion/logs -Djava.net.preferIPv4Stack=true

  CLASSPATH: /usr/local/jboss/bin/run.jar:/usr/java/lib/tools.jar

=========================================================================

19:37:14,029 INFO  [Server] Starting JBoss (MX MicroKernel)...

19:37:14,032 INFO  [Server] Release ID: JBoss [Trinity] 4.2.3.GA (build: SVNTag=JBoss_4_2_3_GA date=200807181439)

19:37:14,035 INFO  [Server] Home Dir: /usr/local/jboss-4.2.3.GA

19:37:14,035 INFO  [Server] Home URL: file:/usr/local/jboss-4.2.3.GA/

19:37:14,037 INFO  [Server] Patch URL: null

19:37:14,037 INFO  [Server] Server Name: default

19:37:14,037 INFO  [Server] Server Home Dir: /usr/local/jboss-4.2.3.GA/server/default

19:37:14,038 INFO  [Server] Server Home URL: file:/usr/local/jboss-4.2.3.GA/server/default/

19:37:14,038 INFO  [Server] Server Log Dir: /usr/local/jboss-4.2.3.GA/server/default/log

19:37:14,038 INFO  [Server] Server Temp Dir: /usr/local/jboss-4.2.3.GA/server/default/tmp

19:37:14,039 INFO  [Server] Root Deployment Filename: jboss-service.xml

19:37:15,320 INFO  [ServerInfo] Java version: 1.6.0_13,Sun Microsystems Inc.

Need help very urgent.

[peterj]

Which OS?

Is this a "pure" AS 4.2.3, or did you make any changes?

>>Java version: 1.6.0_13,Sun Microsystems Inc.

I recommend installing a recent JDK, 6u13 is really old.

[rajasundaram]

OS is RedHat Linux and we did not make any changes in jboss server. And we use default server.

[peterj]

Also, try dropping some of your JAVA_OPTS - I suspect that the JVM might be tripping over some combination of options. Once you determine the options combination, if it still happens in the latest JDK 6, submit a bug report to Oracle.

By the way, your NewSize is way too small! With the current settings you will be working the CMS collector way to hard and will end up negatively impacting your throughput.

[godski]

I think you are probably correct on the options tripping us up.  (I work with Raja, btw)

One thing I am wondering is if I look at the CG time it shows:

That 1 CMS collection was forced by us.  If we let it go too long after 1 GB it will eventually crash.  It looks like the CMS collection never runs.  Also, the "Par Surviovor Space" never gets used.

It also looks like we are at a physical memory ceiling that isn't working well with the JVM memory settings.  I am wondering if we'd be better off just using the default garbage collector with only the max heap size set.

Any insight is appreciated!

[peterj]

Actually, that graph is not showing GC info. Instead it is showing heap usage snapshots. But based on the graph I can't tell if you you are seeing is a memory leak or not. Gathering and plotting actual GC data might help. For suggestions on how to gather GC data and analyze it, see these white papers:

Java Garbage Collection Statistical Analysis 101

Java Garbage Collection Performance Analysis 201

at http://www.cmg.org/cgi-bin/search.cgi?q=java+peter+johnson&x=30&y=10