4 Replies Latest reply on Oct 31, 2008 1:56 AM by dgenkin

    JRockit JVM used by JBoss crashes with a dump.

    dgenkin

      Hi,

      We are using jrockit-R27.6.0-jdk1.6.0_05 jvm to run JBoss 4.2.3-GA. I had several occurences of a JVM crash recently.
      Does anyone have any idea/suggestion of what might be causing the problem? Thank you in advance for your help.

      Details:

      Here are java arguments: JAVA_OPTS=" -Xms2047M -Xmx2047M -jrockit -Xgc:parallel -Xverbose:memory -Xgcreport -XXaggressive -XXlazyunlocking -XXtlasize:min=4k,preferred=512k -XXcallprofiling -Djava.security.policy=$JAVA_HOME/lib/security/java.policy"


      Here is a dump file:


      ===== BEGIN DUMP =============================================================
      JRockit dump produced after 2 days, 08:47:31 on Thu Oct 30 08:33:45 2008

      **********************************************************
      * If you see this dump, please go to *
      * http://edocs.bea.com/jrockit/go2troubleshooting.html *
      * for troubleshooting information. *
      **********************************************************

      Additional information is available in:
      /home/jboss/jrockit.32675.dump
      No snapshot file (core dump) will be created because core dumps have been
      disabled. To enable core dumping, try "ulimit -c unlimited"
      before starting JRockit again.

      Error Message: Illegal memory access. [54]
      Signal info : si_signo=11, si_code=128 si_addr=(nil)
      Version : BEA JRockit(R) R27.6.0-50_o-100423-1.6.0_05-20080626-2104-linux-x86_64
      GC Strategy : Mode: static, with strategy: singleparpar
      GC Status : OC currently running, in phase: marking. This is OC#1307.
      Heap : 0x40b00000 - 0xc0a00000 (Size: 2047 MB)
      Compaction : 0x43afa000 - 0x45af6000 (Current compaction type: internal)
      CompRefs : References are compressed, with heap base 0x0.
      CPU : Intel Core 2 SSE SSE2 SSE3 SSSE3 SSE4.1 EM64T
      Number CPUs : 2
      Tot Phys Mem : 5194665984 (4954 MB)
      OS version : CentOS release 5 (Final)
      Linux version 2.6.18-53.1.21.el5 (mockbuild@builder10.centos.org) (gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)) #1 SMP Tue May 20 09:35:07 EDT 2008 (x86_64)
      Thread System: NPTL
      Java locking : Lazy unlocking enabled (class banning) (transfer banning)
      State : JVM is running (Main thread has finished)
      Command Line : -Dprogram.name=run.sh -Xms2047M -Xmx2047M -Xgc:parallel -Xverbose:memory -Xgcreport -XXaggressive -XXlazyunlocking -XXtlasize:min=4k,preferred=512k -XXcallprofiling -Djava.security.policy=/home/jboss/product/jdk/lib/security/java.policy -Djava.rmi.server.hostname=idc-jboss08.hireright.com -Djavax.management.builder.initial=org.jboss.system.server.jmx.MBeanServerBuilderImpl -Djboss.platform.mbeanserver -Dcom.sun.management.jmxremote.port=1120 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.net.preferIPv4Stack=true -Djava.library.path=/home/jboss/product/jboss/bin/native -Djava.endorsed.dirs=/home/jboss/product/jboss/lib/endorsed -Dsun.java.launcher=SUN_STANDARD org.jboss.Main -c all -b ows01.hireright.com
      java.home : /home/jboss/product/jrockit-R27.6.0-jdk1.6.0_05
      j.class.path : /home/jboss/product/jboss/bin/run.jar
      j.lib.path : /home/jboss/product/jboss/bin/native
      JAVA_HOME : /home/jboss/product/jdk
      _JAVA_OPTIONS: <not set>
      LD_LIBRARY_PATH: /home/jboss/product/jrockit-R27.6.0-jdk1.6.0_05/lib/amd64/jrockit:/home/jboss/product/jrockit-R27.6.0-jdk1.6.0_05/lib/amd64:/home/jboss/product/jrockit-R27.6.0-jdk1.6.0_05/../lib/amd64:/home/jboss/product/jboss/bin/native
      LD_ASSUME_KERNEL: <not set>
      C Heap : Good; no memory allocations have failed
      StackOverFlow: 0 StackOverFlowErrors have occured
      OutOfMemory : 0 OutOfMemoryErrors have occured

      Registers (from ThreadContext: 0x40ac5a10 / OS context: 0x40ac5ad0):
      rax = 03fffffffefd61b0 rcx = 0000000000000007
      rdx = 03ffffffff5d61b0 rbx = 0000000000086a3a
      rsp = 0000000040ac5f10 rbp = 0000000040ac6060
      rsi = 03ffffffff5d61b0 rdi = 0000000000000080
      r8 = 0000000000000001 r9 = 4890924244922448
      r10 = 0000000000000000 r11 = 0000000000000202
      r12 = 0000000040ac5f50 r13 = 0000000040b02600
      r14 = 0000000040ac5f60 r15 = 0000000000000001
      cs = 0000000000000033 fs = 0000000000000000
      gs = 0000000000000000
      rip = 00002aaaaac263ff flags = 0000000000000202

      Stack:
      (* marks the word pointed to by the stack pointer)
      0000000040ac5f10: 0000000006d64e10* 0000000040ac6000 00002aaaaaf15660 0000000006d64e10
      0000000040ac5f30: 0000000040b02608 0000000000000000 0000000040b02608 0000000000000000
      0000000040ac5f50: 0000000040b02608 0000000000000000 0000000000000000 0000000000000000
      0000000040ac5f70: 0000000040b02608 0000000000000000 0000000040b02600 0000000000000000
      0000000040ac5f90: 0000000000539840 0000000000000001 0000000000000000 0000000040b0260c
      0000000040ac5fb0: 0000003b0640fb88 0000000000000001 0000003b06409a79 0000000006c9e5e0

      Code:
      (* marks the word pointed to by the instruction pointer)
      00002aaaaac26398: 058b481172183b48 73183b480028682c d28500000001ba05 fffed0b58b481374
      00002aaaaac263b8: fffffed8958b48ff 058b48ffeec6a1e8 48108b480028605c 89480028673a058b
      00002aaaaac263d8: 03e9c148082b48d9 8306e8c148c88948 e7d348ff894c3fe1 00000008c5048d48
      00002aaaaac263f8: 48c2014802348d48* f90949c989490e8b 394c00000000b841 4cf0c889481374c9
      00002aaaaac26418: df75c839480ab10f 854500000001b841 6546058b482374c0 4808740038830028
      00002aaaaac26438: 48fffb8bbde8df89 fffeb0bd8b48de89 8d48ffec92bbe8ff 858b48ffffff0095

      Loaded modules:
      (* denotes the module causing the exception)
      0000000000400000-0000000000412ffb /home/jboss/product/jrockit-R27.6.0-jdk1.6.0_05/bin/java
      0000003b06400000-0000003b06414aaf /lib64/libpthread.so.0
      0000003b06000000-0000003b06081247 /lib64/libm.so.6
      0000003b05c00000-0000003b05c01f93 /lib64/libdl.so.2
      0000003b05800000-0000003b0594930f /lib64/libc.so.6
      0000003b05400000-0000003b05419c3f /lib64/ld-linux-x86-64.so.2
      00002aaaaaab9000-00002aaaaace7a8f */home/jboss/product/jrockit-R27.6.0-jdk1.6.0_05/lib/amd64/jrockit/libjvm.so
      0000003b07400000-0000003b07406ea3 /lib64/librt.so.1
      00002aaaaaf22000-00002aaaaaf2b64b /lib64/libnss_files.so.2
      00002aaaab1a3000-00002aaaab1af337 /home/jboss/product/jrockit-R27.6.0-jdk1.6.0_05/lib/amd64/libverify.so
      00002aaaab2b2000-00002aaaab2d9ff3 /home/jboss/product/jrockit-R27.6.0-jdk1.6.0_05/lib/amd64/libjava.so
      0000003b09800000-0000003b0981418f /lib64/libnsl.so.1
      00002aaaeb431000-00002aaaeb43765f /home/jboss/product/jrockit-R27.6.0-jdk1.6.0_05/lib/amd64/native_threads/libhpi.so
      00002aaaeeba1000-00002aaaeebae9d3 /home/jboss/product/jrockit-R27.6.0-jdk1.6.0_05/lib/amd64/libzip.so
      00002aaaeee1d000-00002aaaeee2249b /home/jboss/product/jrockit-R27.6.0-jdk1.6.0_05/lib/amd64/libmanagement.so
      00002aaaeef44000-00002aaaeef56687 /home/jboss/product/jrockit-R27.6.0-jdk1.6.0_05/lib/amd64/libnet.so
      00002aaaef78f000-00002aaaef792ccf /lib64/libnss_dns.so.2
      0000003b0a400000-0000003b0a410823 /lib64/libresolv.so.2
      00002aaaefe26000-00002aaaefe268bb /home/jboss/product/jrockit-R27.6.0-jdk1.6.0_05/lib/amd64/librmi.so
      00002aaaf439c000-00002aaaf43ba75b /home/jboss/product/jboss-4.2.3.GA/bin/native/libtcnative-1.so.0.1.13
      00002aaaf44bc000-00002aaaf44fa4c3 /home/jboss/product/jboss/bin/native/libssl.so.0.9.8
      00002aaaf4600000-00002aaaf4726513 /home/jboss/product/jboss/bin/native/libcrypto.so.0.9.8
      00002aaaf484a000-00002aaaf487015b /home/jboss/product/jboss/bin/native/libapr-1.so.0
      00002aaaf497c000-00002aaaf497deb3 /lib64/libuuid.so.1
      0000003b08000000-0000003b08008503 /lib64/libcrypt.so.1
      00002aaaf4f9d000-00002aaaf4fa38ab /home/jboss/product/jrockit-R27.6.0-jdk1.6.0_05/lib/amd64/libnio.so
      00002aaaf6001000-00002aaaf608a623 /home/jboss/product/jrockit-R27.6.0-jdk1.6.0_05/lib/amd64/libawt.so
      00002aaaf61c7000-00002aaaf61cb70f /home/jboss/product/jrockit-R27.6.0-jdk1.6.0_05/lib/amd64/headless/libmawt.so
      00002aaaf62cd000-00002aaaf634a32f /home/jboss/product/jrockit-R27.6.0-jdk1.6.0_05/lib/amd64/libfontmanager.so
      00002aaaf6818000-00002aaaf68630cb /home/jboss/product/jrockit-R27.6.0-jdk1.6.0_05/lib/amd64/libcmm.so
      00002aaaf6a72000-00002aaaf6a9d87f /home/jboss/product/jrockit-R27.6.0-jdk1.6.0_05/lib/amd64/libjpeg.so

      Scan Dump Helper:
      Iterating in objIter over object at address 0x40b02600 (seems to be real).
      The reference that caused the crash is a normal field.
      Scanned object [0000000040b02600 0000000040b02618) of class java/lang/String
      0x40b02600 : 0x00539620 [classblock]
      0x40b02604 : 0x40000000 [flags]
      ===> 0x40b02608 : 0x00086a3a .value [C (java/lang/String)
      0x40b0260c : 0x00000000
      0x40b02610 : 0x00000000
      0x40b02614 : 0x00000000
      The field that caused the crash is marked with "===>"

      The reference is bogus and pointing outside the heap.

      First reference to this object from another object is
      object [0000000040b29af8 0000000040b29b30) of class java/net/URL
      0x40b29af8 : 0x0009f990 [classblock]
      0x40b29afc : 0x0000018c [flags]
      0x40b29b00 : 0x40b29940 .protocol Ljava/lang/String; (java/net/URL)
      ===> 0x40b29b04 : 0x40b02600 .host Ljava/lang/String; (java/net/URL)
      0x40b29b08 : 0x40b29958 .file Ljava/lang/String; (java/net/URL)
      0x40b29b0c : 0x00000000 .query Ljava/lang/String; (java/net/URL)
      0x40b29b10 : 0x40b02600 .authority Ljava/lang/String; (java/net/URL)
      0x40b29b14 : 0x40b29958 .path Ljava/lang/String; (java/net/URL)
      0x40b29b18 : 0x00000000 .userInfo Ljava/lang/String; (java/net/URL)
      0x40b29b1c : 0x00000000 .ref Ljava/lang/String; (java/net/URL)
      0x40b29b20 : 0x00000000 .hostAddress Ljava/net/InetAddress; (java/net/URL)
      0x40b29b24 : 0x40b2c5c8 .handler Ljava/net/URLStreamHandler; (java/net/URL)
      0x40b29b28 : 0xffffffff
      0x40b29b2c : 0xbb73ccd5



      "(GC Worker Thread 1)" id=? idx=0x10 tid=32679 lastJavaFrame=0xfffffffffffffffc

      Stack 0: start=0x40a85000, end=0x40ac7000, guards=0x40a8a000 (ok), forbidden=0x40a88000
      Thread Stack Trace:
      at mmParMarkWorkers+615()@0x2aaaaac263ff
      at mmGCWorkerThread+144()@0x2aaaaab4f1c9
      at tsiCallStartFunction+67()@0x2aaaaabc9e33
      at tsiThreadStub+308()@0x2aaaaabcb03d
      at ptiThreadStub+14()@0x2aaaaac32692
      at start_thread+199()@0x3b06406307
      -- Java stack --

      Extended, platform specific info:
      libc release: 2.5-stable
      Elf headers:
      libc ehdrs: EI: 7f454c46020101000000000000000000 ET: 3 EM: 62 V: 1 ENTRY: 0000003b0581d9b0 PHOFF: 0000000000000040 SHOFF: 000000000019dce8 EF: 0x0 HS: 64 PS: 56 PHN; 10 SS: 64 SHN: 77 STIDX: 76
      libpthread ehdrs: EI: 7f454c46020101000000000000000000 ET: 3 EM: 62 V: 1 ENTRY: 0000003b06405740 PHOFF: 0000000000000040 SHOFF: 0000000000021ed0 EF: 0x0 HS: 64 PS: 56 PHN; 9 SS: 64 SHN: 39 STIDX: 38
      libjvm ehdrs: EI: 7f454c46020101000000000000000000 ET: 3 EM: 62 V: 1 ENTRY: 000000000002de68 PHOFF: 0000000000000040 SHOFF: 000000000252a9e8 EF: 0x0 HS: 64 PS: 56 PHN; 3 SS: 64 SHN: 30 STIDX: 27

      **********************************************************
      * If you see this dump, please go to *
      * http://edocs.bea.com/jrockit/go2troubleshooting.html *
      * for troubleshooting information. *
      **********************************************************

      ===== END DUMP ===============================================================