10 Replies Latest reply on Jun 12, 2015 6:11 AM by Aidan Diffey

    JBoss EAP 6 stops without error

    Aidan Diffey Newbie

      I have an application that is running using JBoss EAP 6 that often stops without error or warning.  It is running on a SuSE Linux machine and when it stops, the process is still listed on the OS, but the jboss-cli does not connect and I can get no response from JConsole. Is there any additional debugging on JBoss or the VM that I can enable to help diagnose the issue?

       

      JBoss EAP 6 is running with the following VM options (Taken using JConsole):

       

      Connection name:

      pid: 11013 jboss-modules.jar -mp /usr/local/jboss-eap-6.3/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=/usr/local/jboss-eap-6.3 -Djboss.server.base.dir=/usr/local/jboss-eap-6.3/standalone -c standalone-full.xml -b <hostname>

      Virtual Machine:

      OpenJDK 64-Bit Server VM version 24.65-b04

      Vendor:

      Oracle Corporation

      Name:

      11013@<hostname>

      Uptime:

      1 hour 43 minutes

      Process CPU time:

      2 hours 15 minutes

      JIT compiler:

      HotSpot 64-Bit Tiered Compilers

      Total compile time:

      1 minute

       

      Live threads:

        391

      Peak:

        479

      Daemon threads:

        145

      Total threads started:

      1,064

      Current classes loaded:

      17,714

      Total classes loaded:

      17,744

      Total classes unloaded:

          30

       

      Current heap size:

      6,780,095 kbytes

      Maximum heap size:

      10,417,664 kbytes

      Committed memory:

      10,417,664 kbytes

      Pending finalization:

      0 objects

      Garbage collector:

      Name = 'ParNew', Collections = 3,095, Total time spent = 1 minute

      Garbage collector:

      Name = 'ConcurrentMarkSweep', Collections = 7, Total time spent = 0.672 seconds

       

      Operating System:

      Linux 3.12.28-4-default

      Architecture:

      amd64

      Number of processors:

      8

      Committed virtual memory:

      15,730,224 kbytes

      Total physical memory:

      24,629,928 kbytes

      Free physical memory:

      1,705,912 kbytes

      Total swap space:

      4,192,252 kbytes

      Free swap space:

      3,781,008 kbytes

       

      VM arguments:

      -D[Standalone] -XX:+UseCompressedOops -verbose:gc -Xloggc:/usr/local/jboss-eap-6.3/standalone/log/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M -XX:-TraceClassUnloading -Xms10240m -Xmx10240m -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/jboss-eap-6.3/standalone/log/jboss.hprof -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.modules.policy-permissions=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000: -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -XX:+DisableExplicitGC -XX:MaxPermSize=256m -Dorg.jboss.boot.log.file=/usr/local/jboss-eap-6.3/standalone/log/server.log -Dlogging.configuration=file:/usr/local/jboss-eap-6.3/standalone/configuration/logging.properties

      Class path:

      /usr/local/jboss-eap-6.3/jboss-modules.jar

      Library path:

      /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

      Boot class path:

      /usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/jre/lib/resources.jar:/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/jre/lib/rt.jar:/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/jre/lib/sunrsasign.jar:/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/jre/lib/jsse.jar:/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/jre/lib/jce.jar:/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/jre/lib/charsets.jar:/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/jre/lib/rhino.jar:/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/jre/lib/jfr.jar:/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/jre/classes

        • 1. Re: JBoss EAP 6 stops without error
          Lyle Wang Novice

          Do you use "nohup" to run JBoss so that when OS user logout it doesn't get killed ?

          Do you see any "*** shutdown ***" or "*** server exit ***" messages in your log before shutting down ?

          Do you have anything like "System.exit(0)" in your application code ?

          • 2. Re: JBoss EAP 6 stops without error
            Aidan Diffey Newbie

            Hello.

             

            Thank you for your reply.

             

            I am starting JBoss with "nohup" from the command line. When it hangs I can still see the -D[Standalon] process when I do a "ps -ef | grep java" so I assume the process is still running.

            I cannot see any "shutdown" or like messages but I did experience strange behavior when logging.  I enabled Verbose GC information and I noticed that one time when it hung it managed to write only half the GC information to the file.  It was as if the buffer had been written to but never flushed.

            Finally, there are no System.exit(0) in the code.

            • 3. Re: JBoss EAP 6 stops without error
              Tomaz Cerar Master

              check /var/log/messages for kernel oom killer messages...

              • 4. Re: JBoss EAP 6 stops without error
                Aidan Diffey Newbie

                Hello.

                 

                The application has once again stopped (2015-06-05T13:16:09). This time I have not killed the process.  It would appear that there was a correlation between the hanging of JBoss and a spike (and continued) high CPU usage.

                 

                I have managed to get some stack information using jstack but I had to use the -F switch (to force the stack) as the -l couldn't connect to the process.

                Can you offer any assistance in understanding the jstack output?

                 

                 

                 

                Also, using jstack with -m shows the following error:

                ----------------

                Attaching to process ID 3673, please wait...

                Debugger attached successfully.

                Server compiler detected.

                JVM version is 24.65-b04

                sun.jvm.hotspot.utilities.AssertionFailure: must be a valid non-zero index

                        at sun.jvm.hotspot.utilities.Assert.that(Assert.java:32)

                        at sun.jvm.hotspot.code.NMethod.getOopAt(NMethod.java:203)

                        at sun.jvm.hotspot.code.DebugInfoReadStream.readOopHandle(DebugInfoReadStream.java:53)

                        at sun.jvm.hotspot.code.ScopeDesc.<init>(ScopeDesc.java:64)

                        at sun.jvm.hotspot.code.ScopeDesc.sender(ScopeDesc.java:121)

                        at sun.jvm.hotspot.runtime.CompiledVFrame.sender(CompiledVFrame.java:182)

                        at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146)

                        at sun.jvm.hotspot.tools.PStack.initJFrameCache(PStack.java:219)

                        at sun.jvm.hotspot.tools.PStack.run(PStack.java:67)

                        at sun.jvm.hotspot.tools.PStack.run(PStack.java:54)

                        at sun.jvm.hotspot.tools.PStack.run(PStack.java:49)

                        at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)

                        at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)

                        at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)

                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                        at java.lang.reflect.Method.invoke(Method.java:606)

                        at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)

                        at sun.tools.jstack.JStack.main(JStack.java:102)

                • 5. Re: JBoss EAP 6 stops without error
                  Aidan Diffey Newbie

                  The process has come back to life again!  I performed a jmap operation but it took a couple of attempts to attach. In the end, I used a -F flag.

                  JConsole is now telling me that the total time spent performing GC operation was 2 days, 21 hours 15 minutes but I don't think it was performing a GC for that long as I can't see anything in the GC output that would suggest having a memory issue.

                   

                  Looking at the gc log in GCViewer I can see what it thinks is a GC pause, but after the pause the used heap actually increased before dropping again on the next collection.

                  • 6. Re: JBoss EAP 6 stops without error
                    Aidan Diffey Newbie

                    Hello.

                     

                    I have seen another JBoss hang, and I managed to get it to resume by performing a jstack -F on the JBoss PID.  After that JBoss resumed where it left off with no warnings, errors or any change in the amount of memory it was using.

                     

                    Do you have any suggestion as to why it would do this?

                    • 7. Re: JBoss EAP 6 stops without error
                      Aidan Diffey Newbie

                      More information.

                      I have had the "hang" again and this time I was able to run GDB and get a core dump out.  Using the command "gdb /usr/java/default/bin/java core.19819" the final line is #0  0x00007f90e9c6705f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

                       

                      Further looking around the interweb shows that this could be an issue with some native library used by JBoss. 

                       

                      Do anyone have any idea on how I can find any more information?

                      • 8. Re: JBoss EAP 6 stops without error
                        Tomaz Cerar Master

                        well, jboss doesn't use any native libs out of the box,

                        there are some add-ons for jbossweb ssl stuff but you need to install that manually.

                         

                        what you are seeing is pure native issue on linux with pthread library and kernel.

                         

                        I would try upgrading to latest version OS patches for start, than try with newer version of JDK

                         

                        could also be related to Google Groups

                        • 9. Re: JBoss EAP 6 stops without error
                          Aidan Diffey Newbie

                          Thanks for the response.  It will take some reading through the Google Groups but the post titled "jstack (with -F), jhat, strace, gdb will attach and then detach from the process. Upon detach the process wakes up, invariably." Does appear to be exactly what I am experiencing.

                           

                          If I understand correctly, you're saying that this is likley to be an issue with Linux or the version of OpenJDK and not JBoss itself.  This would stack up as it is only when I connect to the process using jstack -F or in the last case, with gdb that the JBoss process springs back into life.  interestingly enough it is when the debugger disconnects from the process that it starts running again.  This is sort of reflected in the follow stack overflow post: http://stackoverflow.com/questions/12061226/how-does-jstack-f-affect-a-running-java-process

                           

                           

                          We are currently using SLES 12.0 and OpenJDK:

                           

                          java version "1.7.0_65"

                          OpenJDK Runtime Environment (IcedTea 2.5.2) (suse-3.7-x86_64)

                          OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)

                          • 10. Re: JBoss EAP 6 stops without error
                            Aidan Diffey Newbie

                            Hello.

                             

                            Once again I must say thanks to Tomaz for linking to Google Groups.  It appears that there is a bug in the Linux kernel that prevents a parked thread from being woken up.  The jstack -F command when attach / detach wakes the thread up.

                             

                            I will try updating the Kernel version to the latest version (3.12.39-47.1 for SuSE 12) which has the fix in it.

                             

                            For reference, a link to the github item: https://github.com/torvalds/linux/commit/76835b0ebf8a7fe85beb03c75121419a7dec52f0