JBoss EAP 6 stops without error

[ramiseesniblic]

I have an application that is running using JBoss EAP 6 that often stops without error or warning.  It is running on a SuSE Linux machine and when it stops, the process is still listed on the OS, but the jboss-cli does not connect and I can get no response from JConsole. Is there any additional debugging on JBoss or the VM that I can enable to help diagnose the issue?

JBoss EAP 6 is running with the following VM options (Taken using JConsole):

Connection name:

pid: 11013 jboss-modules.jar -mp /usr/local/jboss-eap-6.3/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=/usr/local/jboss-eap-6.3 -Djboss.server.base.dir=/usr/local/jboss-eap-6.3/standalone -c standalone-full.xml -b <hostname>

Virtual Machine:

OpenJDK 64-Bit Server VM version 24.65-b04

Vendor:

Oracle Corporation

Name:

11013@<hostname>

Uptime:

1 hour 43 minutes

Process CPU time:

2 hours 15 minutes

JIT compiler:

HotSpot 64-Bit Tiered Compilers

Total compile time:

1 minute

Live threads:

  391

Peak:

  479

Daemon threads:

  145

Total threads started:

1,064

Current classes loaded:

17,714

Total classes loaded:

17,744

Total classes unloaded:

    30

Current heap size:

6,780,095 kbytes

Maximum heap size:

10,417,664 kbytes

Committed memory:

10,417,664 kbytes

Pending finalization:

0 objects

Garbage collector:

Name = 'ParNew', Collections = 3,095, Total time spent = 1 minute

Garbage collector:

Name = 'ConcurrentMarkSweep', Collections = 7, Total time spent = 0.672 seconds

Operating System:

Linux 3.12.28-4-default

Architecture:

amd64

Number of processors:

8

Committed virtual memory:

15,730,224 kbytes

Total physical memory:

24,629,928 kbytes

Free physical memory:

1,705,912 kbytes

Total swap space:

4,192,252 kbytes

Free swap space:

3,781,008 kbytes

VM arguments:

-D[Standalone] -XX:+UseCompressedOops -verbose:gc -Xloggc:/usr/local/jboss-eap-6.3/standalone/log/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M -XX:-TraceClassUnloading -Xms10240m -Xmx10240m -XX:-HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/jboss-eap-6.3/standalone/log/jboss.hprof -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.modules.policy-permissions=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000: -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -XX:+DisableExplicitGC -XX:MaxPermSize=256m -Dorg.jboss.boot.log.file=/usr/local/jboss-eap-6.3/standalone/log/server.log -Dlogging.configuration=file:/usr/local/jboss-eap-6.3/standalone/configuration/logging.properties

Class path:

/usr/local/jboss-eap-6.3/jboss-modules.jar

Library path:

/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

Boot class path:

/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/jre/lib/resources.jar:/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/jre/lib/rt.jar:/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/jre/lib/sunrsasign.jar:/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/jre/lib/jsse.jar:/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/jre/lib/jce.jar:/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/jre/lib/charsets.jar:/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/jre/lib/rhino.jar:/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/jre/lib/jfr.jar:/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/jre/classes

[lylewang]

Do you use "nohup" to run JBoss so that when OS user logout it doesn't get killed ?

Do you see any "*** shutdown ***" or "*** server exit ***" messages in your log before shutting down ?

Do you have anything like "System.exit(0)" in your application code ?

[ramiseesniblic]

Hello.

Thank you for your reply.

I am starting JBoss with "nohup" from the command line. When it hangs I can still see the -D[Standalon] process when I do a "ps -ef | grep java" so I assume the process is still running.

I cannot see any "shutdown" or like messages but I did experience strange behavior when logging.  I enabled Verbose GC information and I noticed that one time when it hung it managed to write only half the GC information to the file.  It was as if the buffer had been written to but never flushed.

Finally, there are no System.exit(0) in the code.

[ctomc]

check /var/log/messages for kernel oom killer messages...

[ramiseesniblic]

Hello.

The application has once again stopped (2015-06-05T13:16:09). This time I have not killed the process.  It would appear that there was a correlation between the hanging of JBoss and a spike (and continued) high CPU usage.

I have managed to get some stack information using jstack but I had to use the -F switch (to force the stack) as the -l couldn't connect to the process.

Can you offer any assistance in understanding the jstack output?

Also, using jstack with -m shows the following error:

----------------

Attaching to process ID 3673, please wait...

Debugger attached successfully.

Server compiler detected.

JVM version is 24.65-b04

sun.jvm.hotspot.utilities.AssertionFailure: must be a valid non-zero index

        at sun.jvm.hotspot.utilities.Assert.that(Assert.java:32)

        at sun.jvm.hotspot.code.NMethod.getOopAt(NMethod.java:203)

        at sun.jvm.hotspot.code.DebugInfoReadStream.readOopHandle(DebugInfoReadStream.java:53)

        at sun.jvm.hotspot.code.ScopeDesc.<init>(ScopeDesc.java:64)

        at sun.jvm.hotspot.code.ScopeDesc.sender(ScopeDesc.java:121)

        at sun.jvm.hotspot.runtime.CompiledVFrame.sender(CompiledVFrame.java:182)

        at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146)

        at sun.jvm.hotspot.tools.PStack.initJFrameCache(PStack.java:219)

        at sun.jvm.hotspot.tools.PStack.run(PStack.java:67)

        at sun.jvm.hotspot.tools.PStack.run(PStack.java:54)

        at sun.jvm.hotspot.tools.PStack.run(PStack.java:49)

        at sun.jvm.hotspot.tools.JStack.run(JStack.java:60)

        at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)

        at sun.jvm.hotspot.tools.JStack.main(JStack.java:86)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at sun.tools.jstack.JStack.runJStackTool(JStack.java:136)

        at sun.tools.jstack.JStack.main(JStack.java:102)

[ramiseesniblic]

The process has come back to life again!  I performed a jmap operation but it took a couple of attempts to attach. In the end, I used a -F flag.

JConsole is now telling me that the total time spent performing GC operation was 2 days, 21 hours 15 minutes but I don't think it was performing a GC for that long as I can't see anything in the GC output that would suggest having a memory issue.

Looking at the gc log in GCViewer I can see what it thinks is a GC pause, but after the pause the used heap actually increased before dropping again on the next collection.

[ramiseesniblic]

Hello.

I have seen another JBoss hang, and I managed to get it to resume by performing a jstack -F on the JBoss PID.  After that JBoss resumed where it left off with no warnings, errors or any change in the amount of memory it was using.

Do you have any suggestion as to why it would do this?

[ramiseesniblic]

More information.

I have had the "hang" again and this time I was able to run GDB and get a core dump out.  Using the command "gdb /usr/java/default/bin/java core.19819" the final line is #0  0x00007f90e9c6705f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

Further looking around the interweb shows that this could be an issue with some native library used by JBoss. 

Do anyone have any idea on how I can find any more information?

[ctomc]

well, jboss doesn't use any native libs out of the box,

there are some add-ons for jbossweb ssl stuff but you need to install that manually.

what you are seeing is pure native issue on linux with pthread library and kernel.

I would try upgrading to latest version OS patches for start, than try with newer version of JDK

could also be related to Google Groups

[ramiseesniblic]

Thanks for the response.  It will take some reading through the Google Groups but the post titled "jstack (with -F), jhat, strace, gdb will attach and then detach from the process. Upon detach the process wakes up, invariably." Does appear to be exactly what I am experiencing.

If I understand correctly, you're saying that this is likley to be an issue with Linux or the version of OpenJDK and not JBoss itself.  This would stack up as it is only when I connect to the process using jstack -F or in the last case, with gdb that the JBoss process springs back into life.  interestingly enough it is when the debugger disconnects from the process that it starts running again.  This is sort of reflected in the follow stack overflow post: http://stackoverflow.com/questions/12061226/how-does-jstack-f-affect-a-running-java-process

We are currently using SLES 12.0 and OpenJDK:

java version "1.7.0_65"

OpenJDK Runtime Environment (IcedTea 2.5.2) (suse-3.7-x86_64)

OpenJDK 64-Bit Server VM (build 24.65-b04, mixed mode)

[ramiseesniblic]

Hello.

Once again I must say thanks to Tomaz for linking to Google Groups.  It appears that there is a bug in the Linux kernel that prevents a parked thread from being woken up.  The jstack -F command when attach / detach wakes the thread up.

I will try updating the Kernel version to the latest version (3.12.39-47.1 for SuSE 12) which has the fix in it.

For reference, a link to the github item: https://github.com/torvalds/linux/commit/76835b0ebf8a7fe85beb03c75121419a7dec52f0