Wildfly 8.2 process go down without any logs NEW MESSAGES

[lukasw44]

After we started out WildFly server on production we have some serious issues with stability. WildFly proccess simply shuts down several times per day. We don't have anything in logs that could lead us to source of this issue also we dodn't notice any issues with memory or CPU. Do you have any idea what can we do to find the source of this issue?

[tomekadamski]

Hi Lukasz,

Have you tried looking at operating system logs to check who is killing the process?

Please check your kernel message buffer to see if there is some information about it:

dmesg | egrep -i -B100 'killed process'

Tomek

[mayerw01]

Under which operating system is this WildFly server running?

[lukasw44]

Wildfly is running on Linux Cent os 6

[mayerw01]

Are you using openjdk, which version?

[lukasw44]

No i am using oracle java implementation :

java version "1.8.0_40"

Java(TM) SE Runtime Environment (build 1.8.0_40-b26)

Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)

[lukasw44]

dmesg | egrep -i -B100 'killed process'

How can i run that command on centos ?

[tomekadamski]

Hi Lukasz,

What error do you have when you try to run it?

Tomek

[lukasw44]

I have got error in wildfly/bin : hs_err_pid3212.log

#

# A fatal error has been detected by the Java Runtime Environment:

#

#  Internal Error (sharedRuntime.cpp:834), pid=32127, tid=139730835359488

#  fatal error: exception happened outside interpreter, nmethods and vtable stubs at pc 0x00007f15ad053b6f

#

# JRE version: Java(TM) SE Runtime Environment (8.0_40-b26) (build 1.8.0_40-b26)

# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode linux-amd64 compressed oops)

# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

#

# If you would like to submit a bug report, please visit:

#  http://bugreport.java.com/bugreport/crash.jsp

#

---------------  T H R E A D  ---------------

Current thread (0x00007f156d025000):  JavaThread "default task-48" [_thread_in_Java, id=32443, stack(0x00007f159ebe4000,0x00007f159ece5000)]

Stack: [0x00007f159ebe4000,0x00007f159ece5000],  sp=0x00007f159ecdfb10,  free space=1006k

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

V  [libjvm.so+0xaaca9a]  VMError::report_and_die()+0x2ba

V  [libjvm.so+0x4f2de9]  report_fatal(char const*, int, char const*)+0x59

V  [libjvm.so+0x9ab6ba]  SharedRuntime::continuation_for_implicit_exception(JavaThread*, unsigned char*, SharedRuntime::ImplicitExceptionKind)+0x33a

V  [libjvm.so+0x91501a]  JVM_handle_linux_signal+0x48a

V  [libjvm.so+0x90b593]  signalHandler(int, siginfo*, void*)+0x43

C  [libpthread.so.0+0xf710]

J 18881 C2 com.sun.crypto.provider.CipherCore.finalNoPadding([BI[BII)I (189 bytes) @ 0x00007f15b0202074 [0x00007f15b01ffa40+0x2634]

All logs file is attached:

[jaikiran]

Looks like the JVM is crashing the crypto code:

com.sun.crypto.provider.CipherCore

Have you tried a latest update of JDK 8 (I think it's 70) to see if it's fixed in there.