WildFly shutdown reason

[justinrobbins1986]

Is it possible to get visibility into why WildFly has stopped/shutdown?

I'm experiencing a weird issue with my web application causing WildFly to stop with no indication as to why. My application does not appear to be generating any exceptions and WildFly only prints "Terminated" to server.log before undeploying services. I've also tried upping log levels to no avail.

[jamezp]

If it's just saying "Terminated" that looks like the JVM might be crashing. What OS are you using and what version of WildFly?

--

James R. Perkins

[justinrobbins1986]

Using RHEL 6.8 with WildFly 8.0 Final. Seen the same behavior with OpenJDK 1.7 and Oracle JDK 1.7.

My HSQLDB, running in a different process, is also crashing/stopping at the same time.

[jamezp]

Do you see any hs_err* logs? I'd check the syslog messages too.

--

James R. Perkins

[justinrobbins1986]

Did not see any hs_err* logs. I'll have to double check the syslog, but don't recall seeing anything out of place. I also made sure to turn on the –XX:+HeapDumpOnOutOfMemoryError flag given to java.

Are there any loggers that can be turned on to see WildFly's shutdown process? I've gone so far as to set "org.jboss" to DEBUG and not gotten much. Tried TRACE, but it brought my system to a grinding halt with the volume of messages.

The most unusual thing is that this error doesn't happen instantly after running my web app, but once in this state I can reboot my WildFly server and get it to happen almost immediately. It takes a physical reboot of my PC to get it to run for a bit longer. Is there maybe some weird caching thing going on? Are there any caches I could try clearing when it happens so I don't have to reboot my PC?

[ctomc]

RHEL 6.8? that was never released, at least not yet, do you maybe mean 5.8

anyhow, this looks like kernel oom killer is killing off biggest memory consumer.

i would take a look into system log

/etc/log/messages

see http://linux-mm.org/OOM_Killer for more.

if this is happening you should see something about it in system log.

[dlofthouse]

Another question, how are you starting these processes that are dying? 

Another common cause of dying processes on Linux is where users connect remotely and start the process in the background and then all processes they started are killed after they disconnect - in that case the best strategy is to change to start the processes using nohup.

[justinrobbins1986]

We're using RHEL 6.5. Typoed the version, sorry. Had WildFly 8 on the mind I guess...

After some investigation, it does not appear to be the Linux OoM killer, although reading your link did it make it sound like my WildFly instance might be a good candidate. Grepping /var/log/messages* did not show the kernel killing anything. I've also turned on kernel panics when OoM and not seen anything.

@Darran: We use nohup when launching our WildFly and HSQLDB processes. I've also seen this behavior running standalone.sh in a terminal and keeping that open to watch the log in real-time.

[jaikiran]

I don't recollect "Terminated" being part of WildFly shutdown log messages. But it's been a while since I have paid attention to the logs. Can you post the exact log messages (a few relevant lines when you start seeing the shutdown)?

[ctomc]

"terminated" is displayed in log when jvm process is forcefully killed.

basically something like kill -9 pid would do

why it would be forcefully killed is a different question and it is related to OS not to wildfly itself