If it's just saying "Terminated" that looks like the JVM might be crashing. What OS are you using and what version of WildFly?
James R. Perkins
Using RHEL 6.8 with WildFly 8.0 Final. Seen the same behavior with OpenJDK 1.7 and Oracle JDK 1.7.
My HSQLDB, running in a different process, is also crashing/stopping at the same time.
Do you see any hs_err* logs? I'd check the syslog messages too.
James R. Perkins
Did not see any hs_err* logs. I'll have to double check the syslog, but don't recall seeing anything out of place. I also made sure to turn on the –XX:+HeapDumpOnOutOfMemoryError flag given to java.
Are there any loggers that can be turned on to see WildFly's shutdown process? I've gone so far as to set "org.jboss" to DEBUG and not gotten much. Tried TRACE, but it brought my system to a grinding halt with the volume of messages.
The most unusual thing is that this error doesn't happen instantly after running my web app, but once in this state I can reboot my WildFly server and get it to happen almost immediately. It takes a physical reboot of my PC to get it to run for a bit longer. Is there maybe some weird caching thing going on? Are there any caches I could try clearing when it happens so I don't have to reboot my PC?
RHEL 6.8? that was never released, at least not yet, do you maybe mean 5.8
anyhow, this looks like kernel oom killer is killing off biggest memory consumer.
i would take a look into system log
see http://linux-mm.org/OOM_Killer for more.
if this is happening you should see something about it in system log.
Another question, how are you starting these processes that are dying?
Another common cause of dying processes on Linux is where users connect remotely and start the process in the background and then all processes they started are killed after they disconnect - in that case the best strategy is to change to start the processes using nohup.
We're using RHEL 6.5. Typoed the version, sorry. Had WildFly 8 on the mind I guess...
After some investigation, it does not appear to be the Linux OoM killer, although reading your link did it make it sound like my WildFly instance might be a good candidate. Grepping /var/log/messages* did not show the kernel killing anything. I've also turned on kernel panics when OoM and not seen anything.
@Darran: We use nohup when launching our WildFly and HSQLDB processes. I've also seen this behavior running standalone.sh in a terminal and keeping that open to watch the log in real-time.
I don't recollect "Terminated" being part of WildFly shutdown log messages. But it's been a while since I have paid attention to the logs. Can you post the exact log messages (a few relevant lines when you start seeing the shutdown)?
"terminated" is displayed in log when jvm process is forcefully killed.
basically something like kill -9 pid would do
why it would be forcefully killed is a different question and it is related to OS not to wildfly itself