9 Replies Latest reply on Jul 29, 2014 8:40 AM by Tomaz Cerar

    WildFly shutdown reason

    Justin Robbins Newbie

      Is it possible to get visibility into why WildFly has stopped/shutdown?

       

      I'm experiencing a weird issue with my web application causing WildFly to stop with no indication as to why. My application does not appear to be generating any exceptions and WildFly only prints "Terminated" to server.log before undeploying services. I've also tried upping log levels to no avail.

        • 1. Re: WildFly shutdown reason
          James Perkins Master

          If it's just saying "Terminated" that looks like the JVM might be crashing. What OS are you using and what version of WildFly?

           

          --

          James R. Perkins

          • 2. Re: WildFly shutdown reason
            Justin Robbins Newbie

            Using RHEL 6.8 with WildFly 8.0 Final. Seen the same behavior with OpenJDK 1.7 and Oracle JDK 1.7.

             

            My HSQLDB, running in a different process, is also crashing/stopping at the same time.

            • 3. Re: WildFly shutdown reason
              James Perkins Master

              Do you see any hs_err* logs? I'd check the syslog messages too.

               

              --

              James R. Perkins

              • 4. Re: WildFly shutdown reason
                Justin Robbins Newbie

                Did not see any hs_err* logs. I'll have to double check the syslog, but don't recall seeing anything out of place. I also made sure to turn on the –XX:+HeapDumpOnOutOfMemoryError flag given to java.

                 

                Are there any loggers that can be turned on to see WildFly's shutdown process? I've gone so far as to set "org.jboss" to DEBUG and not gotten much. Tried TRACE, but it brought my system to a grinding halt with the volume of messages.

                 

                The most unusual thing is that this error doesn't happen instantly after running my web app, but once in this state I can reboot my WildFly server and get it to happen almost immediately. It takes a physical reboot of my PC to get it to run for a bit longer. Is there maybe some weird caching thing going on? Are there any caches I could try clearing when it happens so I don't have to reboot my PC?

                • 5. Re: WildFly shutdown reason
                  Tomaz Cerar Master

                  RHEL 6.8? that was never released, at least not yet, do you maybe mean 5.8

                   

                  anyhow, this looks like kernel oom killer is killing off biggest memory consumer.

                   

                  i would take a look into system log

                   

                  /etc/log/messages

                   

                  see http://linux-mm.org/OOM_Killer for more.

                   

                  if this is happening you should see something about it in system log.

                  • 6. Re: WildFly shutdown reason
                    Darran Lofthouse Master

                    Another question, how are you starting these processes that are dying? 

                     

                    Another common cause of dying processes on Linux is where users connect remotely and start the process in the background and then all processes they started are killed after they disconnect - in that case the best strategy is to change to start the processes using nohup.

                    • 7. Re: WildFly shutdown reason
                      Justin Robbins Newbie

                      We're using RHEL 6.5. Typoed the version, sorry. Had WildFly 8 on the mind I guess...

                       

                      After some investigation, it does not appear to be the Linux OoM killer, although reading your link did it make it sound like my WildFly instance might be a good candidate. Grepping /var/log/messages* did not show the kernel killing anything. I've also turned on kernel panics when OoM and not seen anything.

                       

                      @Darran: We use nohup when launching our WildFly and HSQLDB processes. I've also seen this behavior running standalone.sh in a terminal and keeping that open to watch the log in real-time.

                      • 8. Re: WildFly shutdown reason
                        jaikiran pai Master

                        I don't recollect "Terminated" being part of WildFly shutdown log messages. But it's been a while since I have paid attention to the logs. Can you post the exact log messages (a few relevant lines when you start seeing the shutdown)?

                        • 9. Re: WildFly shutdown reason
                          Tomaz Cerar Master

                          "terminated" is displayed in log when jvm process is forcefully killed.

                           

                          basically something like kill -9 pid would do

                           

                          why it would be forcefully killed is a different question and it is related to OS not to wildfly itself