2 Replies Latest reply on Feb 20, 2004 3:51 AM by nwc

    JBoss 3.2.3 crashing and hanging unpredictably

    nwc
    nwc Feb 12, 2004 4:37 PM

      We recently began experiencing crashing and hanging of JBoss 3.2.3 in our production environment at unpredictable time intervals--anywhere from 30 minutes to 12 hours after re-running the app. This began after we upgraded our mysql connector from 3.0.6 to 3.0.10, and made several minor application changes. We have sinced rolled back all of those changes, but continue to experience the same crashing and hanging. Sometimes it hangs with one CPU maxed out, sometimes it hangs with both CPUs idle (we have a two CPU machine). Sometimes it kills itself with no core dump and no trace in the logs, and sometimes it kills itself by shutting itself down and printing this to server.log:

      2004-02-12 09:16:29,755 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Terminate
      2004-02-12 09:16:29,755 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Terminate
      2004-02-12 09:16:29,755 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Terminate
      2004-02-12 09:16:29,755 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Terminate
      2004-02-12 09:16:29,756 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Terminate
      2004-02-12 09:16:29,756 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Terminate
      2004-02-12 09:16:29,765 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Terminate
      2004-02-12 09:16:29,766 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Terminate
      2004-02-12 09:16:29,766 DEBUG [org.apache.tomcat.util.threads.ThreadPool] Terminate

      Our configuration is like this:

      JBoss 3.2.3

      java version "1.4.2_03"
      Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_03-b02)
      Java HotSpot(TM) Client VM (build 1.4.2_03-b02, mixed mode)

      Linux version 2.4.21-4.0.1.ELsmp (bhcompile@bugs.devel.redhat.com) (gcc version 3.2.3 20030502 (Red Hat Linux 3.2.3-20)) #1 SMP Thu Oct 23 01:21:42 EDT 2003

      mysql-connector-java-3.0.10-stable-bin.jar

      mysql Ver 11.18 Distrib 3.23.58, for redhat-linux-gnu (i386)

      This configuration had worked fine for us (except using the 3.0.6 mysql connector) for several weeks without incident.

      • 2855 Views
      • Tags:
        • 1. Re: JBoss 3.2.3 crashing and hanging unpredictably
          adrian.brock Feb 18, 2004 6:31 AM (in response to nwc)

          You are in the wrong forum.

          export LD_ASSUME_KERNEL=2.4.1

          Regards,
          Adrian

            • Actions
          • 2. Re: JBoss 3.2.3 crashing and hanging unpredictably
            nwc
            nwc Feb 20, 2004 3:51 AM (in response to nwc)

             

            "adrian@jboss.org" wrote:
            You are in the wrong forum.


            I put it here because I initially suspected the mysql connector.

            The export setting did not work (we had tried that).

            We did fix the problem, however. After a week, someone finally noticed hs_err_pid files being created all over the place. These files clearly showed that Thread.countStackFrames() was killing the JVM. Evidently this method is no longer supported by Sun in JDK 1.4.2_03, and to underscore this Sun devised a clever algorithm by which the JVM dies every 1000th or so time this method is called. Our 3 second monitoring ping happened to trigger this method, and so we were dying about once an hour.

              • Actions