1 2 Previous Next 27 Replies Latest reply on Nov 26, 2002 3:54 PM by asperkins

    JBoss 3.0.3(jetty) jvm (1.4.0/1.4.1) crash on Solaris/Window

    asperkins

      I've recreated this on 2 different versions of solaris and windows 2000. Using 2 different versions of the jvm.

      Solaris 2.6 and 8
      Windows 2000 pro
      JVM 1.4.0
      JVM 1.4.1

      Here are the java options I used (still cored)
      -Xms512m -Xmx1024m -server

      I'm currently testing with -Xint, and so far it has not crashed yet. I'd prefer not to have to run in interp mode.


      Under heavy load the JVM crashes.

      I get a core dump(on solaris) and this file:

      "hs_err_pid6162.log" 40 lines, 1492 characters

      An unexpected exception has been detected in native code outside the VM.
      Unexpected Signal : 11 occurred at PC=0x0
      Function=[Unknown.]
      Library=(N/A)

      NOTE: We are unable to locate the function name symbol for the error
      just occurred. Please refer to release documentation for possible
      reason and solutions.


      Current Java thread:

      Dynamic libraries:
      0x10000 /export/home/tperkins/j2sdk1.4.1_01//bin/java
      0xff350000 /usr/lib/libthread.so.1
      0xff390000 /usr/lib/libdl.so.1
      0xff200000 /usr/lib/libc.so.1
      0xff340000 /usr/platform/SUNW,Sun-Fire-880/lib/libc_psr.so.1
      0xfe000000 /export/home/tperkins/j2sdk1.4.1_01/jre/lib/sparc/server/libjvm.
      so
      0xff2e0000 /usr/lib/libCrun.so.1
      0xff1e0000 /usr/lib/libsocket.so.1
      0xff100000 /usr/lib/libnsl.so.1
      0xff0d0000 /usr/lib/libm.so.1
      0xff310000 /usr/lib/libw.so.1
      0xff0b0000 /usr/lib/libmp.so.2
      0xff080000 /export/home/tperkins/j2sdk1.4.1_01/jre/lib/sparc/native_threads
      /libhpi.so
      0xff050000 /export/home/tperkins/j2sdk1.4.1_01/jre/lib/sparc/libverify.so
      0xfe7c0000 /export/home/tperkins/j2sdk1.4.1_01/jre/lib/sparc/libjava.so
      0xff030000 /export/home/tperkins/j2sdk1.4.1_01/jre/lib/sparc/libzip.so
      0xfe6b0000 /usr/lib/locale/en_US.ISO8859-1/en_US.ISO8859-1.so.2
      0xfa150000 /export/home/tperkins/j2sdk1.4.1_01/jre/lib/sparc/libnet.so

      Local Time = Mon Oct 28 13:41:22 2002
      Elapsed Time = 325
      #
      # The exception above was detected in native code outside the VM
      #
      # Java VM: Java HotSpot(TM) Server VM (1.4.1_01-b01 mixed mode)
      #

        • 1. Re: JBoss 3.0.3(jetty) jvm (1.4.0/1.4.1) crash on Solaris/Wi

          This is a JVM bug, you need to report it to Sun.

          • 2. Re: JBoss 3.0.3(jetty) jvm (1.4.0/1.4.1) crash on Solaris/Wi
            asperkins

            Are there any work arounds?

            Does this mean that if you expect to have a heavy load that you can not use 1.4 or 1.4.1?

            • 3. Re: JBoss 3.0.3(jetty) jvm (1.4.0/1.4.1) crash on Solaris/Wi
              wpfeiffe

              We are seeing a very similar thing on Win2k Server with JBoss/Jetty 3.0.2 (and 3.0.3) using 1.4.0.02 as well as 1.4.1 jdks. Its been two days and we have not been able to bring our server up under load.

              We get no exceptions and no OS errors. The jvm just shuts down. Even JBoss doesn't do its normal clean shutdown. The jvm just stops.

              Is this consistant with what you are seeing?

              Thanks for any info.

              • 4. Re: JBoss 3.0.3(jetty) jvm (1.4.0/1.4.1) crash on Solaris/Wi
                asperkins

                I see the error like I posted above on Solaris. On windows I get the dialog box that says java has created an error... a report is being..., or something like that.

                • 5. Re: JBoss 3.0.3(jetty) jvm (1.4.0/1.4.1) crash on Solaris/Wi
                  asperkins

                  I have not been able to recreate this problem with Tomcat stand-alone. If it is a jvm bug, it looks like it's a bug that jboss hits.

                  • 6. Re: JBoss 3.0.3(jetty) jvm (1.4.0/1.4.1) crash on Solaris/Wi
                    wpfeiffe

                    At least you are seeing something reported from the OS. What we are experiencing is almost as if someone (or something) killed the process immediately with no error trail or execptions or GPF, etc.

                    I don't even know how to begin to report this. And to who?

                    Thanks again for any info.

                    • 7. Re: JBoss 3.0.3(jetty) jvm (1.4.0/1.4.1) crash on Solaris/Wi
                      asperkins

                      I'm currently running jmeter against tomcat stand-alone. I have 75 threads running 100 request each. Halfway through without a problem.

                      When we get over 10 threads hitting jboss it core dumps.

                      Anyone have a solution?

                      I'd much rather go to production with JBoss-jetty than tomcat stand-alone.

                      Thanks

                      • 8. Re: JBoss 3.0.3(jetty) jvm (1.4.0/1.4.1) crash on Solaris/Wi
                        asperkins

                        I've finished a 75 thread 100 each run with tomcat with no problems. About to finish running 75 threads for 200 hits each. 15,000 hits with no problems.

                        I feel pretty confident in saying that the problem/bug/whatever is in, or has been found by, JBoss.

                        • 9. Re: JBoss 3.0.3(jetty) jvm (1.4.0/1.4.1) crash on Solaris/Wi
                          asperkins

                          Forgot to mention that I tried JBoss3.0.3 with tomcat 4.1.12 and it core dumpted also.

                          Tomcat 4.1.12 by itself seems to work fine tho.

                          • 10. Re: JBoss 3.0.3(jetty) jvm (1.4.0/1.4.1) crash on Solaris/Wi
                            mosborn

                            I know there have to be more people watching this thread than replying, so this is in the spirit of a broadcast. And yes, we will submit a bug report to Sun as soon as we have a moment to breath.

                            We're preparing a website application (MySQL/database access and no EJBs) and have experienced everything mentioned above. We used new 1.4 language features in the app and became quite stuck. We're close to or at a reasonable solution, having learned the following:

                            We tried all sorts of OS configurations and updates, and nothing seemed to help.

                            Crashing "under load" may be misleading. If I let the app run long enough, it crashes under light or almost non-existent load. (Although I agree with the first post indicating 35-40 threads being a fast path to ruination. We're using jmeter, too ;) ).

                            I get the behavior on my W2K laptop where Netbeans runs happily for days with the same JVM.

                            A quick predictor of unstable behavior is when memory use jumps up (in Task Manager or top under Linux) on every request, even though you're reasonably sure no new web server threads needed to be spawned for it. The memory is NOT reclaimed/GCed no matter how long you wait.

                            The abrupt halt in Linux/Solaris seems to be the interaction with the shell. It does essentially the same thing in Windows when run under Cygwin - you get a "java.exe error" dialog box. However, when running from a DOS shell, I get a happy out of memory error (albeit with a JBoss shutdown). Watching more closely after seeing this, I noticed memory use did actually spike way up over whatever limit you feed the JVM (spiking very quickly and briefly).

                            On Win2K (by far the most stable/forgiving JVM), the app is much harder to crash. For instance, using a JBoss/MySQL app, it was happy under load for W2K but crashed quickly under Linux/Solaris. However, after migrating MySQL tables from MyISAM to InnoDB, it crashed in Windows, too. In fact, in one of our crashes the error output indicated a database-related thread, so the problem may be around there. On that note, we still need to review our database code soon to make it as well-behaved as possible (even though it is routine stuff right now).

                            Now for the happy part - I think we finally managed a stable Windows configuration by compiling JBoss under 1.4.1. (I think the binary distro is a 1.3.1 product). Linux became much happier and stayed up longer, but crashed eventually. Haven't tried Solaris yet.

                            If you haven't done this yet, it's extremely easy if and only if you follow the "Source Code" page directions to grab it from CVS. I never could get the snapshot tarball to work. I got the 3.0.3 branch with "-r JBoss_3_0_3". Just cd to the build dir and run build.bat or build.sh; works every time!

                            If this is a jvm cross-version compatibility issue, my next casual question is: Does the JBoss build compile everything in the distro from source, or does it package certain libraries (e.g. Jakarta stuff) from their binary distibutions? I'll answer this for myself soon, but if someone knows I appreciate the shortcut. I still need to go back through all extra libraries and drivers our app using and compile them. Maybe Linux will settle down then, too.

                            Hope this helps someone!

                            • 11. Re: JBoss 3.0.3(jetty) jvm (1.4.0/1.4.1) crash on Solaris/Wi
                              asperkins

                              THANKS!

                              I didn't even think about building jboss under 1.4.1.

                              • 12. Re: JBoss 3.0.3(jetty) jvm (1.4.0/1.4.1) crash on Solaris/Wi
                                asperkins

                                Bad news. :(

                                Rebuilt JBoss with 1.4.1 . . . Still core dumps :(:(:(

                                All tests with tomcat stand-alone have been 100% successful.

                                • 13. Re: JBoss 3.0.3(jetty) jvm (1.4.0/1.4.1) crash on Solaris/Wi
                                  mosborn

                                  Oh, one more thing I forgot. Especially if you're using jsp includes, esp. nested jsp includes, they must be precompiled (thank you Ant). That was an original source of crashes for us under Windows.

                                  What can you say about the architecture/configuration of your application?

                                  • 14. Re: JBoss 3.0.3(jetty) jvm (1.4.0/1.4.1) crash on Solaris/Wi


                                    > If this is a jvm cross-version compatibility issue,
                                    > my next casual question is: Does the JBoss build
                                    > compile everything in the distro from source, or does
                                    > it package certain libraries (e.g. Jakarta stuff)
                                    > from their binary distibutions?

                                    We don't compile third party libraries.

                                    1 2 Previous Next