11 Replies Latest reply on May 20, 2004 8:33 PM by jmer

    weird hang-ups

    mandrews-flarion

       

      "mandrews-flarion" wrote:
      hi there,

      i'm running 3.2.2/all on redhat linux 9.0, and keep getting weird hangups while my mbeans are running. what happens is logging stops, and the jmx-console stops responding via http as well. basically the whole java vm seems to pause permanently, with nothing running and the vm taking up 0% of the cpu. never indications anything went wrong in the logs, either.

      if i run jboss on the command line and watch stdout, during the hang-up ctrl-c does not exit the vm, so something seems amiss there. but when i do ctrl-z, which stops the java vm and exits to the linux shell, followed by 'fg' (to put the stopped process back into the foreground), my mbeans and logging start up again magically, and the jboss server starts to shutdown. so somehow the java vm recovers when going through the ctrl-z stopping process. weird!

      any ideas about what may be causing this crazy and deeply troubling behaviour? i'm using jdbc and xml/xslt in my mbeans, as well as running a multi-threaded custom tcp/ip server. i once thought this problem was a flawed jaxp implementation in sun-sdk-1.4.2, but the same thing happens sometimes without the jaxp (xslt).

      mike


        • 1. Re: weird hang-ups
          mandrews-flarion

          p.s.,

          another weird thing is if i run the jboss java vm in debug mode, and connect to it over a socket with jdb (sun's debugging client), first doing 'suspend' then 'resume' gets the vm out of its hung-up state and starts it running fine again. 'suspend' and 'resume' suspends and then resumes all threads in the vm. hopefully this information will 'ring a bell' with someone who can suggest a cause or solution.

          thanks!
          mike

          • 2. Re: weird hang-ups
            genman


            Can you get a stack trace (kill -3 pid) of the process?

            • 3. Re: weird hang-ups
              mandrews-flarion

              i will certainly plan to try, but what's interesting is the problem seems to have gone away, now that i switched JRE's from sun and started using IBM's 1.4.1 earlier today. this tends to confirm it was a buggy jre or some other low-level problem, and not the java bytecodes code per-se.

              so it seems i only run into the problem when running sun's 1.4.2 jre, and i will notify folks if i ever see anything similar with IBM's now that i'm using that one.

              thanks!
              mike

              • 4. Re: weird hang-ups
                mandrews-flarion

                folks,

                unfortunately the IBM vm hung up last night too, so it looks like a more basic problem than just a buggy vm as i originally thought. seems unlikely that a similar bug would be in two different implementations. maybe its something wrong with my OS?

                what i may try next is to follow the recommendation to get a stack dump of a crashed/paused vm. also, i may try to move the jboss server to a windows box to see if the problem still occurs.

                mike

                • 5. Re: weird hang-ups
                  strkrobertm

                  We had the same problem under RED Hat 9. It boils down to threading changes in the OS. The VM has not caught up with this yet.

                  Add this to the top of your run.sh

                  export LD_ASSUME_KERNEL=2.4.1

                  • 6. Re: weird hang-ups
                    mandrews-flarion

                    thanks so much for your suggestion ---

                    i've just started running my IBM VM with that export statement, and will report back with any other problems. or, hopefully, if in 1 week the VM is still running, i'll report back with success!

                    best regards,
                    mike

                    • 7. Re: weird hang-ups
                      mandrews-flarion

                      hi folks,

                      i've been running jboss 3.2.2 on a sun jre 1.4.2 (linux) with 'export LD_ASSUME_KERNEL=2.4.1' without a single hangup in over a week!

                      so i think the problem is solved.

                      does anybody have an idea as to when the underlying issue will be dealt with, namely the inability of various java vm's to interoperate with the new linux threading code?

                      also, for those other folks using jboss on linux, it may be useful for the jboss group to issue an advisory or incorporate the fix into the startup scripts.

                      thanks so much,
                      mike

                      • 8. Re: weird hang-ups

                        Please submit it as a patch.

                        • 9. Re: weird hang-ups
                          strkrobertm

                          My first explaination was not exactly correct.

                          You can see the bug explained in glory detail at:

                          http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=90301

                          • 10. Re: weird hang-ups
                            martin.stechert

                            Did anybody face that sort of problem on a different UNIX? I'm facing a similar problem on SunSolaris with JRE 1.3.1 and JBoss 3.2.1:
                            JBoss just does not respond to any HTTP requests any more.
                            Thanks in advance,
                            Martin.

                            • 11. Re: weird hang-ups
                              jmer

                              i had experience this too on linux box 9, so i make a watch dog script to restart jboss is things when down. but anyway thanks to the environment export its all hone now. On windows we have'nt experience such things.