6 Replies Latest reply on Nov 18, 2011 7:00 PM by calvin21

    JBoss spinning up multiple child processes on Solaris

    calvin21

      I have JBoss running on Solaris and there's a module that launces an external perl process via Runtime.exec().

       

      However, in addition to the perl process I also end up with child JBoss processes as shown below. Any idea why these are spun up?

       

      5266 is the parent process here..

       

      root 5266 5245 0 11:15:57 pts/2 17:06 /opt/JDK/bin/java -Dprogram.name=jboss_run.sh -s

      root 6668 5266 0 11:33:48 pts/2 0:00 /opt/JDK/bin/java -Dprogram.name=jboss_run.sh -s

      root 16061 5266 0 12:13:03 pts/2 0:00 /opt/JDK/bin/java -Dprogram.name=jboss_run.sh -s

      root 17973 5266 0 12:33:38 pts/2 0:00 /opt/JDK/bin/java -Dprogram.name=jboss_run.sh -s

      root 11358 5266 0 11:51:22 pts/2 0:00 /opt/JDK/bin/java -Dprogram.name=jboss_run.sh -s

      root 11733 5266 0 11:53:30 pts/2 0:00 /opt/JDK/bin/java -Dprogram.name=jboss_run.sh -s

      root 13117 5266 0 12:00:28 pts/2 0:00 /opt/JDK/bin/java -Dprogram.name=jboss_run.sh -s

        • 1. Re: JBoss spinning up multiple child processes on Solaris
          wdfink

          Do you mean that an EJB component starts an external process?

          After JBoss start there is only one process? For each exec a new will be created?

           

          You know that this will be against the JEE specification.

          • 2. Re: JBoss spinning up multiple child processes on Solaris
            calvin21

            There's no EJB, a POJO that spawns a new process to execute some non Java tasks.. why should that be against the J2EE spec?    

             

            After JBoss starts there's only one process, but after spawning the perl process, we end up with a child jboss process..         

            • 3. Re: JBoss spinning up multiple child processes on Solaris
              peterj

              On Linux systems (and thus I suspect on UNIX in general) you can configure them to show individual threads as processes. In Linux there is some option that governs this, if I took the time I could probably find posts from a while back that discussed that issue - people were seeing multiple java  "processes" for JBoss AS when they did a ps, when actually what they were seeing were the threads running in the one java process.

               

              I suspect that what you are seeing is a feature/quirk in Solaris where if a thread of a process spawns another process, then the orignal process' thread shows up in the process report, possibly so that the report can denote that the spawn process came from that specific thread.

              • 4. Re: JBoss spinning up multiple child processes on Solaris
                calvin21

                Thats a good point. But i got a thread dump on that process and also a core dump. It was about 750 MB with a lot of threads in blocked state.

                 

                That indicates that it is in fact a clone of the original jboss process.

                 

                To reiterate the issue.

                 

                On startup, i have a jboss proc, say jboss_proc1.

                 

                On spawning a perl process, a have a tiny perl process, say perl_proc.

                But at this point i also have a full clone of the original jboss proc, say jboss_proc2 which is a child of jboss_proc1.

                 

                I am clueless as to why jboss_proc2 was spawned..

                • 5. Re: JBoss spinning up multiple child processes on Solaris
                  peterj

                  You do NOT have a full new copy of the java process running - what you have is a "reference"(?) to the same java process. So yes you will get a full process dump, and a full thread dump. Like i said, its a quirk of the OS, but each supposed process entry is merely a reference of the original process, one per thread (or in your case, one per thread that spawned another process). If you kill one of these processes, all of them should terminate (unless Solaris somehow can kill individual threads, which is a possibility, but then you will see that hardly any memory at all gets freed - just the thread stack).

                  1 of 1 people found this helpful
                  • 6. Re: JBoss spinning up multiple child processes on Solaris
                    calvin21

                    Hey Peter,

                     

                    There's a similar thread that you were also helping out with.. http://community.jboss.org/thread/153697?start=15&tstart=0

                     

                    My issue is that sometimes the "sub process" hangs and other times there are enough sub process to chew up the entire ram, bringing JBoss down.

                     

                    The gist of the other thread seems to be, that not consuming the error and input stream of the child process properly can cause this to happen. This is also documented here - http://download.oracle.com/javase/1.4.2/docs/api/java/lang/Process.html#waitFor%28%29

                     

                    My app does consume both of those properly.

                     

                    The only thing I dont do is a process.destroy in the finally block, so I'm going to try and see if adding that makes a difference (although the waitFor call might never return if the sub process is blocked on something).

                     

                    try {    
                              process.waitFor();       
                    } catch (InterruptedException  e) {
                           log.error("shellTimer wait for command execution  exception:",e);
                    }finally{
                            process.destroy();
                    }

                     

                    Is there anything else I need to do to ensure those "sub processes" terminate cleanly?

                     

                    Thanks,

                    Kailash.