1 2 Previous Next 23 Replies Latest reply on Jul 18, 2010 10:49 PM by cluster.lover

    It's Strange: many jboss instances are started by run.sh

    cluster.lover

      Environments:

           OS: Solaris10(T2000)

           JBoss: 5.1.0.GA

       

      How to reproduce

      1.  start jboss 5.1.0 with bellow arguments:

       

      bash-3.00# ps -ef|grep run.sh
          root 17414 17347   0 19:31:07 pts/2     169:00 /opt/genband/gvu/active/jdk/bin/java -Dprogram.name=run.sh -server -Xms1024m -X
          root 17347     1   0 19:31:06 pts/2       0:00 sh /opt/genband/gvu/01.00.71.03/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
          root 10584 10553   0 10:09:37 pts/5       0:00 grep run.sh

       


      bash-3.00# pargs 17347
      17347:  sh /opt/genband/gvu/01.00.71.03/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
      argv[0]: sh
      argv[1]: /opt/genband/gvu/01.00.71.03/jboss/bin/run.sh
      argv[2]: -c
      argv[3]: gvu
      argv[4]: -g
      argv[5]: P228.80.4.221
      argv[6]: -u
      argv[7]: 228.80.4.221
      argv[8]: -b
      argv[9]: 10.80.4.231

       

       

      bash-3.00# pargs 17414
      17414:  /opt/genband/gvu/active/jdk/bin/java -Dprogram.name=run.sh -server -Xms1024m -X
      argv[0]: /opt/genband/gvu/active/jdk/bin/java
      argv[1]: -Dprogram.name=run.sh
      argv[2]: -server
      argv[3]: -Xms1024m
      argv[4]: -Xmx2048m
      argv[5]: -XX:MaxPermSize=512m
      argv[6]: -Dorg.jboss.resolver.warning=true
      argv[7]: -Dsun.rmi.dgc.client.gcInterval=3600000
      argv[8]: -Dsun.rmi.dgc.server.gcInterval=3600000
      argv[9]: -Djboss.messaging.ServerPeerID=136
      argv[10]: -DserverName=T2000G2
      argv[11]: -Djavax.net.ssl.keyStore=************
      argv[12]: -Djavax.net.ssl.trustStore=**********
      argv[13]: -Djavax.net.ssl.trustStorePassword=************
      argv[14]: -Djavax.net.ssl.keyStorePassword=******
      argv[15]: -Djboss.gvu.ejb.interface.protocol=sslsocket
      argv[16]: -Dsslport=3843
      argv[17]: -Dsslport=3843
      argv[18]: -Djava.endorsed.dirs=/opt/genband/gvu/01.00.71.03/jboss/lib/endorsed
      argv[19]: -classpath
      argv[20]: /opt/genband/gvu/01.00.71.03/jboss/bin/run.jar:/opt/genband/gvu/active/jdk/lib/tools.jar
      argv[21]: org.jboss.Main
      argv[22]: -c
      argv[23]: gvu
      argv[24]: -g
      argv[25]: P228.80.4.221
      argv[26]: -u
      argv[27]: 228.80.4.221
      argv[28]: -b
      argv[29]: 10.80.4.231
      bash-3.00#

       

       

       

       

      2. my xxx.ear has an EJB3 timer which execute below method every 5 minutes:

      public static String execute(String command, String[] envp, String workingDirectory)throws Exception{
              String returnResult = null;       
                 
              Process process = null;
              if(workingDirectory==null){
                  process = Runtime.getRuntime().exec(command, envp,null);
              }else{
                  process = Runtime.getRuntime().exec(command, envp,new File(workingDirectory));
              }
              StringBuffer sb = new StringBuffer();       
             
              //read from [Process child]: If xx.bat causes screen print something, then you must read it.
              //if NOT, this bat file will NOT be executed to last line.       
              BufferedReader in = new BufferedReader(new InputStreamReader(process.getInputStream())); 
              String line = null; 
              while ((line = in.readLine()) != null) { 
                  sb.append(line); 
                  if(log.isDebugEnabled() || EmsServer.isInDevelopingMode())
                      log.info("shell> "+line);
              }
              try {
                  in.close();//added         
              } catch (Exception e) {
                  log.error("shellTimer close BufferedReader Exception:",e);
              }
              //
              try {         
                   process.waitFor();            
              } catch (InterruptedException e) {
                  returnResult = "shellTimer wait for command execution exception:"+e;
                  log.error("shellTimer wait for command execution exception:",e);
              }finally{
                  process.destroy();
              }

       

              if(log.isDebugEnabled() || EmsServer.isInDevelopingMode())
                  log.info("shell> "+"Command exit value:"+ String.valueOf(process.exitValue()));       
              //
              returnResult = sb.toString();//returnResult MUST be the true output of the command. Do NOT append anything to it.
              return returnResult;   
          }

       

       

      3. After running about 1 days, we find server jboss instances are started and OS resources are exausted:

      bash-3.00# ps -ef|grep run.sh

         root yyyy xxxxxx   0 19:31:07 pts/2     169:00  /opt/genband/gvu/active/jdk/bin/java -Dprogram.name=run.sh -server  -Xms1024m -X
         root yyyy xxxxxx   0 19:31:07 pts/2     169:00  /opt/genband/gvu/active/jdk/bin/java -Dprogram.name=run.sh -server  -Xms1024m -X
         root yyyy xxxxxx   0 19:31:07 pts/2     169:00  /opt/genband/gvu/active/jdk/bin/java -Dprogram.name=run.sh -server  -Xms1024m -X
         root yyyy xxxxxx   0 19:31:07 pts/2     169:00  /opt/genband/gvu/active/jdk/bin/java -Dprogram.name=run.sh -server  -Xms1024m -X
         root yyyy 17347    0 19:31:07 pts/2     169:00  /opt/genband/gvu/active/jdk/bin/java -Dprogram.name=run.sh -server  -Xms1024m -X
         root 17347     1   0 19:31:06 pts/2       0:00 sh  /opt/genband/gvu/01.00.71.03/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u  228
         root 10584 10553   0 10:09:37 pts/5       0:00 grep run.sh

       

       

      4. I undeployed ems.ear and run the above method {public static String execute(String command, String[] envp, String  workingDirectory)} in a simple Java Main class every 500 millseconds for about 1 hour, everything is normal.

       

       

       

      Why??? Has somebody met such case?

      Any tip is appreciated.

      Thank you in advance.

        • 1. Re: It's Strange: many jboss instances are started by run.sh
          cluster.lover

          More info:

           

          public static String execute(String command, String[] envp, String  workingDirectory)throws Exception

          ~~~~~~~~~This method restart another weight JAVA backend application if it's NOT started successfully. Because this Java APP is NOT installed, related starting shell is executed many times.  After about one day, the OS become VERY VERY slow.

          • 2. Re: It's Strange: many jboss instances are started by run.sh
            peterj

            In Linux, there is a kernel option whereby each htread shows up as a separate process when doing a ps. I don't know if Solaris has a similar option, but it appears like that is what is happening.

            • 3. Re: It's Strange: many jboss instances are started by run.sh
              cluster.lover

              Peter Johnson

               


                      You are famous man, I am VERY VERY glad to see your reply.

               

               

               

               

               


              This issue does NOT occur easily. Now, i describe it in detail.

               

               

               

              OS: Solaris 10(T2000)

               


              JBoss: 5.1.0.GA.

               


              Our xxx.ear is about 100M. This xxx.ear is an NMS which manages EMS1,EMS2,EMS3.

               

               

               

              (1)

               


              When xxx.ear is deployed successfully, xxx.ear will Runtime.getRuntime().exec(command, envp,null) to start EMS1, the command is:

               


                     su - gvadmin -c ". /export/home/gvadmin/.bashrc; /opt/genband/genview/bin/genview.run start_noninteractive_exit"

               

               

               

                     [ some guys from sun say that Runtime.getRuntime().exec(command, envp,null) maybe exhaust resource and cause strange exceptions.

               


                        but, from server.log, this command is executed ONLY twice!!!

               


                     ]

               

               

               

              (2) After xxx.ear is deployed, JBoss keep running about  3 days. During those days, we execute bellow ping every 15 seconds:

               


                  public static boolean ping(String ip,int port){
                      Socket socket = null;
                      try{
                          socket = new Socket();
                          socket.connect(new InetSocketAddress(ip,port), 5000);
                          return true;
                      }catch (Exception e){
                          return false;
                      }finally{
                          try{
                              if(socket!=null)
                                  socket.close();
                          }catch(Exception e){             
                          }
                      }     
                  }

               

               

               

              (3) We have many Hosts,

               


              Host1, the load is weight: Everything is normal even jboss is run 15 days.

               


              Host2, very very heavy load: about 3 days later, Many jboss instances are started,

               

               

               

              NOTE:

               


                       Memeory are exausted!!!

               

               

               

               

               


              (4) This is the process list:

               

               

               

              Grand Father process           Father process         Children process            grand children process

               


                       1                         ->           1938                  -->           2005               --->       995 ,3579 , 26427, 1518028293

               

               

               


              bash-3.00# ps -ef|grep run.sh
                  root   995  2005   0   Jun 30 ?           0:00 /opt/xxx/active/jdk/bin/java -Dprogram.name=run.sh -server -Xms1024m -X
                  root  3523 11215   0 13:43:27 pts/5       0:00 grep run.sh
                  root  3579  2005   0   Jun 30 ?           0:00 /opt/xxx/active/jdk/bin/java -Dprogram.name=run.sh -server -Xms1024m -X
                  root 26427  2005   0   Jun 30 ?           0:00 /opt/xxx/active/jdk/bin/java -Dprogram.name=run.sh -server -Xms1024m -X
                  root  1938     1   0   Jun 30 ?           0:00 sh /opt/xxx/01.00.71.03/jboss/bin/run.sh -c nms -g P228.80.5.129 -u 228
                  root  2005  1938   0   Jun 30 ?         750:18 /opt/xxx/active/jdk/bin/java -Dprogram.name=run.sh -server -Xms1024m -X
                  root 15810  2005   0   Jun 30 ?           0:00 /opt/xxx/active/jdk/bin/java -Dprogram.name=run.sh -server -Xms1024m -X
                  root 28293  2005   0   Jun 30 ?           0:00 /opt/xxx/active/jdk/bin/java -Dprogram.name=run.sh -server -Xms1024m -X
              bash-3.00#

               

               

               


              (4.1) Father process info

               


              bash-3.00# pargs 1
              1:      /sbin/init
              argv[0]: /sbin/init
              bash-3.00#

               

               

               


              (4.2)

               

               

               

              bash-3.00# pargs 1938
              1938:   sh /opt/xxx/01.00.71.03/jboss/bin/run.sh -c nms -g P228.80.5.129 -u 228
              argv[0]: sh
              argv[1]: /opt/xxx/01.00.71.03/jboss/bin/run.sh
              argv[2]: -c
              argv[3]: nms
              argv[4]: -g
              argv[5]: P228.80.5.129
              argv[6]: -u
              argv[7]: 228.80.5.129
              argv[8]: -b
              argv[9]: 10.80.5.129
              bash-3.00#

               

               

               


              (4.3)

               


              bash-3.00# pargs 2005
              2005:   /opt/xxx/active/jdk/bin/java -Dprogram.name=run.sh -server -Xms1024m -X
              argv[0]: /opt/xxx/active/jdk/bin/java
              argv[1]: -Dprogram.name=run.sh
              argv[2]: -server
              argv[3]: -Xms1024m
              argv[4]: -Xmx2048m
              argv[5]: -XX:MaxPermSize=512m
              argv[6]: -Dorg.jboss.resolver.warning=true
              argv[7]: -Dsun.rmi.dgc.client.gcInterval=3600000
              argv[8]: -Dsun.rmi.dgc.server.gcInterval=3600000
              argv[9]: -Djboss.messaging.ServerPeerID=35
              argv[10]: -DserverName=T2000nms3
              argv[11]: -Djavax.net.ssl.keyStore=/opt/xxx/active/jboss/server/nms/conf/nmsserverkeystore
              argv[12]: -Djavax.net.ssl.trustStore=/opt/xxx/active/jboss/server/nms/conf/nmsservertruststore
              argv[13]: -Djavax.net.ssl.trustStorePassword=*******!
              argv[14]: -Djavax.net.ssl.keyStorePassword=*******!
              argv[15]: -Djboss.nms.ejb.interface.protocol=sslsocket
              argv[16]: -Dsslport=3843
              argv[17]: -Dsslport=3843
              argv[18]: -Djava.endorsed.dirs=/opt/xxx/01.00.71.03/jboss/lib/endorsed
              argv[19]: -classpath
              argv[20]: /opt/xxx/01.00.71.03/jboss/bin/run.jar:/opt/xxx/active/jdk/lib/tools.jar
              argv[21]: org.jboss.Main
              argv[22]: -c
              argv[23]: nms
              argv[24]: -g
              argv[25]: P228.80.5.129
              argv[26]: -u
              argv[27]: 228.80.5.129
              argv[28]: -b
              argv[29]: 10.80.5.129
              bash-3.00#

              • 4. Re: It's Strange: many jboss instances are started by run.sh
                cluster.lover

                Our host is T2000, 8/16 CPUS,  16G memory.

                 

                After long running, JBoss become abnormal.

                This issue appeared 3 times with heavy load.

                 

                Is this a issue of Solaris10???

                • 5. Re: It's Strange: many jboss instances are started by run.sh
                  peterj

                  Unfortunately I don't know enough about to offer an opinion as to whether this is a Solaris issue or not. I am also not sure what I am supposed to do with all of the information you posted. If I understand correctly from your original post, you are wondering why multiple java processes appear to be running. I know of only two ways that could happen:

                   

                  1) The kernel option I mentioned earlier where each thread is a process

                  2) You ran the app server multiple times

                   

                  If #2 were the case, then you would see various error regarding ports being in use already.

                   

                  I suppose that there could be some glitch in Solaris that causes thep rocess to show up multiple times, but you would have to ask Oracle about that.

                   

                  If you kill one of the processes, do they all go away?

                  • 6. Re: It's Strange: many jboss instances are started by run.sh
                    dmlloyd

                    I'll hazard a guess.  If you're doing a Runtime.exec() this translates to a fork/exec on the "insides".  On a very busy/large server this could have some adverse effects:

                    • Since there's no way (that I'm aware of) to atomically open a file descriptor and mark it as close-on-exec, you might be getting bogus file descriptors in the child processes that hang around.
                    • Runtime.exec() may cause a full copy of the process core space to be made before the exec can occur (which would subsequently delete this space).  If the exec then fails (due to exhausted resources or whatever), after the fork, perhaps the presence of those file descriptors and other resources is somehow preventing the process from terminating.

                     

                    Basically my recommendation is - don't use Runtime.exec() from big processes, especially containers.  It's hard to say what would happen with any certainty.

                    • 7. Re: It's Strange: many jboss instances are started by run.sh
                      cluster.lover

                      Unfortunately I don't know enough  about to offer an opinion as to whether this is a Solaris issue or not. I  am also not sure what I am supposed to do with all of the information  you posted. If I understand correctly from your original post, you are  wondering why multiple java processes appear to be running. I know of  only two ways that could happen:

                       

                      1) The kernel option I mentioned earlier  where each thread is a process

                       

                      2) You ran the app server multiple  times

                       

                      If  #2 were the case, then you would see various error regarding ports  being in use already.

                      [ybxiang]: #2 is NOT the case, we start jboss app server ONLY once, namely execute run.sh ONLY once. If we run app server multiple times with same port, later started app server will print exception.

                       

                       

                      I suppose that there could be some glitch in  Solaris that causes thep rocess to show up multiple times, but you would  have to ask Oracle about that.

                      [ybxiang]: I will.   Guys from Sun / Oracle are NOT as kind as guys from JBoss. They often tell me: NOT solaris' fault. That's why i come here. I like guys from JBoss, and I want to exclude JBoss's fault at first then ask help from sun.

                       

                      If you kill one of the processes, do they  all go away?

                      [ybxiang]: NO. I killed one of them, everything works as normal and the rest [about 5] instances are still there.

                                      Then, i killed most of them, only [#1938 and #2005] are left, the result is the same: no exception occurs, we can access jboss as normal.

                       

                       

                      If #1 were the case, then Solaris with  default setting is horrible:

                      a) if every thread occupied 2G memeroy which is the memory value set for run.sh and if there are 100 users, the machine will NOT work any longer.

                      b) ACTUALLY, every jboss instance on solaris DO occupied 2G memory, and the host get sticky!!! We almost can do nothing on it untill we kill some instances of jboss.

                       

                      • 8. Re: It's Strange: many jboss instances are started by run.sh
                        peterj

                        Your assumption regarding option #1 is incorrect - there is only one process running, but each thread shows up when you do a ps. All of the threads are sharing the same 2GB or memory that you sited in your example. Check the available memory before and after killing one of the extra processes, does it change? My guess is that it does not. But if it doesn't, then Solaris is really misbehaving.

                         

                        By the way, I did not say you started JBoss AS multiple times; rather I was outlining the only instances under which I have noticed multiple ps entries for JBoss AS.

                         

                        Since killing the extra entries had no bearing on JBoss AS, my guess is that Solaris got confused and made extra process entries for some reason. Perhaps someone who uses SAolaris and is reading this might chime in and offer an opinion.

                         

                        There is no situation under which I know that JBoss AS launches multiple instances. But if you look at the end of run.sh you will notice a "while true; do" loop. If the app server is terminated in a certain fashion, then it will restart itself. You might try playing around with that, either adding some debugging code or removing the "while true; do" statement entirely and then see if the problem goes away. Of course, if the "while true; do" statement is the issue, then it would matter which processes you killed because only one of them would really be running - the others would be in some sort of zombie state: exited but not quite done exiting.

                         

                        By the way, it would best if you did not use a lot of colored text, it is hard to read.

                        • 9. Re: It's Strange: many jboss instances are started by run.sh
                          cluster.lover

                          I'll hazard a guess.  If you're  doing a Runtime.exec() this translates to a fork/exec on the "insides".   On a very busy/large server this could have some  adverse effects:

                          • Since there's no way (that I'm aware of) to  atomically open a file descriptor and mark it as close-on-exec, you  might be getting bogus file descriptors in the child processes that hang  around.
                          • Runtime.exec() may cause a full copy  of the process core space to be made before the exec can occur (which  would subsequently delete this space).  If the exec then fails (due to  exhausted resources or whatever), after the fork, perhaps the presence  of those file descriptors and other resources is somehow preventing the  process from terminating.


                          Basically my recommendation is - don't use Runtime.exec() from big processes, especially containers.  It's  hard to say what would happen with any certainty.

                           

                           

                          [ybxiang]: When see word "fork" in your reply, i remember that after multiple jboss instances are started, the host is sticky and sometimes we get such exception when excecute "ls" command on solaris:

                              Could not fork: Not enough space

                          Maybe this issues is caused by Runtime.exec(). But from the log, i can see that Runtime.exec() is executed successfully.


                          I MUST use Runtime.exec() because in my case, JBoss must start Legacy EMS(c++). Maybe i should run the starting command in a new Thread???

                          • 10. Re: It's Strange: many jboss instances are started by run.sh
                            cluster.lover

                            Peter Johnson and David Lloyd,

                             

                                  Thank you for your valuable replies.

                                  I'll ask guys from Sun/Oracle and post the  result here if the result is valuable.

                            • 11. Re: It's Strange: many jboss instances are started by run.sh
                              cluster.lover

                              In Linux, there is a kernel option whereby each htread shows up as a  separate process when doing a ps. I don't know if Solaris has a similar  option, but it appears like that is what is happening.

                              ~~~~~~~~~~~This info is important too.  I'll study it.

                               

                               

                              All research about this issue will be recorded here.

                              Now, two things will be studied deeply:

                              a) kernel option of new thread.

                              b) Runtime.exec();      [ACUTALLY, this topic has been studied many many times.]

                              • 12. Re: It's Strange: many jboss instances are started by run.sh
                                cluster.lover

                                Your assumption regarding option  #1 is incorrect - there is only one process running, but each thread  shows up when you do a ps. All of the threads are sharing the same 2GB  or memory that you sited in your example. Check the available memory  before and after killing one of the extra processes, does it change? My  guess is that it does not. But if it doesn't, then Solaris is really  misbehaving.

                                 

                                By  the way, I did not say you started JBoss AS multiple times; rather I  was outlining the only instances under which I have noticed multiple ps  entries for JBoss AS.

                                [ybxiang]: sorry, i should had read your replies slowly and carefully.

                                 

                                 

                                Since killing the extra entries had no bearing on  JBoss AS, my guess is that Solaris got confused and made extra process  entries for some reason. Perhaps someone who uses SAolaris and is  reading this might chime in and offer an opinion.

                                 

                                There is no  situation under which I know that JBoss AS launches multiple instances.  But if you look at the end of run.sh you will notice a "while true; do"  loop. If the app server is terminated in a certain fashion, then it will  restart itself. You might try playing around with that, either adding  some debugging code or removing the "while true; do" statement entirely  and then see if the problem goes away. Of course, if the "while true;  do" statement is the issue, then it would matter which processes you  killed because only one of them would really be running - the others  would be in some sort of zombie state: exited but not quite done  exiting.

                                [ybxiang]: From core dump, we DO find that jboss crashed certain times.

                                 

                                 

                                By  the way, it would best if you did not use a lot of colored text, it is  hard to read.

                                [ybxiang]: sorry. I changed the color to gray.

                                • 13. Re: It's Strange: many jboss instances are started by run.sh
                                  cluster.lover

                                  progress of today:

                                   

                                   

                                  We found this case occurs again.

                                   

                                  1. Stable process info:

                                  bash-3.00# ps -ef|grep run.sh |grep genband
                                      root 22013 21992   0 08:33:02 pts/7     266:53 /opt/genband/jdk/bin/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 16238 22013   0 13:35:47 pts/7       0:00 /opt/genband/jdk/bin/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 21992     1   0 08:33:01 pts/7       0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
                                      root 25114 22013   0 11:40:45 pts/7       0:00 /opt/genband/jdk/bin/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX

                                   

                                  2. I execute "ps -ef|grep run.sh |grep genband" every 1 seconds, and find that new instance is started and shutdown again before long:

                                  bash-3.00# ps -ef|grep run.sh |grep genband
                                      root 22013 21992   0 08:33:02 pts/7     266:23 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 16238 22013   0 13:35:47 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 21520 22013   0 13:56:40 pts/7       0:01 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX  !!!!!
                                      root 21992     1   0 08:33:01 pts/7       0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
                                      root 25114 22013   0 11:40:45 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                  bash-3.00#
                                  bash-3.00#
                                  bash-3.00# ps -ef|grep run.sh |grep genband
                                      root 22013 21992   0 08:33:02 pts/7     266:53 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 16238 22013   0 13:35:47 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 21992     1   0 08:33:01 pts/7       0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
                                      root 25114 22013   0 11:40:45 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                  bash-3.00#
                                  bash-3.00# ps -ef|grep run.sh |grep genband
                                      root 22013 21992   0 08:33:02 pts/7     273:07 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 16238 22013   0 13:35:47 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 21992     1   0 08:33:01 pts/7       0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
                                      root 24530 22013   0 14:13:49 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX   !!!!!!
                                      root 25114 22013   0 11:40:45 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                  bash-3.00#
                                  bash-3.00# ps -ef|grep run.sh |grep genband
                                      root 22013 21992   0 08:33:02 pts/7     273:07 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 16238 22013   0 13:35:47 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 21992     1   0 08:33:01 pts/7       0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
                                      root 25114 22013   0 11:40:45 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                  bash-3.00#
                                  bash-3.00# ps -ef|grep run.sh |grep genband
                                      root 22013 21992   0 08:33:02 pts/7     276:46 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 16238 22013   0 13:35:47 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 21992     1   0 08:33:01 pts/7       0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
                                      root 25114 22013   0 11:40:45 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                  bash-3.00# ps -ef|grep run.sh |grep genband
                                      root 22013 21992   1 08:33:02 pts/7     276:47 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 16238 22013   0 13:35:47 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 21992     1   0 08:33:01 pts/7       0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
                                      root 25114 22013   0 11:40:45 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                  bash-3.00# ps -ef|grep run.sh |grep genband
                                      root 22013 21992   1 08:33:02 pts/7     276:48 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 16238 22013   0 13:35:47 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 21992     1   0 08:33:01 pts/7       0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
                                      root 25114 22013   0 11:40:45 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 26352 22013   0 14:23:35 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX !!!!!!!!!!!!!!!
                                  bash-3.00# ps -ef|grep run.sh |grep genband
                                      root 22013 21992   1 08:33:02 pts/7     276:50 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 16238 22013   0 13:35:47 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 21992     1   0 08:33:01 pts/7       0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
                                      root 25114 22013   0 11:40:45 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 26352 22013   0 14:23:35 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX !!!!!!!!!!!!!!!
                                  bash-3.00# ps -ef|grep run.sh |grep genband
                                      root 22013 21992   0 08:33:02 pts/7     276:50 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 16238 22013   0 13:35:47 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 21992     1   0 08:33:01 pts/7       0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
                                      root 25114 22013   0 11:40:45 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                  bash-3.00#
                                  bash-3.00# ps -ef|grep run.sh |grep genband
                                      root 22013 21992   0 08:33:02 pts/7     276:55 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 16238 22013   0 13:35:47 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                      root 21992     1   0 08:33:01 pts/7       0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
                                      root 25114 22013   0 11:40:45 pts/7       0:00 /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
                                  bash-3.00#

                                  • 14. Re: It's Strange: many jboss instances are started by run.sh
                                    cluster.lover

                                    1.

                                    Processes hierachy:

                                     

                                    1     -->   21992   -->   22013    -->      ( 25114 / 16238 / 21520 / 24530 /27504 )

                                     

                                     

                                     

                                    2. related process stack info:

                                     

                                    bash-3.00# pstack 21992
                                    21992:  sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.2
                                    ff2cc91c waitid   (7, 0, ffbfecc0, f)
                                    ff2bc2a4 waitpid  (ffffffff, ffbfee24, c, 0, 0, 53000) + 60
                                    000265c8 job_wait (55fd, 0, 55fd, 0, 53400, 55fd) + 1bc
                                    000334f8 sh_exec  (204, 0, 550c8, 0, 1000, 53400) + d44
                                    00033b18 sh_exec  (3, 32400, 53000, 53000, 0, 53400) + 1364
                                    00034154 sh_exec  (4, 32400, 53000, 53000, 0, 53400) + 19a0
                                    00033b18 sh_exec  (3, 32400, 53000, 53000, 0, 53400) + 1364
                                    00033fe4 sh_exec  (57328, 57328, 53000, 53000, 0, 53400) + 1830
                                    0002a474 ???????? (53000, c, 527e0, 53800, 53000, 204)
                                    00029ce0 main     (20000000, 2c000, ffbffcbb, 53000, 53000, 53000) + a30
                                    00016b20 _start   (0, 0, 0, 0, 0, 0) + 108
                                    bash-3.00#

                                     

                                    -------------------------

                                    bash-3.00# pstack 22013

                                    22013:    /opt/genband/jdk/bin/java -Dprogram.name=run.sh -server -Xms1024m -Xmx
                                    -----------------  lwp# 1 / thread# 1  --------------------
                                    ff2cc624 lwp_wait (2, ffbfee14)
                                    ff2c54bc _thrp_join (2, 0, ffbfeed8, 1, ffbfee14, ff335800) + 34
                                    ff2c5628 thr_join (2, 0, ffbfeed8, ffbfef68, 0, 0) + 10
                                    00018c8c ContinueInNewThread (124c0, 0, 80000, ffbfef68, fffe79d4, 0) + 30
                                    00012480 main     (18400, 2af38, 10000, 2b98c, 44c, 10001) + eb0
                                    000111a0 _start   (0, 0, 0, 0, 0, 0) + 108
                                    -----------------  lwp# 2 / thread# 2  --------------------
                                    ff2cc6d8 lwp_cond_wait (32448, 32430, 0, 0)
                                    feb321b0 __1cCosNPlatformEventEpark6M_v_ (32430, 1, fed4a47b, b1800, 32400, fee4c748) + 100
                                    feb10afc __1cHMonitorFIWait6MpnGThread_x_i_ (301b8, 31400, 0, 301c8, fee2d820, b8000) + dc
                                    feb116c4 __1cHMonitorEwait6Mblb_b_ (301b8, 31400, 0, 1, fee2d771, 2) + 1c8
                                    febfe9d4 __1cHThreadsKdestroy_vm6F_b_ (fee389d0, 31400, 301b8, 1, fee3e630, 3d400) + ac
                                    fe93cefc jni_DestroyJavaVM (0, 31400, 0, fee2d771, fedfc000, 31771) + 198
                                    00013b24 JavaMain (0, 2b9ac, 0, fe93cd64, 8, fee32d24) + 1664
                                    ff2c88ec _lwp_start (0, 0, 0, 0, 0, 0)

                                    ............

                                    -----------------  lwp# 90 / thread# 90  --------------------
                                    ff2cc6d8 lwp_cond_wait (239e1e8, 239e1d0, 0, 0)
                                    feb32a28 __1cGParkerEpark6Mbx_v_ (239e1d0, a97000, 0, a97000, 2, fedfc000) + 37c
                                    fec1cfd0 Unsafe_Park (fee3e84c, 42800, 0, a97000, 4284c, fedfc000) + 274
                                    fce4d638 * java/util/concurrent/locks/LockSupport.park(Ljava/lang/Object;)V+14 (line 314)
                                    fc005ab0 * java/util/concurrent/locks/LockSupport.park(Ljava/lang/Object;)V+14 (line 314)
                                    fc005ab0 * java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await()V+42 (line 3844)
                                    fc005fd0 * java/util/concurrent/LinkedBlockingQueue.take()Ljava/lang/Object;+29 (line 792)
                                    fc005e60 * org/jboss/ha/framework/server/AsynchEventHandler.run()V+55 (line 192)
                                    fc005fd0 * java/lang/Thread.run()V+11 (line 1237)
                                    fc00021c * StubRoutines (1)
                                    fe5594fc __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (fc0001c0, a97000, 1, 11454a0, 5b830e80, 55fffd78) + 208
                                    fe90af98 __1cJJavaCallsMcall_virtual6FpnJJavaValue_nLKlassHandle_nMsymbolHandle_4pnRJavaCallArguments_pnGThread__v_ (55fffd70, 11454a4, 11454b0, 8a2c00, 55fffc58, ff75d2f4) + 10c
                                    fe5ecd2c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (55fffd70, 55fffd6c, 55fffd68, 55fffd64, 55fffd60, 11454a4) + 74
                                    fe5ffe24 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (5b8330b0, a97000, 50800, fee4d104, fee4ce38, fee4cb28) + 110
                                    febf99f0 __1cKJavaThreadRthread_main_inner6M_v_ (a97000, 26dce18, 5a, 1d, fedfc000, 0) + 48
                                    feb2a198 java_start (a97000, 55fd, fedfc000, fed4888d, 25d1800, fee488b4) + 22c
                                    ff2c88ec _lwp_start (0, 0, 0, 0, 0, 0)
                                    -----------------  lwp# 91 / thread# 91  --------------------

                                    .................


                                    -----------------  lwp# 272 / thread# 272  --------------------
                                    ff2cc6d8 lwp_cond_wait (4091e48, 4091e30, 0, 0)
                                    feb321b0 __1cCosNPlatformEventEpark6M_v_ (4091e30, 0, fed4a47b, b1800, 4091e00, fee4c748) + 100
                                    febda5ac __1cNObjectMonitorEwait6MxbpnGThread__v_ (31e6e50, 3d400, 1, 0, ae29400, ae29400) + 3d4
                                    fe5e4f48 __1cSObjectSynchronizerEwait6FnGHandle_xpnGThread__v_ (4d47f8c4, ae29400, 3184d, 3216b94, fedfc000, 3216b94) + 180
                                    fe5e4b58 JVM_MonitorWait (0, 3216b94, ae29400, 4284c, 1, fedfc000) + 2bc
                                    fc00d480 * java/lang/Object.wait(J)V+0
                                    fc00d42c * java/lang/Object.wait(J)V+0
                                    fc005ab0 * java/lang/Object.wait()V+2 (line 970)
                                    fc005ab0 * org/jboss/bootstrap/AbstractServerImpl$LifeThread.run()V+11 (line 1728)
                                    fc00021c * StubRoutines (1)
                                    fe5594fc __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (fc0001c0, ae29400, 1, 3216b80, 5bf08770, 4d47fe78) + 208
                                    fe90af98 __1cJJavaCallsMcall_virtual6FpnJJavaValue_nLKlassHandle_nMsymbolHandle_4pnRJavaCallArguments_pnGThread__v_ (4d47fe70, 3216b84, 3216b90, 8a2c00, 4d47fd58, ff75d2f4) + 10c
                                    fe5ecd2c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (4d47fe70, 4d47fe6c, 4d47fe68, 4d47fe64, 4d47fe60, 3216b84) + 74
                                    fe5ffe24 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (5b8330b0, ae29400, 50800, fee4d104, fee4ce38, fee4cb28) + 110
                                    febf99f0 __1cKJavaThreadRthread_main_inner6M_v_ (ae29400, 1416ec0, 110, f, fedfc000, 0) + 48
                                    feb2a198 java_start (ae29400, 55fd, fedfc000, fed4888d, 2a2b420, fee488b4) + 22c
                                    ff2c88ec _lwp_start (0, 0, 0, 0, 0, 0)
                                    -----------------  lwp# 273 / thread# 273  --------------------

                                    ...........

                                    -----------------  lwp# 13138 / thread# 13138  --------------------
                                    ff2cc6d8 lwp_cond_wait (12a1298, 12a1280, 4317f6a8, 0)
                                    ff2b3938 _lwp_cond_timedwait (12a1298, 12a1280, 4317f748, 50000, fee2d810, 31810) + 1c
                                    feb32a5c __1cGParkerEpark6Mbx_v_ (12a1280, 120e000, 0, 120e000, 2, fedfc000) + 3b0
                                    fec1cfd0 Unsafe_Park (fee3e84c, 42800, 0, 120e000, 4284c, fedfc000) + 274
                                    fce4d638 * *sun/misc/Unsafe.park(ZJ)V [compiled]
                                    fc2b32d8 * *java/util/concurrent/locks/LockSupport.parkNanos(Ljava/lang/Object;J)V [compiled] +42 (line 396)
                                    fc2b32d8 * *java/util/concurrent/SynchronousQueue$TransferStack.awaitFulfill(Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;ZJ)Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;+350 (line 822)
                                    fce45f6c * *java/util/concurrent/SynchronousQueue$TransferStack.transfer(Ljava/lang/Object;ZJ)Ljava/lang/Object; [compiled] +206 (line 645)
                                    fcc73e20 * *java/util/concurrent/SynchronousQueue.poll(JLjava/util/concurrent/TimeUnit;)Ljava/lang/Object; [compiled] +24 (line 1749)
                                    fcc73e20 * *java/util/concurrent/ThreadPoolExecutor.getTask()Ljava/lang/Runnable;+126 (line 1901)
                                    fcc73e20 * *java/util/concurrent/ThreadPoolExecutor$Worker.run()V+38 (line 1816)
                                    fc005fd0 * java/lang/Thread.run()V+11 (line 1237)
                                    fc00021c * StubRoutines (1)
                                    fe5594fc __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (fc0001c0, 120e000, 1, b7e1360, 5b830e80, 4317fd78) + 208
                                    fe90af98 __1cJJavaCallsMcall_virtual6FpnJJavaValue_nLKlassHandle_nMsymbolHandle_4pnRJavaCallArguments_pnGThread__v_ (4317fd70, b7e1364, b7e1370, 8a2c00, 4317fc58, ff75d2f4) + 10c
                                    fe5ecd2c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (4317fd70, 4317fd6c, 4317fd68, 4317fd64, 4317fd60, b7e1364) + 74
                                    fe5ffe24 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (5b8330b0, 120e000, 50800, fee4d104, fee4ce38, fee4cb28) + 110
                                    febf99f0 __1cKJavaThreadRthread_main_inner6M_v_ (120e000, 1be5770, 3352, b, fedfc000, 0) + 48
                                    feb2a198 java_start (120e000, 55fd, fedfc000, fed4888d, af88de8, fee488b4) + 22c
                                    ff2c88ec _lwp_start (0, 0, 0, 0, 0, 0)
                                    -----------------  lwp# 13137 / thread# 13137  --------------------
                                    ff2cc6d8 lwp_cond_wait (399a858, 399a840, 438ff628, 0)
                                    ff2b3938 _lwp_cond_timedwait (399a858, 399a840, 438ff6c8, 50000, fee2d810, 31810) + 1c
                                    feb32a5c __1cGParkerEpark6Mbx_v_ (399a840, 21f8d000, 0, 21f8d000, 2, fedfc000) + 3b0
                                    fec1cfd0 Unsafe_Park (fee3e84c, 42800, 0, 21f8d000, 4284c, fedfc000) + 274
                                    fce4d638 * *sun/misc/Unsafe.park(ZJ)V [compiled]
                                    fc2b32d8 * *java/util/concurrent/locks/LockSupport.parkNanos(Ljava/lang/Object;J)V [compiled] +42 (line 396)
                                    fc2b32d8 * *java/util/concurrent/SynchronousQueue$TransferStack.awaitFulfill(Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;ZJ)Ljava/util/concurrent/SynchronousQueue$TransferStack$SNode;+350 (line 822)
                                    fce45f6c * *java/util/concurrent/SynchronousQueue$TransferStack.transfer(Ljava/lang/Object;ZJ)Ljava/lang/Object; [compiled] +206 (line 645)
                                    fcc73e20 * *java/util/concurrent/SynchronousQueue.poll(JLjava/util/concurrent/TimeUnit;)Ljava/lang/Object; [compiled] +24 (line 1749)
                                    fcc73e20 * *java/util/concurrent/ThreadPoolExecutor.getTask()Ljava/lang/Runnable;+126 (line 1901)
                                    fcc73e20 * *java/util/concurrent/ThreadPoolExecutor$Worker.run()V+38 (line 1816)
                                    fc005fd0 * java/lang/Thread.run()V+11 (line 1237)
                                    fc00021c * StubRoutines (1)
                                    fe5594fc __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (fc0001c0, 21f8d000, 1, eecd2a0, 5b830e80, 438ffcf8) + 208
                                    fe90af98 __1cJJavaCallsMcall_virtual6FpnJJavaValue_nLKlassHandle_nMsymbolHandle_4pnRJavaCallArguments_pnGThread__v_ (438ffcf0, eecd2a4, eecd2b0, 8a2c00, 438ffbd8, ff75d2f4) + 10c
                                    fe5ecd2c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (438ffcf0, 438ffcec, 438ffce8, 438ffce4, 438ffce0, eecd2a4) + 74
                                    fe5ffe24 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (5b8330b0, 21f8d000, 50800, fee4d104, fee4ce38, fee4cb28) + 110
                                    febf99f0 __1cKJavaThreadRthread_main_inner6M_v_ (21f8d000, d6f0730, 3351, b, fedfc000, 0) + 48
                                    feb2a198 java_start (21f8d000, 55fd, fedfc000, fed4888d, 1064250, fee488b4) + 22c
                                    ff2c88ec _lwp_start (0, 0, 0, 0, 0, 0)
                                    -----------------  lwp# 8711 / thread# 8711  --------------------
                                    ff2cc2e4 read     (fb, 436ff55c, 4)
                                    fe3da49c Java_java_lang_UNIXProcess_forkAndExec (d6a0d10, 0, 223f938, 4, 436ff55c, 0) + 404
                                    fc00d480 * java/lang/UNIXProcess.forkAndExec([B[BI[BI[BZLjava/io/FileDescriptor;Ljava/io/FileDescriptor;Ljava/io/FileDescriptor;)I+0
                                    fc00d42c * java/lang/UNIXProcess.forkAndExec([B[BI[BI[BZLjava/io/FileDescriptor;Ljava/io/FileDescriptor;Ljava/io/FileDescriptor;)I+0
                                    fc005988 * java/lang/UNIXProcess.<init>([B[BI[BI[BZ)V+62 (line 101)
                                    fc005ab0 * java/lang/ProcessImpl.start([Ljava/lang/String;Ljava/util/Map;Ljava/lang/String;Z)Ljava/lang/Process;+182 (line 119)
                                    fc005940 * java/lang/ProcessBuilder.start()Ljava/lang/Process;+120 (line 896)
                                    fc005940 * java/lang/Runtime.exec([Ljava/lang/String;[Ljava/lang/String;Ljava/io/File;)Ljava/lang/Process;+16 (line 1186)
                                    fc005940 * java/lang/Runtime.exec([Ljava/lang/String;)Ljava/lang/Process;+4 (line 932)
                                    fc005940 * com/nextone/bn/devicecomm/devicemanager/CheckMachineStatusThread.run()V+183 (line 895)
                                    fc00021c * StubRoutines (1)
                                    fe5594fc __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (fc0001c0, d6a0c00, 1, 21fbda08, 6197fd60, 436ffdf8) + 208
                                    fe90af98 __1cJJavaCallsMcall_virtual6FpnJJavaValue_nLKlassHandle_nMsymbolHandle_4pnRJavaCallArguments_pnGThread__v_ (436ffdf0, 21fbda0c, 21fbda18, 8a2c00, 436ffcd8, ff75d2f4) + 10c
                                    fe5ecd2c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (436ffdf0, 436ffdec, 436ffde8, 436ffde4, 436ffde0, 21fbda0c) + 74
                                    fe5ffe24 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (5b8330b0, d6a0c00, 50800, fee4d104, fee4ce38, fee4cb28) + 110
                                    febf99f0 __1cKJavaThreadRthread_main_inner6M_v_ (d6a0c00, 1bcdb30, 2207, b, fedfc000, 0) + 48
                                    feb2a198 java_start (d6a0c00, 55fd, fedfc000, fed4888d, 24981d0, fee488b4) + 22c
                                    ff2c88ec _lwp_start (0, 0, 0, 0, 0, 0)



                                    Please refer to the attachment for details.


                                    -------------------------


                                    bash-3.00# pstack 25114
                                    25114:  /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx
                                    ff2c898c lwp_park (0, 0, 0)
                                    ff2c29a0 cond_wait_queue (18d3520, 18d34f0, 0, 0, 18d3520, 0) + 4c
                                    ff2c2de4 cond_wait_common (18d3520, 18d34f0, 0, 0, 0, 0) + 294
                                    ff2c2ee8 cond_wait (18d3520, 18d34f0, ecf0b002, 18d34f0, 18d3520, ecf0b000) + 10
                                    ff2c2f24 pthread_cond_wait (18d3520, 18d34f0, 0, 0, 18d34f0, 5b7b4058) + 8
                                    5b7b3f50 soft_delete_object_cleanup (18d33c0, ecf0b002, ecf0b002, 18d34f0, 18d3520, ecf0b000) + 48
                                    5b7b4058 soft_delete_all_objects_in_session (1, 120f6bc8, 0, ecf00002, ecf00002, 3131c38) + 20
                                    5b7aa37c soft_delete_session (3131c38, 1, 3131c40, 24, 3131c58, 25) + 178
                                    5b7a9f98 soft_delete_all_sessions (0, 3131c38, 1, 5b7a4530, 5b7fa000, 5b7a4530) + 4c
                                    5b7a4370 finalize_common (1, 0, 0, 55cf4, 5b7fa924, 5b7e5e8c) + 6c
                                    5b7a4530 softtoken_fini (ff3f40fc, 1, ff3c5804, 5b7fa000, 55b0c, 5b7fa914) + 44
                                    5b7e5e8c _fini    (ff3f40fc, ff3f5ad8, 2b0a4, 0, ff3f4910, 821) + 4
                                    ff3c5804 call_fini (ff3f40fc, 5aad0208, 5b7e5e88, ff3f42f0, ff3f42a8, ff3f4910) + cc
                                    ff3d01a8 remove_hdl (5aad0208, 4307ef94, 0, 4000, 5aad0188, 4821) + ac8
                                    ff3ca7f0 dlclose_intn (fef01ae8, ff3f4910, ff3f40fc, 2a138, fbc93ad8, 2) + 24
                                    ff3ca8d0 dlclose  (fef01ae8, 0, fef006c0, 0, 6dd350, 1) + 24
                                    fbc93ad8 pkcs11_slottable_delete (2, 600500, 6d3dc0, fbca6bc0, 1, 2) + 138
                                    fbc8e600 pkcs11_fini (fbca6b9c, 1, ff2ba730, fbca6000, 17a44, fbca6b94) + 4c
                                    ff2426ec _postfork_child_handler (1d18, ff335800, 1c00, 4, 4ca4a200, ff335800) + 30
                                    ff2ba730 fork     (0, 10e, 0, 10, ff335800, 4ca4a200) + 144
                                    fe3da328 Java_java_lang_UNIXProcess_forkAndExec (10afad10, 0, 3333e20, 0, 4, 0) + 290
                                    fc00d480 * java/lang/UNIXProcess.forkAndExec([B[BI[BI[BZLjava/io/FileDescriptor;Ljava/io/FileDescriptor;Ljava/io/FileDescriptor;)I+0
                                    fc00d42c * java/lang/UNIXProcess.forkAndExec([B[BI[BI[BZLjava/io/FileDescriptor;Ljava/io/FileDescriptor;Ljava/io/FileDescriptor;)I+0
                                    fc005988 * java/lang/UNIXProcess.<init>([B[BI[BI[BZ)V+62 (line 101)
                                    fc005ab0 * java/lang/ProcessImpl.start([Ljava/lang/String;Ljava/util/Map;Ljava/lang/String;Z)Ljava/lang/Process;+182 (line 119)
                                    fc005940 * java/lang/ProcessBuilder.start()Ljava/lang/Process;+120 (line 896)
                                    fc005940 * java/lang/Runtime.exec([Ljava/lang/String;[Ljava/lang/String;Ljava/io/File;)Ljava/lang/Process;+16 (line 1186)
                                    fc005940 * java/lang/Runtime.exec([Ljava/lang/String;)Ljava/lang/Process;+4 (line 932)
                                    fc005940 * com/nextone/bn/devicecomm/devicemanager/CheckMachineStatusThread.run()V+183 (line 895)
                                    fc00021c * StubRoutines (1)
                                    fe5594fc __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (fc0001c0, 10afac00, 1, e4ae120, 6197fd60, 4307faf8) + 208
                                    fe90af98 __1cJJavaCallsMcall_virtual6FpnJJavaValue_nLKlassHandle_nMsymbolHandle_4pnRJavaCallArguments_pnGThread__v_ (4307faf0, e4ae124, e4ae130, 8a2c00, 4307f9d8, ff75d2f4) + 10c
                                    fe5ecd2c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (4307faf0, 4307faec, 4307fae8, 4307fae4, 4307fae0, e4ae124) + 74
                                    fe5ffe24 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (5b8330b0, 10afac00, 50800, fee4d104, fee4ce38, fee4cb28) + 110
                                    febf99f0 __1cKJavaThreadRthread_main_inner6M_v_ (10afac00, 6ca108, 1095, a, fedfc000, 0) + 48
                                    feb2a198 java_start (10afac00, 55fd, fedfc000, fed4888d, 6ae3e0, fee488b4) + 22c
                                    ff2c88ec _lwp_start (0, 0, 0, 0, 0, 0)
                                    bash-3.00#

                                     

                                     

                                     

                                    bash-3.00# pstack 16238
                                    16238:  /xxx/java -Dprogram.name=run.sh -server -Xms1024m -Xmx
                                    ff2c898c lwp_park (0, 0, 0)
                                    ff2c29a0 cond_wait_queue (d085a70, d085a40, 0, 0, d085a70, 0) + 4c
                                    ff2c2de4 cond_wait_common (d085a70, d085a40, 0, 0, 0, 0) + 294
                                    ff2c2ee8 cond_wait (d085a70, d085a40, ecf0b002, d085a40, d085a70, ecf0b000) + 10
                                    ff2c2f24 pthread_cond_wait (d085a70, d085a40, 0, 0, d085a40, 5b7b4058) + 8
                                    5b7b3f50 soft_delete_object_cleanup (d085910, ecf0b002, ecf0b002, d085a40, d085a70, ecf0b000) + 48
                                    5b7b4058 soft_delete_all_objects_in_session (1, 21f80440, 0, ecf00002, ecf00002, b120298) + 20
                                    5b7aa37c soft_delete_session (b120298, 1, b1202a0, 21, b1202b8, 22) + 178
                                    5b7a9f98 soft_delete_all_sessions (0, b120298, 1, 5b7a4530, 5b7fa000, 5b7a4530) + 4c
                                    5b7a4370 finalize_common (1, 0, 0, 55cf4, 5b7fa924, 5b7e5e8c) + 6c
                                    5b7a4530 softtoken_fini (ff3f40fc, 1, ff3c5804, 5b7fa000, 55b0c, 5b7fa914) + 44
                                    5b7e5e8c _fini    (ff3f40fc, ff3f5ad8, 2b0a4, 0, ff3f4910, 821) + 4
                                    ff3c5804 call_fini (ff3f40fc, 5aad0208, 5b7e5e88, ff3f42f0, ff3f42a8, ff3f4910) + cc
                                    ff3d01a8 remove_hdl (5aad0208, 436ff294, 0, 4000, 5aad0188, 4821) + ac8
                                    ff3ca7f0 dlclose_intn (fef01ae8, ff3f4910, ff3f40fc, 2a138, fbc93ad8, 2) + 24
                                    ff3ca8d0 dlclose  (fef01ae8, 0, fef006c0, 0, 6dd350, 1) + 24
                                    fbc93ad8 pkcs11_slottable_delete (2, 600500, 6d3dc0, fbca6bc0, 1, 2) + 138
                                    fbc8e600 pkcs11_fini (fbca6b9c, 1, ff2ba730, fbca6000, 17a44, fbca6b94) + 4c
                                    ff2426ec _postfork_child_handler (1d18, ff335800, 1c00, 4, 4ca47200, ff335800) + 30
                                    ff2ba730 fork     (0, fc, 0, 10, ff335800, 4ca47200) + 144
                                    fe3da328 Java_java_lang_UNIXProcess_forkAndExec (d6a0d10, 0, 223f938, 0, 4, 0) + 290
                                    fc00d480 * java/lang/UNIXProcess.forkAndExec([B[BI[BI[BZLjava/io/FileDescriptor;Ljava/io/FileDescriptor;Ljava/io/FileDescriptor;)I+0
                                    fc00d42c * java/lang/UNIXProcess.forkAndExec([B[BI[BI[BZLjava/io/FileDescriptor;Ljava/io/FileDescriptor;Ljava/io/FileDescriptor;)I+0
                                    fc005988 * java/lang/UNIXProcess.<init>([B[BI[BI[BZ)V+62 (line 101)
                                    fc005ab0 * java/lang/ProcessImpl.start([Ljava/lang/String;Ljava/util/Map;Ljava/lang/String;Z)Ljava/lang/Process;+182 (line 119)
                                    fc005940 * java/lang/ProcessBuilder.start()Ljava/lang/Process;+120 (line 896)
                                    fc005940 * java/lang/Runtime.exec([Ljava/lang/String;[Ljava/lang/String;Ljava/io/File;)Ljava/lang/Process;+16 (line 1186)
                                    fc005940 * java/lang/Runtime.exec([Ljava/lang/String;)Ljava/lang/Process;+4 (line 932)
                                    fc005940 * com/nextone/bn/devicecomm/devicemanager/CheckMachineStatusThread.run()V+183 (line 895)
                                    fc00021c * StubRoutines (1)
                                    fe5594fc __1cJJavaCallsLcall_helper6FpnJJavaValue_pnMmethodHandle_pnRJavaCallArguments_pnGThread__v_ (fc0001c0, d6a0c00, 1, 21fbda08, 6197fd60, 436ffdf8                        ) + 208
                                    fe90af98 __1cJJavaCallsMcall_virtual6FpnJJavaValue_nLKlassHandle_nMsymbolHandle_4pnRJavaCallArguments_pnGThread__v_ (436ffdf0, 21fbda0c, 21fbda18, 8a2c0                        0, 436ffcd8, ff75d2f4) + 10c
                                    fe5ecd2c __1cJJavaCallsMcall_virtual6FpnJJavaValue_nGHandle_nLKlassHandle_nMsymbolHandle_5pnGThread__v_ (436ffdf0, 436ffdec, 436ffde8, 436ffde4, 436ffde                        0, 21fbda0c) + 74
                                    fe5ffe24 __1cMthread_entry6FpnKJavaThread_pnGThread__v_ (5b8330b0, d6a0c00, 50800, fee4d104, fee4ce38, fee4cb28) + 110
                                    febf99f0 __1cKJavaThreadRthread_main_inner6M_v_ (d6a0c00, 1bcdb30, 2207, b, fedfc000, 0) + 48
                                    feb2a198 java_start (d6a0c00, 55fd, fedfc000, fed4888d, 24981d0, fee488b4) + 22c
                                    ff2c88ec _lwp_start (0, 0, 0, 0, 0, 0)
                                    bash-3.00#

                                    -------------------------


                                    -------------------------


                                    -------------------------

                                    1 2 Previous Next