[cluster.lover]

fc005940 *  java/lang/Runtime.exec([Ljava/lang/String;[Ljava/lang/String;Ljava/io/File;)Ljava/lang/Process;+16  (line 1186)
fc005940 *  java/lang/Runtime.exec([Ljava/lang/String;)Ljava/lang/Process;+4 (line  932)
fc005940 *  com/nextone/bn/devicecomm/devicemanager/CheckMachineStatusThread.run()V+183  (line 895)
fc00021c * StubRoutines (1)

～～～～～～～～～～

Is this the cause of this case???

We fixed related code with suggestion from http://www.velocityreviews.com/forums/t130884-process-runtime-exec-causes-subprocess-hang.html

I'll post related testing result here before long.

[peterj]

   >>[ybxiang]: sorry. I changed the  color to gray.

No! Leave it black! Grey is a color and grey text is harder to read than black text, especially on a litght-grey background as is found in the forums.

[cluster.lover]

[ybxiang]: OK, leave it black.

You are right.

test of memory:

====================[5 jboss instances, 17G swap used]======================
bash-3.00# ps -ef|grep run.sh
    root 22013 21992   0   Jul 05 pts/7    2075:51 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
    root 16238 22013   0   Jul 05 pts/7       0:00 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
    root  9940 16146   0 11:19:03 pts/3       0:00 grep run.sh
    root 21992     1   0   Jul 05 pts/7       0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
    root   165 22013   0   Jul 05 pts/7       0:00 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
    root 25114 22013   0   Jul 05 pts/7       0:00 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
    root 13507 22013   0   Jul 06 pts/7       0:00 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX

   
bash-3.00# top
load averages:  2.78,  2.83,  2.82                                                                                                11:20:08
159 processes: 148 sleeping, 9 stopped, 2 on cpu
CPU states: 88.2% idle,  7.2% user,  4.7% kernel,  0.0% iowait,  0.0% swap
Memory: 8184M real, 917M free, 17G swap in use, 1088M swap free

  PID USERNAME THR PRI NICE  SIZE   RES STATE   TIME    CPU COMMAND
22013 root     361  59    0 2665M 1971M sleep  34.6H  3.88% java
11635 root      33  60    0    0K    0K sleep 374:47  3.86% ndbd
27405 mysql     73  59    0    0K    0K sleep 386:24  2.05% mysqld
19415 gvadmin   48  59    0  187M  145M sleep 517:45  1.89% java
19380 gvadmin   77  59    0  325M  290M cpu4  313:42  1.06% java
4587 santera   24 120  -20  494M   28M sleep 508:29  0.33% BsDm
  485 root       1 150  -20 5400K 2408K sleep 855:59  0.22% mscMonitor
4585 root       8 130  -20   24M 4872K sleep  67:22  0.11% BsPm
  421 root      16 106  -20 4768K 2344K sleep  84:45  0.06% syslogd
12211 santera    9 115  -20  120M   29M sleep   4:58  0.05% OamTrap_0801401
11630 root      12  59    0    0K    0K sleep  27:36  0.05% ndb_mgmd
12180 root      59 105  -20   25M   15M sleep   4:41  0.04% timestend
10051 root       1  59    0 2464K 2048K cpu11   0:00  0.04% top
  163 root      39  59    0   13M 8688K sleep 117:36  0.04% nscd
4686 santera   76  59    0  346M  250M sleep  46:24  0.03% java

[cluster.lover]

Now, i kill one of the jboss instances and show the memory again:

====================[4 jboss instances, 15G swap used]======================
bash-3.00# kill -9 13507
bash-3.00# ps -ef|grep run.sh
    root 22013 21992   2   Jul 05 pts/7    2079:42 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
    root 16238 22013   0   Jul 05 pts/7       0:00 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
    root 21992     1   0   Jul 05 pts/7       0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
    root   165 22013   0   Jul 05 pts/7       0:00 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
    root 25114 22013   0   Jul 05 pts/7       0:00 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
    root 10639 16146   0 11:22:33 pts/3       0:00 grep run.sh
bash-3.00#

bash-3.00# top
load averages:  3.21,  3.06,  2.92                                                                                                11:24:55
159 processes: 145 sleeping, 9 stopped, 5 on cpu
CPU states: 80.0% idle, 10.7% user,  9.2% kernel,  0.0% iowait,  0.0% swap
Memory: 8184M real, 1406M free, 15G swap in use, 3037M swap free

  PID USERNAME THR PRI NICE  SIZE   RES STATE   TIME    CPU COMMAND
22013 root     365  59    0 2645M 1949M cpu14  34.7H  8.13% java
27405 mysql     72  44    0    0K    0K sleep 388:08  2.44% mysqld
19415 gvadmin   47  10    0  187M  144M cpu4  519:31  1.97% java
11635 root      33  21    0    0K    0K cpu5  376:20  1.45% ndbd
19380 gvadmin   77  52    0  325M  287M cpu0  314:40  1.14% java
4587 santera   24 120  -20  494M   26M sleep 508:45  0.34% BsDm
11079 root       1  59    0 3408K 2184K sleep   0:01  0.28% ping
  485 root       1 150  -20 5400K 2408K sleep 856:10  0.25% mscMonitor
4585 root       8 130  -20   24M 4864K sleep  67:27  0.12% BsPm
12211 santera    9 115  -20  120M   28M sleep   5:00  0.05% OamTrap_0801401
  163 root      39  59    0   13M 8568K sleep 117:38  0.05% nscd
11630 root      12  59    0    0K    0K sleep  27:38  0.05% ndb_mgmd
  421 root      16 106  -20 4768K 2280K sleep  84:46  0.04% syslogd
10051 root       1  49    0 2592K 2096K cpu11   0:02  0.04% top
19404 gvadmin   38  59    0  164M  138M sleep 175:51  0.03% java

[cluster.lover]

Now, kill one again.

====================[3 jboss instances, 13G swap used]======================
bash-3.00# kill -9 25114
bash-3.00# ps -ef|grep run.sh
    root 22013 21992   0   Jul 05 pts/7    2083:30 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
    root 16238 22013   0   Jul 05 pts/7       0:00 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
    root 21992     1   0   Jul 05 pts/7       0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
    root   165 22013   0   Jul 05 pts/7       0:00 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
    root 11286 16146   0 11:25:47 pts/3       0:00 grep run.sh
bash-3.00#

bash-3.00# top
load averages:  3.19,  3.09,  2.95                                                                                                11:26:46
157 processes: 144 sleeping, 9 stopped, 4 on cpu
CPU states: 76.7% idle, 17.2% user,  6.2% kernel,  0.0% iowait,  0.0% swap
Memory: 8184M real, 1497M free, 13G swap in use, 5041M swap free

  PID USERNAME THR PRI NICE  SIZE   RES STATE   TIME    CPU COMMAND
22013 root     364  20    0 2642M 1944M cpu11  34.7H  8.15% java
27405 mysql     71  59    0    0K    0K sleep 388:52  2.82% mysqld
19415 gvadmin   46  59    0  187M  143M sleep 520:13  2.36% java
11635 root      33  32    0    0K    0K cpu9  377:12  2.00% ndbd
19380 gvadmin   77  59    0  325M  286M sleep 315:03  1.30% java
4587 santera   24 120  -20  494M   26M sleep 508:51  0.35% BsDm
  485 root       1 150  -20 5400K 2408K cpu13 856:15  0.25% mscMonitor
4585 root       8 130  -20   24M 4800K sleep  67:29  0.11% BsPm
12211 santera    9 115  -20  120M   27M sleep   5:01  0.05% OamTrap_0801401
  163 root      39  59    0   13M 8304K sleep 117:39  0.05% nscd
11630 root      12  59    0    0K    0K sleep  27:39  0.05% ndb_mgmd
10051 root       1  59    0 2592K 2088K cpu15   0:02  0.04% top
  421 root      16 106  -20 4768K 2264K sleep  84:47  0.03% syslogd
19347 gvadmin   34  59    0   62M   37M sleep   9:51  0.03% java
4686 santera   76  59    0  346M  249M sleep  46:26  0.03% java

[cluster.lover]

Now, leave only one jboss instance and the "sh .../run.sh":

====================[1 jboss instances, 9G swap used]======================
bash-3.00# kill -9 16238
bash-3.00# kill -9 165
bash-3.00# ps -ef|grep run.sh
    root 22013 21992   1   Jul 05 pts/7    2086:32 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
    root 11680 16146   0 11:27:58 pts/3       0:00 grep run.sh
    root 21992     1   0   Jul 05 pts/7       0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
bash-3.00#

bash-3.00# top
load averages:  3.59,  3.23,  3.02                                                                                                11:28:32
155 processes: 145 sleeping, 9 stopped, 1 on cpu
CPU states: 73.6% idle, 19.1% user,  7.3% kernel,  0.0% iowait,  0.0% swap
Memory: 8184M real, 2045M free, 9216M swap in use, 8950M swap free

  PID USERNAME THR PRI NICE  SIZE   RES STATE   TIME    CPU COMMAND
22013 root     360  50    0 2710M 1949M sleep  34.8H  7.74% java
11635 root      33  50    0    0K    0K sleep 377:56  3.74% ndbd
19415 gvadmin   46  59    0  187M  143M sleep 520:53  2.35% java
27405 mysql     72  59    0    0K    0K sleep 389:35  2.27% mysqld
19404 gvadmin   39  59    0  164M  139M sleep 176:05  2.19% java
19380 gvadmin   77  59    0  325M  285M sleep 315:24  1.30% java
4587 santera   24 120  -20  494M   26M sleep 508:57  0.36% BsDm
  485 root       1 150  -20 5400K 2344K sleep 856:19  0.22% mscMonitor
4585 root       8 130  -20   24M 4864K sleep  67:31  0.12% BsPm
12211 santera    9 115  -20  120M   27M sleep   5:02  0.05% OamTrap_0801401
  163 root      39  59    0   13M 8496K sleep 117:40  0.05% nscd
11630 root      12  59    0    0K    0K sleep  27:40  0.05% ndb_mgmd
  421 root      16 106  -20 4768K 2264K sleep  84:48  0.04% syslogd
10051 root       1  49    0 2592K 2088K cpu10   0:03  0.04% top
4686 santera   76  59    0  346M  249M sleep  46:27  0.03% java

[cluster.lover]

Wait about 30 minutes, another jboss instance is started..........
====================[2 jboss instances, 12G swap used]======================
bash-3.00# ps -ef|grep run.sh
    root 22013 21992   0   Jul 05 pts/7    2141:24 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
    root 21992     1   0   Jul 05 pts/7       0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
    root 14224 22013   0 11:40:26 pts/7       0:00 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
    root 20848 16146   0 12:14:26 pts/3       0:00 grep run.sh
bash-3.00#

[cluster.lover]

Now, I ask guys to fix all  java.lang.Runtime.exec(...) with bellow format.

I'll post my result here after retest it. The application will be run about 4.5 days.

====================[how to run java.lang.Runtime.exec(...)]======================
import java.io.*;

public  class GVUStreamGobbler implements Runnable {
    private Thread  thread;   
    private String name;
    private InputStream  is;   
    private String result;

    public  GVUStreamGobbler(String name, InputStream is) {
        this.name =  name;
        this.is = is;
    }

    public void  start() {
        thread = new Thread(this);
         thread.start();
    }

    public void run() {
         StringBuffer sb = new StringBuffer();
        try {
             InputStreamReader isr = new InputStreamReader(is);
             BufferedReader br = new BufferedReader(isr);
            while (true)  {
                String s = br.readLine();
                if (s  == null){
                    break;
                }
                 //
                sb.append(s).append("\n");
            }
             is.close();
        } catch (Exception ex) {
             ex.printStackTrace();
             sb.append("Exception:").append(ex.toString());
        }
         this.result = sb.toString();
    }

    public String  getResult() {
        return result;
    }
}

public  class GVUShellTimerExecutor implements ...{

...
    public  static String execute(String command, String[] envp, String  workingDirectory)throws Exception{
        String returnResult =  null;   
       
                   
        Process process  = null;
        if(workingDirectory==null){
            process =  Runtime.getRuntime().exec(command, envp,null);
        }else{
             process = Runtime.getRuntime().exec(command, envp,new  File(workingDirectory));
        }
        GVUStreamGobbler  inputStreamReader = new  GVUStreamGobbler("InputStreamReader",process.getInputStream());
         GVUStreamGobbler errorStreamReader = new  GVUStreamGobbler("ErrorStreamReader",process.getErrorStream());
         inputStreamReader.start();
        errorStreamReader.start();
         //
        try {         
              process.waitFor();            
        } catch (InterruptedException  e) {
            log.error("shellTimer wait for command execution  exception:",e);
        }finally{
            process.destroy();
         }
        //
        String inputStreamReader_result =  inputStreamReader.getResult();
        String  errorStreamReader_result = errorStreamReader.getResult();
         if(inputStreamReader_result!=null &&  inputStreamReader_result.trim().length()>0){
             returnResult = inputStreamReader_result;
        }else  if(errorStreamReader_result!=null){
            returnResult =  errorStreamReader_result;
        }else{
            returnResult =  "";
        }

        //returnResult MUST be the  true output of the command. Do NOT append anything to it.
         return returnResult;   
    }

[cluster.lover]

The final test result:

     This "multiple jboss instances" issue is caused by java.lang.Runtime.exec on SOLARIS.

     This issue does NOT occurs on windows and linux(SUSE).

Solution:

     Read out BOTH inputstream AND errorstream of the process shown above.

Peter Johnson and David Lloyd,

     Thank you very much!!!