-
15. Re: It's Strange: many jboss instances are started by run.sh
cluster.lover Jul 5, 2010 6:26 AM (in response to cluster.lover)fc005940 * java/lang/Runtime.exec([Ljava/lang/String;[Ljava/lang/String;Ljava/io/File;)Ljava/lang/Process;+16 (line 1186)
fc005940 * java/lang/Runtime.exec([Ljava/lang/String;)Ljava/lang/Process;+4 (line 932)
fc005940 * com/nextone/bn/devicecomm/devicemanager/CheckMachineStatusThread.run()V+183 (line 895)
fc00021c * StubRoutines (1)~~~~~~~~~~
Is this the cause of this case???
We fixed related code with suggestion from http://www.velocityreviews.com/forums/t130884-process-runtime-exec-causes-subprocess-hang.html
I'll post related testing result here before long.
-
16. Re: It's Strange: many jboss instances are started by run.sh
peterj Jul 5, 2010 2:34 PM (in response to cluster.lover)>>[ybxiang]: sorry. I changed the color to gray.
No! Leave it black! Grey is a color and grey text is harder to read than black text, especially on a litght-grey background as is found in the forums.
-
17. Re: It's Strange: many jboss instances are started by run.sh
cluster.lover Jul 7, 2010 2:35 AM (in response to peterj)[ybxiang]: OK, leave it black.
You are right.
test of memory:
====================[5 jboss instances, 17G swap used]======================
bash-3.00# ps -ef|grep run.sh
root 22013 21992 0 Jul 05 pts/7 2075:51 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
root 16238 22013 0 Jul 05 pts/7 0:00 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
root 9940 16146 0 11:19:03 pts/3 0:00 grep run.sh
root 21992 1 0 Jul 05 pts/7 0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
root 165 22013 0 Jul 05 pts/7 0:00 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
root 25114 22013 0 Jul 05 pts/7 0:00 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
root 13507 22013 0 Jul 06 pts/7 0:00 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
bash-3.00# top
load averages: 2.78, 2.83, 2.82 11:20:08
159 processes: 148 sleeping, 9 stopped, 2 on cpu
CPU states: 88.2% idle, 7.2% user, 4.7% kernel, 0.0% iowait, 0.0% swap
Memory: 8184M real, 917M free, 17G swap in use, 1088M swap freePID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
22013 root 361 59 0 2665M 1971M sleep 34.6H 3.88% java
11635 root 33 60 0 0K 0K sleep 374:47 3.86% ndbd
27405 mysql 73 59 0 0K 0K sleep 386:24 2.05% mysqld
19415 gvadmin 48 59 0 187M 145M sleep 517:45 1.89% java
19380 gvadmin 77 59 0 325M 290M cpu4 313:42 1.06% java
4587 santera 24 120 -20 494M 28M sleep 508:29 0.33% BsDm
485 root 1 150 -20 5400K 2408K sleep 855:59 0.22% mscMonitor
4585 root 8 130 -20 24M 4872K sleep 67:22 0.11% BsPm
421 root 16 106 -20 4768K 2344K sleep 84:45 0.06% syslogd
12211 santera 9 115 -20 120M 29M sleep 4:58 0.05% OamTrap_0801401
11630 root 12 59 0 0K 0K sleep 27:36 0.05% ndb_mgmd
12180 root 59 105 -20 25M 15M sleep 4:41 0.04% timestend
10051 root 1 59 0 2464K 2048K cpu11 0:00 0.04% top
163 root 39 59 0 13M 8688K sleep 117:36 0.04% nscd
4686 santera 76 59 0 346M 250M sleep 46:24 0.03% java -
18. Re: It's Strange: many jboss instances are started by run.sh
cluster.lover Jul 7, 2010 2:36 AM (in response to cluster.lover)Now, i kill one of the jboss instances and show the memory again:
====================[4 jboss instances, 15G swap used]======================
bash-3.00# kill -9 13507
bash-3.00# ps -ef|grep run.sh
root 22013 21992 2 Jul 05 pts/7 2079:42 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
root 16238 22013 0 Jul 05 pts/7 0:00 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
root 21992 1 0 Jul 05 pts/7 0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
root 165 22013 0 Jul 05 pts/7 0:00 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
root 25114 22013 0 Jul 05 pts/7 0:00 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
root 10639 16146 0 11:22:33 pts/3 0:00 grep run.sh
bash-3.00#
bash-3.00# top
load averages: 3.21, 3.06, 2.92 11:24:55
159 processes: 145 sleeping, 9 stopped, 5 on cpu
CPU states: 80.0% idle, 10.7% user, 9.2% kernel, 0.0% iowait, 0.0% swap
Memory: 8184M real, 1406M free, 15G swap in use, 3037M swap freePID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
22013 root 365 59 0 2645M 1949M cpu14 34.7H 8.13% java
27405 mysql 72 44 0 0K 0K sleep 388:08 2.44% mysqld
19415 gvadmin 47 10 0 187M 144M cpu4 519:31 1.97% java
11635 root 33 21 0 0K 0K cpu5 376:20 1.45% ndbd
19380 gvadmin 77 52 0 325M 287M cpu0 314:40 1.14% java
4587 santera 24 120 -20 494M 26M sleep 508:45 0.34% BsDm
11079 root 1 59 0 3408K 2184K sleep 0:01 0.28% ping
485 root 1 150 -20 5400K 2408K sleep 856:10 0.25% mscMonitor
4585 root 8 130 -20 24M 4864K sleep 67:27 0.12% BsPm
12211 santera 9 115 -20 120M 28M sleep 5:00 0.05% OamTrap_0801401
163 root 39 59 0 13M 8568K sleep 117:38 0.05% nscd
11630 root 12 59 0 0K 0K sleep 27:38 0.05% ndb_mgmd
421 root 16 106 -20 4768K 2280K sleep 84:46 0.04% syslogd
10051 root 1 49 0 2592K 2096K cpu11 0:02 0.04% top
19404 gvadmin 38 59 0 164M 138M sleep 175:51 0.03% java -
19. Re: It's Strange: many jboss instances are started by run.sh
cluster.lover Jul 7, 2010 2:37 AM (in response to cluster.lover)Now, kill one again.
====================[3 jboss instances, 13G swap used]======================
bash-3.00# kill -9 25114
bash-3.00# ps -ef|grep run.sh
root 22013 21992 0 Jul 05 pts/7 2083:30 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
root 16238 22013 0 Jul 05 pts/7 0:00 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
root 21992 1 0 Jul 05 pts/7 0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
root 165 22013 0 Jul 05 pts/7 0:00 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
root 11286 16146 0 11:25:47 pts/3 0:00 grep run.sh
bash-3.00#bash-3.00# top
load averages: 3.19, 3.09, 2.95 11:26:46
157 processes: 144 sleeping, 9 stopped, 4 on cpu
CPU states: 76.7% idle, 17.2% user, 6.2% kernel, 0.0% iowait, 0.0% swap
Memory: 8184M real, 1497M free, 13G swap in use, 5041M swap freePID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
22013 root 364 20 0 2642M 1944M cpu11 34.7H 8.15% java
27405 mysql 71 59 0 0K 0K sleep 388:52 2.82% mysqld
19415 gvadmin 46 59 0 187M 143M sleep 520:13 2.36% java
11635 root 33 32 0 0K 0K cpu9 377:12 2.00% ndbd
19380 gvadmin 77 59 0 325M 286M sleep 315:03 1.30% java
4587 santera 24 120 -20 494M 26M sleep 508:51 0.35% BsDm
485 root 1 150 -20 5400K 2408K cpu13 856:15 0.25% mscMonitor
4585 root 8 130 -20 24M 4800K sleep 67:29 0.11% BsPm
12211 santera 9 115 -20 120M 27M sleep 5:01 0.05% OamTrap_0801401
163 root 39 59 0 13M 8304K sleep 117:39 0.05% nscd
11630 root 12 59 0 0K 0K sleep 27:39 0.05% ndb_mgmd
10051 root 1 59 0 2592K 2088K cpu15 0:02 0.04% top
421 root 16 106 -20 4768K 2264K sleep 84:47 0.03% syslogd
19347 gvadmin 34 59 0 62M 37M sleep 9:51 0.03% java
4686 santera 76 59 0 346M 249M sleep 46:26 0.03% java -
20. Re: It's Strange: many jboss instances are started by run.sh
cluster.lover Jul 7, 2010 2:38 AM (in response to cluster.lover)Now, leave only one jboss instance and the "sh .../run.sh":
====================[1 jboss instances, 9G swap used]======================
bash-3.00# kill -9 16238
bash-3.00# kill -9 165
bash-3.00# ps -ef|grep run.sh
root 22013 21992 1 Jul 05 pts/7 2086:32 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
root 11680 16146 0 11:27:58 pts/3 0:00 grep run.sh
root 21992 1 0 Jul 05 pts/7 0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
bash-3.00#bash-3.00# top
load averages: 3.59, 3.23, 3.02 11:28:32
155 processes: 145 sleeping, 9 stopped, 1 on cpu
CPU states: 73.6% idle, 19.1% user, 7.3% kernel, 0.0% iowait, 0.0% swap
Memory: 8184M real, 2045M free, 9216M swap in use, 8950M swap freePID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
22013 root 360 50 0 2710M 1949M sleep 34.8H 7.74% java
11635 root 33 50 0 0K 0K sleep 377:56 3.74% ndbd
19415 gvadmin 46 59 0 187M 143M sleep 520:53 2.35% java
27405 mysql 72 59 0 0K 0K sleep 389:35 2.27% mysqld
19404 gvadmin 39 59 0 164M 139M sleep 176:05 2.19% java
19380 gvadmin 77 59 0 325M 285M sleep 315:24 1.30% java
4587 santera 24 120 -20 494M 26M sleep 508:57 0.36% BsDm
485 root 1 150 -20 5400K 2344K sleep 856:19 0.22% mscMonitor
4585 root 8 130 -20 24M 4864K sleep 67:31 0.12% BsPm
12211 santera 9 115 -20 120M 27M sleep 5:02 0.05% OamTrap_0801401
163 root 39 59 0 13M 8496K sleep 117:40 0.05% nscd
11630 root 12 59 0 0K 0K sleep 27:40 0.05% ndb_mgmd
421 root 16 106 -20 4768K 2264K sleep 84:48 0.04% syslogd
10051 root 1 49 0 2592K 2088K cpu10 0:03 0.04% top
4686 santera 76 59 0 346M 249M sleep 46:27 0.03% java -
21. Re: It's Strange: many jboss instances are started by run.sh
cluster.lover Jul 7, 2010 2:47 AM (in response to cluster.lover)Wait about 30 minutes, another jboss instance is started..........
====================[2 jboss instances, 12G swap used]======================
bash-3.00# ps -ef|grep run.sh
root 22013 21992 0 Jul 05 pts/7 2141:24 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
root 21992 1 0 Jul 05 pts/7 0:00 sh /opt/genband/gvu/01.00.71.04/jboss/bin/run.sh -c gvu -g P228.80.4.221 -u 228
root 14224 22013 0 11:40:26 pts/7 0:00 /.../java -Dprogram.name=run.sh -server -Xms1024m -Xmx2048m -XX
root 20848 16146 0 12:14:26 pts/3 0:00 grep run.sh
bash-3.00# -
22. Re: It's Strange: many jboss instances are started by run.sh
cluster.lover Jul 7, 2010 2:50 AM (in response to cluster.lover)Now, I ask guys to fix all java.lang.Runtime.exec(...) with bellow format.
I'll post my result here after retest it. The application will be run about 4.5 days.
====================[how to run java.lang.Runtime.exec(...)]======================
import java.io.*;public class GVUStreamGobbler implements Runnable {
private Thread thread;
private String name;
private InputStream is;
private String result;public GVUStreamGobbler(String name, InputStream is) {
this.name = name;
this.is = is;
}public void start() {
thread = new Thread(this);
thread.start();
}public void run() {
StringBuffer sb = new StringBuffer();
try {
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
while (true) {
String s = br.readLine();
if (s == null){
break;
}
//
sb.append(s).append("\n");
}
is.close();
} catch (Exception ex) {
ex.printStackTrace();
sb.append("Exception:").append(ex.toString());
}
this.result = sb.toString();
}public String getResult() {
return result;
}
}
public class GVUShellTimerExecutor implements ...{...
public static String execute(String command, String[] envp, String workingDirectory)throws Exception{
String returnResult = null;
Process process = null;
if(workingDirectory==null){
process = Runtime.getRuntime().exec(command, envp,null);
}else{
process = Runtime.getRuntime().exec(command, envp,new File(workingDirectory));
}
GVUStreamGobbler inputStreamReader = new GVUStreamGobbler("InputStreamReader",process.getInputStream());
GVUStreamGobbler errorStreamReader = new GVUStreamGobbler("ErrorStreamReader",process.getErrorStream());
inputStreamReader.start();
errorStreamReader.start();
//
try {
process.waitFor();
} catch (InterruptedException e) {
log.error("shellTimer wait for command execution exception:",e);
}finally{
process.destroy();
}
//
String inputStreamReader_result = inputStreamReader.getResult();
String errorStreamReader_result = errorStreamReader.getResult();
if(inputStreamReader_result!=null && inputStreamReader_result.trim().length()>0){
returnResult = inputStreamReader_result;
}else if(errorStreamReader_result!=null){
returnResult = errorStreamReader_result;
}else{
returnResult = "";
}//returnResult MUST be the true output of the command. Do NOT append anything to it.
return returnResult;
} -
23. Re: It's Strange: many jboss instances are started by run.sh
cluster.lover Jul 18, 2010 10:49 PM (in response to cluster.lover)The final test result:
This "multiple jboss instances" issue is caused by java.lang.Runtime.exec on SOLARIS.
This issue does NOT occurs on windows and linux(SUSE).
Solution:
Read out BOTH inputstream AND errorstream of the process shown above.
Peter Johnson and David Lloyd,
Thank you very much!!!