shutdown.bat and cntrl-c not working but web-console and jmx

[amast]

When I run shutdown.bat on Windows nothing is happening(our shutdown.sh on linux works fine).

I have used EVERY flag/parameter in the shutdown class in every possible fashion as described in just about any forum I could find.

The java command is being launched. If I put in the wrong types of parameters, it barks. If I echo and run the full java command, it just sits there forever!

I activated the .bat debug line and it just sits at Listening for transport dt_socket at address: 8787".

If I hit cntrl-c, it doesn't shutdown, but rather says 'Terminate batch job (Y/N)?" AND regardless of Y/N, it stops jboss. Nice, eh?

Yet, if I log into the web-console, I can restart from the applet and I can run the shutdown() process from the jmx console.

What is the web-console or jmx console doing that the script isn't? I merely need to "programatically" stop the service. I'm losing the only few hairs I have left on my head! :)

[peterj]

If you have not changed the ports JBossAS is using, you should be able to stop JBossAS (running on the same machine) using:

shutdown -S

Are you running shutdown on the same machine on which JBossAS is running? If not, did you start JBossAS using the "-b 0.0.0.0" option? If you did, then stopping a remote JBossAS can be done using:

shutdown -S --server=hostname

If you tried the above and still have problems, do this:

1) run netstat (or tcpview) and note the ports opened by JBossAS (they will be assigned to 'java', if you are running multiple java apps, you will have to also look at the process id to identify the ports assigned to JBossAS)

2) Post the java command run when you run shutdown.bat

3) Once shutdown.bat hangs, type CTRL-Break and post the thread dump.

[amast]

First, I'd like to thank you for your clarity and comprehensive series of tests. I appreciate it greatly.

"PeterJ" wrote:
If you have not changed the ports JBossAS is using, you should be able to stop JBossAS (running on the same machine) using:

shutdown -S

Well, I'm not sure if we changed the ports JBossAS is using, but I do know that during our installation, we are required to enter an unused http, https, and AJP port. I used 14080 for the http, which you'll see in the netstat below. I tried shutdown -S from the same machine on another console to no avail.

"PeterJ" wrote:

Are you running shutdown on the same machine on which JBossAS is running? If not, did you start JBossAS using the "-b 0.0.0.0" option? If you did, then stopping a remote JBossAS can be done using:

shutdown -S --server=hostname

Yes, I am running shutdown on the same machine. Yes, I do start JBossAS using the "-b 0.0.0.0" option. When attempting to run from a remote location, I get an connectException that localhost:1099 is unreachable. If I add hostname:port to the -s flag, it takes a while and throws a failed to receive stub from server:port.

"PeterJ" wrote:

If you tried the above and still have problems, do this:

1) run netstat (or tcpview) and note the ports opened by JBossAS (they will be assigned to 'java', if you are running multiple java apps, you will have to also look at the process id to identify the ports assigned to JBossAS)

Here you go... fortunately I'm only running one java.exe process: (edited for privacy)
TCP JBossServer:2847 JBossServer.my.domain.com:0 LISTENING 3440 [java.exe]
TCP JBossServer:2848 JBossServer.my.domain.com:0 LISTENING 3440 [java.exe]
TCP JBossServer:2849 JBossServer.my.domain.com:0 LISTENING 3440 [java.exe]
TCP JBossServer:9098 JBossServer.my.domain.com:0 LISTENING 3440 [java.exe]
TCP JBossServer:9099 JBossServer.my.domain.com:0 LISTENING 3440 [java.exe]
TCP JBossServer:12444 JBossServer.my.domain.com:0 LISTENING 3440 [java.exe]
TCP JBossServer:12445 JBossServer.my.domain.com:0 LISTENING 3440 [java.exe]
TCP JBossServer:12446 JBossServer.my.domain.com:0 LISTENING 3440 [java.exe]
TCP JBossServer:14009 JBossServer.my.domain.com:0 LISTENING 3440 [java.exe]
TCP JBossServer:14080 JBossServer.my.domain.com:0 LISTENING 3440 [java.exe]
TCP JBossServer:16083 JBossServer.my.domain.com:0 LISTENING 3440 [java.exe]
TCP JBossServer:16093 JBossServer.my.domain.com:0 LISTENING 3440 [java.exe]
TCP JBossServer:11873 JBossServer.my.domain.com:0 LISTENING 3440 [java.exe]

For fun, I ran "shutdown -s JBossServer:port -S" on each of those ports and all of them sat there, as they have.

"PeterJ" wrote:

2) Post the java command run when you run shutdown.bat

3) Once shutdown.bat hangs, type CTRL-Break and post the thread dump.


Here's that Java command .bat is running and the resulting thread dump. (thanks for that cntrl-Break tip, very handy!)
C:\myProgram\instance8\OITAppServer\bin>shutdown -S --server=jsmith
"..\jvm\win32\bin\java" -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=y -Djboss.boot.loader.name=shutdown.bat -classpath ";C:\myProgram\instance8\AppServer\bin\\shutdown.jar;C:\myProgram\instance8\AppServer\bin\/../client/jbossall-client.jar" org.jboss.Shutdown -S --server jsmith

Listening for transport dt_socket at address: 8787

===== FULL THREAD DUMP ===============
Wed Jun 25 07:38:02 2008
BEA JRockit(R) R27.4.0-90-89592-1.6.0_02-20070928-1715-windows-ia32

"Main Thread" id=1 idx=0x4 tid=716 prio=5 alive, in native, native_waiting

"(Signal Handler)" id=2 idx=0x8 tid=3188 prio=5 alive, in native, daemon

"(GC Main Thread)" id=3 idx=0xc tid=1988 prio=5 alive, in native, native_waiting, daemon

"(GC Worker Thread 1)" id=? idx=0x10 tid=1044 prio=5 alive, in native, daemon

"(Code Generation Thread 1)" id=4 idx=0x14 tid=2780 prio=5 alive, in native, native_waiting, daemon

"(Code Optimization Thread 1)" id=5 idx=0x18 tid=324 prio=5 alive, in native, native_waiting, daemon

"(VM Periodic Task)" id=6 idx=0x1c tid=4012 prio=10 alive, in native, daemon

"(Attach Listener)" id=7 idx=0x20 tid=612 prio=5 alive, in native, daemon

"Finalizer" id=8 idx=0x24 tid=3380 prio=8 alive, in native, native_waiting, daemon
at jrockit/memory/Finalizer.waitForFinalizees([Ljava/lang/Object;)I(Native Method)
at jrockit/memory/Finalizer.access$500(Finalizer.java:12)
at jrockit/memory/Finalizer$4.run(Finalizer.java:143)
at java/lang/Thread.run(Thread.java:619)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace

"Reference Handler" id=9 idx=0x28 tid=3116 prio=10 alive, in native, native_waiting, daemon
at java/lang/ref/Reference.getPending()Ljava/lang/ref/Reference;(Native Method)
at java/lang/ref/Reference.access$000(Reference.java:24)
at java/lang/ref/Reference$ReferenceHandler.run(Reference.java:144)
at jrockit/vm/RNI.c2java(IIIII)V(Native Method)
-- end of trace

"(Sensor Event Thread)" id=10 idx=0x2c tid=3360 prio=5 alive, in native, daemon

"JDWP Transport Listener: dt_socket" id=11 idx=0x30 tid=2352 prio=10 alive, in native, daemon

"JDWP Event Helper Thread" id=12 idx=0x34 tid=2792 prio=10 alive, in native, native_waiting, daemon

===== END OF THREAD DUMP ===============


This was the netstat -ab

TCP jsmith:2847 jsmith.OIT.DocFinity.com:0 LISTENING 3440 [java.exe]
TCP jsmith:2848 jsmith.OIT.DocFinity.com:0 LISTENING 3440 [java.exe]
TCP jsmith:2849 jsmith.OIT.DocFinity.com:0 LISTENING 3440 [java.exe]
TCP jsmith:9098 jsmith.OIT.DocFinity.com:0 LISTENING 3440 [java.exe]
TCP jsmith:9099 jsmith.OIT.DocFinity.com:0 LISTENING 3440 [java.exe]
TCP jsmith:12444 jsmith.OIT.DocFinity.com:0 LISTENING 3440 [java.exe]
TCP jsmith:12445 jsmith.OIT.DocFinity.com:0 LISTENING 3440 [java.exe]
TCP jsmith:12446 jsmith.OIT.DocFinity.com:0 LISTENING 3440 [java.exe]
TCP jsmith:14009 jsmith.OIT.DocFinity.com:0 LISTENING 3440 [java.exe]
TCP jsmith:14080 jsmith.OIT.DocFinity.com:0 LISTENING 3440 [java.exe]
TCP jsmith:16083 jsmith.OIT.DocFinity.com:0 LISTENING 3440 [java.exe]
TCP jsmith:16093 jsmith.OIT.DocFinity.com:0 LISTENING 3440 [java.exe]
TCP jsmith:11873 jsmith.OIT.DocFinity.com:0 LISTENING 3440 [java.exe]

I also have ESTABLISHED connections but I didn't include them because they simply were database connections for process id 3440 (above).

Thanks Again!

[peterj]

Based on your netstat output, I think that you changed the JNDI port to 9099 (the default is 1099). So you should be able to shut down from the same machine using:

shutdown -S -s localhost:9099

To access the port remotely you need to ensure that port 9099 is open. You will also need ports 9098, 12444, 12445 and possibly 12446 open (this is a guess based on the port-change pattern I think you used).

I see you are using jrockit. Could you try with a Sun JDK?

If those don't work, I have a few more ideas.

Please don't bold your text, makes it too difficult to read.

[amast]

Thanks again for your persistence!

"PeterJ" wrote:
Based on your netstat output, I think that you changed the JNDI port to 9099 (the default is 1099). So you should be able to shut down from the same machine using:
shutdown -S -s localhost:9099

That didn't shutdown, same "pause"

"PeterJ" wrote:

Could you try with a Sun JDK?

Yea, I tried that, but the same result (thread break below).

Listening for transport dt_socket at address: 8787
2008-06-26 08:30:27
Full thread dump Java HotSpot(TM) Client VM (10.0-b19 mixed mode):

"JDWP Event Helper Thread" daemon prio=6 tid=0x0aac5800 nid=0xf1c runnable [0x00000000..0x00000000]
java.lang.Thread.State: RUNNABLE

"JDWP Transport Listener: dt_socket" daemon prio=6 tid=0x0aac2800 nid=0x650 runnable [0x00000000..0x00000000]
java.lang.Thread.State: RUNNABLE

"Attach Listener" daemon prio=10 tid=0x0aab7800 nid=0x2ac runnable [0x00000000..0x00000000]
java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x0aab6c00 nid=0xfc4 waiting on condition [0x00000000..0x00000000]
java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=8 tid=0x0aaa9000 nid=0xed8 in Object.wait() [0x0ac1f000..0x0ac1fa98]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x02990b38> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
- locked <0x02990b38> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(Unknown Source)
at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

"Reference Handler" daemon prio=10 tid=0x0aaa4800 nid=0xf40 in Object.wait() [0x0abcf000..0x0abcfb18]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x02990a40> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
- locked <0x02990a40> (a java.lang.ref.Reference$Lock)

"main" prio=6 tid=0x00295c00 nid=0x204 runnable [0x00000000..0x0090fdec]
java.lang.Thread.State: RUNNABLE

"VM Thread" prio=10 tid=0x0aaa0000 nid=0x450 runnable


JNI global references: 324

Heap
def new generation total 960K, used 157K [0x02990000, 0x02a90000, 0x02e70000)
eden space 896K, 17% used [0x02990000, 0x029b7790, 0x02a70000)
from space 64K, 0% used [0x02a70000, 0x02a70000, 0x02a80000)
to space 64K, 0% used [0x02a80000, 0x02a80000, 0x02a90000)
tenured generation total 4096K, used 0K [0x02e70000, 0x03270000, 0x06990000)
the space 4096K, 0% used [0x02e70000, 0x02e70000, 0x02e70200, 0x03270000)
compacting perm gen total 12288K, used 1450K [0x06990000, 0x07590000, 0x0a990000)
the space 12288K, 11% used [0x06990000, 0x06afabb0, 0x06afac00, 0x07590000)
No shared spaces configured.

"PeterJ" wrote:

Please don't bold your text, makes it too difficult to read.

Sorry!

[amast]

Good news! I found a port-bindings.xml file and run the shutdown in each of them, FINALLY finding one that would respond. Thanks a bunch for helping me out. Very much appreciated!

[peterj]

Yes, it is critical to know the JNDI port to help shut JBossAS down.

If I recall correctly, some older versions of JBossAS used to log the JNDI port to the console log, but the more recent versions do not appear to do that. Knowing the JNDI port is somewhat crucial. Wonder if there is an MBean that has that info? Yes there is: jboss:service=Naming, the Port attribute.

[amast]

Thanks! Alright, very cool info. I'll run with it. Have a good one.