6 Replies Latest reply on Nov 24, 2006 2:25 AM by svenkataraman_celequest

    Shutting down JBOSS hangs..

    svenkataraman_celequest

      We are running multiple instances of JBOSS (3.2.6) on the same box. We use the shutdown script with the argument "-s jnp://localhost:<jnp_port>" to shut down. The command never completes. We do not see the usual, "message sent, check logs" statement.

      Alternatively we tried issuing a "kill ". This caused the following to occur but hung at that point. The process never terminated.

      18:42:51,270 INFO [Server] JBoss (MX MicroKernel) [3.2.6 (build: CVSTag=JBoss_3_2_6 date=200410140106)] Started in 9s:750ms
      18:44:03,290 INFO [Server] JBoss SHUTDOWN: Undeploying all packages
      18:44:03,291 INFO [TomcatDeployer] undeploy, ctxPath=/web-console, warUrl=file:/usr/master/jboss-3.2.6/server/default/deploy/management/web-console.war/
      18:44:03,306 INFO [TomcatDeployer] undeploy, ctxPath=/jmx-console, warUrl=file:/usr/master/jboss-3.2.6/server/default/deploy/jmx-console.war/

      Any ideas?

      - Shankar

        • 1. Re: Shutting down JBOSS hangs..
          genman

          Do you have a stack trace of where it's stuck?

          • 2. Re: Shutting down JBOSS hangs..
            svenkataraman_celequest

            Please find below, the thread dump of both the JBoss instance and the shutdown process. I see the main thread waiting on socketConnect in the shutdown process dump.

            THREAD DUMP OF RUNNING JBOSS INSTANCE :

            /usr/java/jdk1.5.0_06
            =========================================================================

            JBoss Bootstrap Environment

            JBOSS_HOME: /usr/master/jboss-3.2.6

            JAVA: /usr/java/jdk1.5.0_06/bin/java

            JAVA_OPTS: -server -D64 -Dfile.encoding=utf8 -Xms2000m -Xmx2000m -Dsun.rmi.dgc.server.gcInterval=600000 -Dsun.rmi.dgc.client.gcInterval=600000 -XX:ThreadStackSize=150 -Dprogram.name=run.sh -Dorg.jboss.logging.Log4jService.catchSystemOut=false -Dorg.jboss.logging.Log4jService.catchSystemErr=false

            JAVA_DEBUG_OPTS:

            JAVA_GCOPTS: -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=4 -XX:NewRatio=2

            CLASSPATH: /usr/master/jboss-3.2.6/bin/run.jar:/usr/java/jdk1.5.0_06/lib/tools.jar

            =========================================================================

            17:10:25,071 INFO [Server] Starting JBoss (MX MicroKernel)...
            17:10:25,072 INFO [Server] Release ID: JBoss [WonderLand] 3.2.6 (build: CVSTag=JBoss_3_2_6 date=200410140106)
            17:10:25,072 INFO [Server] Home Dir: /usr/master/jboss-3.2.6
            17:10:25,073 INFO [Server] Home URL: file:/usr/master/jboss-3.2.6/
            17:10:25,073 INFO [Server] Library URL: file:/usr/master/jboss-3.2.6/lib/
            17:10:25,074 INFO [Server] Patch URL: null
            17:10:25,074 INFO [Server] Server Name: default
            17:10:25,074 INFO [Server] Server Home Dir: /usr/master/jboss-3.2.6/server/default
            17:10:25,074 INFO [Server] Server Home URL: file:/usr/master/jboss-3.2.6/server/default/
            17:10:25,074 INFO [Server] Server Data Dir: /usr/master/jboss-3.2.6/server/default/data
            17:10:25,075 INFO [Server] Server Temp Dir: /usr/master/jboss-3.2.6/server/default/tmp
            17:10:25,075 INFO [Server] Server Config URL: file:/usr/master/jboss-3.2.6/server/default/conf/
            17:10:25,078 INFO [Server] Server Library URL: file:/usr/master/jboss-3.2.6/server/default/lib/
            17:10:25,078 INFO [Server] Root Deployment Filename: jboss-service.xml
            17:10:25,081 INFO [Server] Starting General Purpose Architecture (GPA)...
            17:10:25,337 INFO [ServerInfo] Java version: 1.5.0_06,Sun Microsystems Inc.
            17:10:25,337 INFO [ServerInfo] Java VM: Java HotSpot(TM) 64-Bit Server VM 1.5.0_06-b05,Sun Microsystems Inc.
            17:10:25,337 INFO [ServerInfo] OS-System: Linux 2.6.9-42.0.3.ELlargesmp,amd64
            17:10:25,653 INFO [Server] Core system initialized
            17:10:28,472 INFO [Log4jService$URLWatchTimerTask] Configuring from URL: resource:log4j.xml
            17:10:28,570 INFO [WebService] Using RMI server codebase: http://103894-app1.app1.celequest.com:8083/
            17:10:28,906 INFO [NamingService] Started jndi bootstrap jnpPort=1099, rmiPort=1098, backlog=50, bindAddress=/0.0.0.0, Client SocketFactory=null, Server SocketFactory=org.jboss.net.sockets.DefaultSocketFactory@ad093076
            17:10:31,039 INFO [RARMetaData] Loading JBoss Resource Adapter for JDBC 2 XA drivers
            17:10:31,039 INFO [RARMetaData] Required license terms present. See deployment descriptor.
            17:10:32,915 INFO [MailService] Mail Service bound to java:/Mail
            17:10:33,202 INFO [Embedded] Catalina naming disabled
            17:10:33,541 INFO [Http11Protocol] Initializing Coyote HTTP/1.1 on http-0.0.0.0-8080
            17:10:33,564 INFO [Catalina] Initialization processed in 322 ms
            17:10:33,564 INFO [StandardService] Starting service jboss.web
            17:10:33,567 INFO [StandardEngine] Starting Servlet Engine: Apache Tomcat/5.0.28
            17:10:33,580 INFO [StandardHost] XML validation disabled
            17:10:33,595 INFO [Catalina] Server startup in 31 ms
            17:10:33,652 INFO [TomcatDeployer] deploy, ctxPath=/invoker, warUrl=file:/usr/master/jboss-3.2.6/server/default/deploy/http-invoker.sar/invoker.war/
            17:10:34,115 INFO [TomcatDeployer] deploy, ctxPath=/, warUrl=file:/usr/master/jboss-3.2.6/server/default/deploy/jbossweb-tomcat50.sar/ROOT.war/
            17:10:34,216 INFO [TomcatDeployer] deploy, ctxPath=/jbossmq-httpil, warUrl=file:/usr/master/jboss-3.2.6/server/default/deploy/jms/jbossmq-httpil.sar/jbossmq-httpil.war/
            17:10:34,306 INFO [context] Bound connection factory for resource adapter for ConnectionManager 'jboss.jca:service=NoTxCM,name=com.celequest.context to JNDI name 'java:/com.celequest.context'
            17:10:34,308 INFO [metaDatasource] Bound connection factory for resource adapter for ConnectionManager 'jboss.jca:service=NoTxCM,name=com.celequest.metadata.metaDatasource to JNDI name 'java:/com.celequest.metadata.metaDatasource'
            17:10:34,324 INFO [DefaultDS] Bound connection factory for resource adapter for ConnectionManager 'jboss.jca:service=LocalTxCM,name=DefaultDS to JNDI name 'java:/DefaultDS'
            17:10:34,485 INFO [A] Bound to JNDI name: queue/A
            17:10:34,487 INFO [B] Bound to JNDI name: queue/B
            17:10:34,489 INFO [C] Bound to JNDI name: queue/C
            17:10:34,490 INFO [D] Bound to JNDI name: queue/D
            17:10:34,492 INFO [ex] Bound to JNDI name: queue/ex
            17:10:34,513 INFO [testTopic] Bound to JNDI name: topic/testTopic
            17:10:34,516 INFO [securedTopic] Bound to JNDI name: topic/securedTopic
            17:10:34,517 INFO [testDurableTopic] Bound to JNDI name: topic/testDurableTopic
            17:10:34,520 INFO [testQueue] Bound to JNDI name: queue/testQueue
            17:10:34,568 INFO [UILServerILService] JBossMQ UIL service available at : /0.0.0.0:8093
            17:10:34,601 INFO [DLQ] Bound to JNDI name: queue/DLQ
            17:10:34,616 INFO [JmsXA] Bound connection factory for resource adapter for ConnectionManager 'jboss.jca:service=TxCM,name=JmsXA to JNDI name 'java:/JmsXA'
            17:10:34,685 INFO [TomcatDeployer] deploy, ctxPath=/jmx-console, warUrl=file:/usr/master/jboss-3.2.6/server/default/deploy/jmx-console.war/
            17:10:34,844 INFO [TomcatDeployer] deploy, ctxPath=/web-console, warUrl=file:/usr/master/jboss-3.2.6/server/default/deploy/management/web-console.war/
            17:10:35,280 INFO [Http11Protocol] Starting Coyote HTTP/1.1 on http-0.0.0.0-8080
            17:10:35,363 INFO [ChannelSocket] JK2: ajp13 listening on /0.0.0.0:8009
            17:10:35,367 INFO [JkMain] Jk running ID=0 time=0/20 config=null
            17:10:35,376 INFO [Server] JBoss (MX MicroKernel) [3.2.6 (build: CVSTag=JBoss_3_2_6 date=200410140106)] Started in 10s:173ms
            Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.5.0_06-b05 mixed mode):

            "TP-Monitor" daemon prio=1 tid=0x0000002b67e8bac0 nid=0x6889 in Object.wait() [0x00000000412a5000..0x00000000412a5cb0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002adadeac90> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
            at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:559)
            - locked <0x0000002adadeac90> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "TP-Processor4" daemon prio=1 tid=0x0000002b671612f0 nid=0x6888 runnable [0x000000004127f000..0x000000004127fc30]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0x0000002adadde910> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:295)
            at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:638)
            at org.apache.jk.common.SocketAcceptor.runIt(ChannelSocket.java:847)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
            at java.lang.Thread.run(Thread.java:595)

            "TP-Processor3" daemon prio=1 tid=0x0000002b67161040 nid=0x6887 in Object.wait() [0x0000000041259000..0x0000000041259db0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002adade9e00> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002adade9e00> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "TP-Processor2" daemon prio=1 tid=0x0000002b67163400 nid=0x6886 in Object.wait() [0x0000000041233000..0x0000000041233d30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002adade96b8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002adade96b8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "TP-Processor1" daemon prio=1 tid=0x0000002b668bb250 nid=0x6885 in Object.wait() [0x000000004120d000..0x000000004120deb0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002adade8f70> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002adade8f70> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Monitor" prio=1 tid=0x0000002b5f7d0890 nid=0x6884 in Object.wait() [0x00000000411e7000..0x00000000411e7e30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada8a9218> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
            at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:559)
            - locked <0x0000002ada8a9218> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor25" daemon prio=1 tid=0x0000002b5f7cf480 nid=0x6883 runnable [0x00000000411c1000..0x00000000411c1bb0]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0x0000002adca9c0a8> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
            at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:368)
            at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:549)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor24" daemon prio=1 tid=0x0000002b654a43a0 nid=0x6882 in Object.wait() [0x000000004119b000..0x000000004119bb30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada8a8248> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada8a8248> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor23" daemon prio=1 tid=0x0000002b654a2f90 nid=0x6881 in Object.wait() [0x0000000041175000..0x0000000041175cb0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada8a7a60> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada8a7a60> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor22" daemon prio=1 tid=0x0000002b64e1a510 nid=0x6880 in Object.wait() [0x000000004114f000..0x000000004114fc30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada8a7278> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada8a7278> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor21" daemon prio=1 tid=0x0000002b64e19170 nid=0x687f in Object.wait() [0x0000000041129000..0x0000000041129db0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada8a6a90> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada8a6a90> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor20" daemon prio=1 tid=0x0000002b63312c70 nid=0x687e in Object.wait() [0x0000000041103000..0x0000000041103d30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada8a6090> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada8a6090> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor19" daemon prio=1 tid=0x0000002b63311890 nid=0x687d in Object.wait() [0x00000000410dd000..0x00000000410ddeb0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada8a58a8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada8a58a8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor18" daemon prio=1 tid=0x0000002b5f7d5de0 nid=0x687c in Object.wait() [0x00000000410b7000..0x00000000410b7e30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada8a4f30> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada8a4f30> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor17" daemon prio=1 tid=0x0000002b5f7d49d0 nid=0x687b in Object.wait() [0x0000000041091000..0x0000000041091bb0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada8a4748> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada8a4748> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor16" daemon prio=1 tid=0x0000002b637e9240 nid=0x687a in Object.wait() [0x000000004106b000..0x000000004106bb30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada8a3f60> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada8a3f60> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor15" daemon prio=1 tid=0x0000002b6333dcd0 nid=0x6879 in Object.wait() [0x0000000041045000..0x0000000041045cb0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada8a3778> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada8a3778> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor14" daemon prio=1 tid=0x0000002b6333c980 nid=0x6878 in Object.wait() [0x000000004101f000..0x000000004101fc30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada8a2f90> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada8a2f90> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor13" daemon prio=1 tid=0x0000002b6375edc0 nid=0x6877 in Object.wait() [0x0000000040ff9000..0x0000000040ff9db0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada8a27a8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada8a27a8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor12" daemon prio=1 tid=0x0000002b6375d9b0 nid=0x6876 in Object.wait() [0x0000000040fd3000..0x0000000040fd3d30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada8a1fc0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada8a1fc0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor11" daemon prio=1 tid=0x0000002b6375cfd0 nid=0x6875 in Object.wait() [0x0000000040fad000..0x0000000040fadeb0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada8a17d8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada8a17d8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor10" daemon prio=1 tid=0x0000002b63d8b930 nid=0x6874 in Object.wait() [0x0000000040f87000..0x0000000040f87e30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada8a0ff0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada8a0ff0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor9" daemon prio=1 tid=0x0000002b62ee3e50 nid=0x6873 in Object.wait() [0x0000000040f61000..0x0000000040f61bb0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada8a0748> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada8a0748> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor8" daemon prio=1 tid=0x0000002b62ee2a40 nid=0x6872 in Object.wait() [0x0000000040f3b000..0x0000000040f3bb30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada89ff70> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada89ff70> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor7" daemon prio=1 tid=0x0000002b6359e5d0 nid=0x6871 in Object.wait() [0x0000000040f15000..0x0000000040f15cb0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada89f798> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada89f798> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor6" daemon prio=1 tid=0x0000002b6359d640 nid=0x6870 in Object.wait() [0x0000000040eef000..0x0000000040eefc30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada89efc0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada89efc0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor5" daemon prio=1 tid=0x0000002b63afaae0 nid=0x686f in Object.wait() [0x0000000040ec9000..0x0000000040ec9db0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada89e7f8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada89e7f8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor4" daemon prio=1 tid=0x0000002b63af9930 nid=0x686e in Object.wait() [0x0000000040ea3000..0x0000000040ea3d30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada89df18> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada89df18> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor3" daemon prio=1 tid=0x0000002b65778e60 nid=0x686d in Object.wait() [0x0000000040e7d000..0x0000000040e7deb0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada89d750> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada89d750> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor2" daemon prio=1 tid=0x0000002b64ef3330 nid=0x686c in Object.wait() [0x0000000040e57000..0x0000000040e57e30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada89cf88> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada89cf88> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "http-0.0.0.0-8080-Processor1" daemon prio=1 tid=0x0000002b66a74060 nid=0x686b in Object.wait() [0x0000000040e31000..0x0000000040e31bb0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada89c7c0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:655)
            - locked <0x0000002ada89c7c0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
            at java.lang.Thread.run(Thread.java:595)

            "JBossLifeThread" prio=1 tid=0x0000002b670fbd30 nid=0x686a in Object.wait() [0x0000000040e0b000..0x0000000040e0bb30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002ada7750c8> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:474)
            at org.jboss.system.server.ServerImpl$LifeThread.run(ServerImpl.java:824)
            - locked <0x0000002ada7750c8> (a java.lang.Object)

            "UILServerILService Accept Thread" prio=1 tid=0x0000002b64eb4ce0 nid=0x6869 runnable [0x0000000040de5000..0x0000000040de5cb0]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0x0000002adc90bea0> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at org.jboss.mq.il.uil2.UILServerILService.run(UILServerILService.java:123)
            at java.lang.Thread.run(Thread.java:595)

            "JCA PoolFiller" prio=1 tid=0x0000002b63b150f0 nid=0x6868 in Object.wait() [0x0000000040dbf000..0x0000000040dbfc30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002addaea818> (a java.util.LinkedList)
            at java.lang.Object.wait(Object.java:474)
            at org.jboss.resource.connectionmanager.PoolFiller.run(PoolFiller.java:81)
            - locked <0x0000002addaea818> (a java.util.LinkedList)
            at java.lang.Thread.run(Thread.java:595)

            "IdleRemover" prio=1 tid=0x0000002b6530da40 nid=0x6867 in Object.wait() [0x0000000040d99000..0x0000000040d99db0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002addae3ae0> (a java.util.ArrayList)
            at org.jboss.resource.connectionmanager.IdleRemover$1.run(IdleRemover.java:66)
            - locked <0x0000002addae3ae0> (a java.util.ArrayList)
            at java.lang.Thread.run(Thread.java:595)

            "TimeoutFactory" daemon prio=1 tid=0x0000002b635b5240 nid=0x6866 in Object.wait() [0x0000000040d73000..0x0000000040d73d30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002addaccb30> (a org.jboss.util.timeout.TimeoutFactory)
            at org.jboss.util.timeout.TimeoutFactory.doWork(TimeoutFactory.java:503)
            - locked <0x0000002addaccb30> (a org.jboss.util.timeout.TimeoutFactory)
            at org.jboss.util.timeout.TimeoutFactory.access$300(TimeoutFactory.java:28)
            at org.jboss.util.timeout.TimeoutFactory$1.run(TimeoutFactory.java:549)

            "Timer-2" daemon prio=1 tid=0x0000002b6332c0f0 nid=0x6865 in Object.wait() [0x0000000040d4d000..0x0000000040d4deb0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002addae5d40> (a java.util.TaskQueue)
            at java.util.TimerThread.mainLoop(Timer.java:509)
            - locked <0x0000002addae5d40> (a java.util.TaskQueue)
            at java.util.TimerThread.run(Timer.java:462)

            "ContainerBackgroundProcessor[StandardEngine[jboss.web]]" daemon prio=1 tid=0x0000002b6357c920 nid=0x6864 waiting on condition [0x0000000040d27000..0x0000000040d27e30]
            at java.lang.Thread.sleep(Native Method)
            at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1597)
            at java.lang.Thread.run(Thread.java:595)

            "JBossMQ Cache Reference Softner" daemon prio=1 tid=0x0000002b65628080 nid=0x6863 in Object.wait() [0x0000000040d01000..0x0000000040d01bb0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002adc73abf8> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
            - locked <0x0000002adc73abf8> (a java.lang.ref.ReferenceQueue$Lock)
            at org.jboss.mq.server.MessageCache.run(MessageCache.java:241)
            at java.lang.Thread.run(Thread.java:595)

            "HSQLDB Timer @38e580e1" daemon prio=1 tid=0x0000002b639bfa30 nid=0x6862 in Object.wait() [0x0000000040cdb000..0x0000000040cdbb30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002adc2cef30> (a org.hsqldb.lib.HsqlTimer)
            at org.hsqldb.lib.HsqlTimer.nextTask(Unknown Source)
            - locked <0x0000002adc2cef30> (a org.hsqldb.lib.HsqlTimer)
            at org.hsqldb.lib.HsqlTimer$TaskRunner.run(Unknown Source)
            at java.lang.Thread.run(Thread.java:595)

            "PooledInvokerAcceptor#0-4445" prio=1 tid=0x0000002b65f1a720 nid=0x6751 runnable [0x0000000040cb5000..0x0000000040cb5cb0]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0x0000002adbc80be0> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at org.jboss.invocation.pooled.server.PooledInvoker.run(PooledInvoker.java:202)
            at java.lang.Thread.run(Thread.java:595)

            "RMI TCP Accept-4444" daemon prio=1 tid=0x0000002b65f19c50 nid=0x6750 runnable [0x0000000040c8f000..0x0000000040c8fc30]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0x0000002adbc83318> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
            at java.lang.Thread.run(Thread.java:595)

            "NamingBootstrap Pool(2)-1" daemon prio=1 tid=0x0000002b65b2bd20 nid=0x674f runnable [0x0000000040c69000..0x0000000040c69db0]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0x0000002adbc859e8> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at org.jnp.server.Main$AcceptHandler.run(Main.java:417)
            at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:97)
            at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
            at java.lang.Thread.run(Thread.java:595)

            "GC Daemon" daemon prio=1 tid=0x0000002b65b032b0 nid=0x674e in Object.wait() [0x0000000040c43000..0x0000000040c43d30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002adbc7c428> (a sun.misc.GC$LatencyLock)
            at sun.misc.GC$Daemon.run(GC.java:100)
            - locked <0x0000002adbc7c428> (a sun.misc.GC$LatencyLock)

            "RMI Reaper" prio=1 tid=0x0000002b65b17a60 nid=0x674d in Object.wait() [0x0000000040c1d000..0x0000000040c1deb0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002adbc7a9e8> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
            - locked <0x0000002adbc7a9e8> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
            at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:336)
            at java.lang.Thread.run(Thread.java:595)

            "Timer-1" daemon prio=1 tid=0x0000002b65b17260 nid=0x674c in Object.wait() [0x0000000040bf7000..0x0000000040bf7e30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002adbc7b000> (a java.util.TaskQueue)
            at java.lang.Object.wait(Object.java:474)
            at java.util.TimerThread.mainLoop(Timer.java:483)
            - locked <0x0000002adbc7b000> (a java.util.TaskQueue)
            at java.util.TimerThread.run(Timer.java:462)

            "RMI TCP Accept-1098" daemon prio=1 tid=0x0000002b65b2c520 nid=0x674b runnable [0x0000000040bd1000..0x0000000040bd1bb0]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0x0000002adbc7aab0> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:334)
            at java.lang.Thread.run(Thread.java:595)

            "ClassLoadingPool(1)-1" daemon prio=1 tid=0x0000002b6591e790 nid=0x674a runnable [0x00000000400f0000..0x00000000400f0b30]
            at java.net.PlainSocketImpl.socketAccept(Native Method)
            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
            - locked <0x0000002adbc6d108> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at org.jboss.web.WebServer.run(WebServer.java:278)
            at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:97)
            at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:743)
            at java.lang.Thread.run(Thread.java:595)

            "ScannerThread" daemon prio=1 tid=0x0000002b65bff840 nid=0x6749 waiting on condition [0x00000000400ca000..0x00000000400cacb0]
            at java.lang.Thread.sleep(Native Method)
            at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:218)
            at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:191)

            "Timer-0" daemon prio=1 tid=0x0000002b63b370b0 nid=0x6748 in Object.wait() [0x00000000400a4000..0x00000000400a4c30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002adba2fcb0> (a java.util.TaskQueue)
            at java.util.TimerThread.mainLoop(Timer.java:509)
            - locked <0x0000002adba2fcb0> (a java.util.TaskQueue)
            at java.util.TimerThread.run(Timer.java:462)

            "DestroyJavaVM" prio=1 tid=0x0000000040116460 nid=0x6735 waiting on condition [0x0000000000000000..0x0000007fbfffce60]

            "Low Memory Detector" daemon prio=1 tid=0x0000002b5f816e70 nid=0x6743 runnable [0x0000000000000000..0x0000000000000000]

            "CompilerThread1" daemon prio=1 tid=0x0000002b5f8150d0 nid=0x6742 waiting on condition [0x0000000000000000..0x0000000040a83750]

            "CompilerThread0" daemon prio=1 tid=0x0000002b5f813cb0 nid=0x6741 waiting on condition [0x0000000000000000..0x0000000040982440]

            "AdapterThread" daemon prio=1 tid=0x0000002b5f812560 nid=0x6740 waiting on condition [0x0000000000000000..0x0000000000000000]

            "Signal Dispatcher" daemon prio=1 tid=0x0000002b5f811130 nid=0x673f waiting on condition [0x0000000000000000..0x0000000000000000]

            "Surrogate Locker Thread (CMS)" daemon prio=1 tid=0x0000002b5f80ff40 nid=0x673e waiting on condition [0x0000000000000000..0x0000000000000000]

            "Finalizer" daemon prio=1 tid=0x0000002b5f7fdd00 nid=0x673d in Object.wait() [0x0000000040781000..0x0000000040781db0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002adb855c80> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
            - locked <0x0000002adb855c80> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
            at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

            "Reference Handler" daemon prio=1 tid=0x0000002b5f7fb320 nid=0x673c in Object.wait() [0x000000004075b000..0x000000004075bd30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002adb830030> (a java.lang.ref.Reference$Lock)
            at java.lang.Object.wait(Object.java:474)
            at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
            - locked <0x0000002adb830030> (a java.lang.ref.Reference$Lock)

            "VM Thread" prio=1 tid=0x0000002b5f7f6f10 nid=0x673b runnable

            "Gang worker#0 (Parallel GC Threads)" prio=1 tid=0x0000000040127660 nid=0x6736 runnable

            "Gang worker#1 (Parallel GC Threads)" prio=1 tid=0x00000000401284e0 nid=0x6737 runnable

            "Gang worker#2 (Parallel GC Threads)" prio=1 tid=0x00000000401293a0 nid=0x6738 runnable

            "Gang worker#3 (Parallel GC Threads)" prio=1 tid=0x000000004012a220 nid=0x6739 runnable

            "Concurrent Mark-Sweep GC Thread#0" prio=1 tid=0x0000002b5ca4e670 nid=0x673a runnable

            "VM Periodic Task Thread" prio=1 tid=0x0000002b5f818aa0 nid=0x6744 waiting on condition

            THREAD DUMP OF SHUTDOWN PROCESS:

            CMD_START = cd /usr/master/jboss-3.2.6/bin; nohup /usr/master/jboss-3.2.6/bin/run.sh -c default
            /usr/java/jdk1.5.0_06
            Full thread dump Java HotSpot(TM) 64-Bit Server VM (1.5.0_06-b05 mixed mode):

            "GC Daemon" daemon prio=1 tid=0x0000002b60281420 nid=0x70aa in Object.wait() [0x000000004007e000..0x000000004007ed30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002adf577580> (a sun.misc.GC$LatencyLock)
            at sun.misc.GC$Daemon.run(GC.java:100)
            - locked <0x0000002adf577580> (a sun.misc.GC$LatencyLock)

            "RMI RenewClean-[72.32.155.244:1098]" daemon prio=1 tid=0x0000002b5fe2a320 nid=0x70a9 in Object.wait() [0x0000000040058000..0x0000000040058eb0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002adf577f70> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
            - locked <0x0000002adf577f70> (a java.lang.ref.ReferenceQueue$Lock)
            at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:500)
            at java.lang.Thread.run(Thread.java:595)

            "Low Memory Detector" daemon prio=1 tid=0x0000002b5fd54b60 nid=0x70a4 runnable [0x0000000000000000..0x0000000000000000]

            "CompilerThread1" daemon prio=1 tid=0x0000002b5fd52f40 nid=0x70a3 waiting on condition [0x0000000000000000..0x0000000040da8290]

            "CompilerThread0" daemon prio=1 tid=0x0000002b5fd519a0 nid=0x70a2 waiting on condition [0x0000000000000000..0x0000000040ca7410]

            "AdapterThread" daemon prio=1 tid=0x0000002b5fd50250 nid=0x70a1 waiting on condition [0x0000000000000000..0x0000000000000000]

            "Signal Dispatcher" daemon prio=1 tid=0x0000002b5ceff210 nid=0x70a0 waiting on condition [0x0000000000000000..0x0000000000000000]

            "Finalizer" daemon prio=1 tid=0x0000002b5cef3bb0 nid=0x709f in Object.wait() [0x0000000040aa6000..0x0000000040aa6d30]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002adf5a5ee8> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
            - locked <0x0000002adf5a5ee8> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
            at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

            "Reference Handler" daemon prio=1 tid=0x0000002b5cef32d0 nid=0x709e in Object.wait() [0x0000000040a80000..0x0000000040a80eb0]
            at java.lang.Object.wait(Native Method)
            - waiting on <0x0000002adf5703c0> (a java.lang.ref.Reference$Lock)
            at java.lang.Object.wait(Object.java:474)
            at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
            - locked <0x0000002adf5703c0> (a java.lang.ref.Reference$Lock)

            "main" prio=1 tid=0x0000000040115cd0 nid=0x7091 runnable [0x0000007fbfffb000..0x0000007fbfffd1a0]
            at java.net.PlainSocketImpl.socketConnect(Native Method)
            at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
            - locked <0x0000002adf5c2520> (a java.net.SocksSocketImpl)
            at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
            at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
            at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
            at java.net.Socket.connect(Socket.java:507)
            at java.net.Socket.connect(Socket.java:457)
            at java.net.Socket.(Socket.java:365)
            at java.net.Socket.(Socket.java:178)
            at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
            at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
            at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:569)
            at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:185)
            at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171)
            at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:306)
            at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown Source)
            at sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:328)
            at sun.rmi.transport.DGCClient$EndpointEntry.registerRefs(DGCClient.java:275)
            at sun.rmi.transport.DGCClient.registerRefs(DGCClient.java:112)
            at sun.rmi.transport.LiveRef.read(LiveRef.java:277)
            at sun.rmi.server.UnicastRef2.readExternal(UnicastRef2.java:54)
            at java.rmi.server.RemoteObject.readObject(RemoteObject.java:438)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:585)
            at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:919)
            at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1813)
            at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1713)
            at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1299)
            at java.io.ObjectInputStream.readObject(ObjectInputStream.java:339)
            at java.rmi.MarshalledObject.get(MarshalledObject.java:135)
            at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:196)
            at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1187)
            at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:516)
            at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:509)
            at javax.naming.InitialContext.lookup(InitialContext.java:351)
            at org.jboss.Shutdown.main(Shutdown.java:188)

            "VM Thread" prio=1 tid=0x0000002b5ceeeec0 nid=0x709d runnable

            "GC task thread#0 (ParallelGC)" prio=1 tid=0x00000000401330f0 nid=0x7095 runnable

            "GC task thread#1 (ParallelGC)" prio=1 tid=0x0000000040133f70 nid=0x7096 runnable

            "GC task thread#2 (ParallelGC)" prio=1 tid=0x0000000040134df0 nid=0x7097 runnable

            "GC task thread#3 (ParallelGC)" prio=1 tid=0x0000000040135c70 nid=0x7098 runnable

            "GC task thread#4 (ParallelGC)" prio=1 tid=0x0000000040136af0 nid=0x7099 runnable

            "GC task thread#5 (ParallelGC)" prio=1 tid=0x0000000040137970 nid=0x709a runnable

            "GC task thread#6 (ParallelGC)" prio=1 tid=0x00000000401387f0 nid=0x709b runnable

            "GC task thread#7 (ParallelGC)" prio=1 tid=0x0000000040139670 nid=0x709c runnable

            "VM Periodic Task Thread" prio=1 tid=0x0000002b5fd567a0 nid=0x70a5 waiting on condition

            • 3. Re: Shutting down JBOSS hangs..
              svenkataraman_celequest

              A few additional pointers.

              1. The machine has been configured for multiple VIPs.
              2. Setting java.rmi.server.hostname to localhost in the JBoss startup command line has now helped the shutdown command to be posted and the server starts to shutdown.
              2. However, the server takes a very long time to shutdown as seen from the following log..
              00:20:24,656 INFO [Server] JBoss SHUTDOWN: Undeploying all packages
              00:20:24,657 INFO [TomcatDeployer] undeploy, ctxPath=/web-console, warUrl=file:/usr/master/jboss-3.2.6/server/default/deploy/management/web-console.war/
              00:20:24,671 INFO [TomcatDeployer] undeploy, ctxPath=/jmx-console, warUrl=file:/usr/master/jboss-3.2.6/server/default/deploy/jmx-console.war/
              00:23:33,663 INFO [Http11Protocol] Pausing Coyote HTTP/1.1 on http-0.0.0.0-8080

              • 4. Re: Shutting down JBOSS hangs..
                svenkataraman_celequest

                One more update...

                There is a connection timed out exception during Coyote shutdown. Looks to be related to the multiple network cards situation. Any clues?

                00:20:24,656 INFO [Server] JBoss SHUTDOWN: Undeploying all packages
                00:20:24,657 INFO [TomcatDeployer] undeploy, ctxPath=/web-console, warUrl=file:/usr/master/jboss-3.2.6/server/default/deploy/management/web-console.war/
                00:20:24,671 INFO [TomcatDeployer] undeploy, ctxPath=/jmx-console, warUrl=file:/usr/master/jboss-3.2.6/server/default/deploy/jmx-console.war/
                00:23:33,663 INFO [Http11Protocol] Pausing Coyote HTTP/1.1 on http-0.0.0.0-8080
                00:26:42,641 ERROR [CoyoteConnector] Protocol handler pause failed
                java.net.ConnectException: Connection timed out
                at java.net.PlainSocketImpl.socketConnect(Native Method)
                at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
                at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
                at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
                at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
                at java.net.Socket.connect(Socket.java:507)
                at java.net.Socket.connect(Socket.java:457)
                at java.net.Socket.(Socket.java:365)
                at java.net.Socket.(Socket.java:207)
                at org.apache.jk.common.ChannelSocket.unLockSocket(ChannelSocket.java:460)
                at org.apache.jk.common.ChannelSocket.pause(ChannelSocket.java:272)
                at org.apache.jk.server.JkMain.pause(JkMain.java:657)
                at org.apache.jk.server.JkCoyoteHandler.pause(JkCoyoteHandler.java:202)
                at org.apache.coyote.tomcat5.CoyoteConnector.pause(CoyoteConnector.java:1444)
                at org.jboss.web.tomcat.tc5.StandardService.stop(StandardService.java:583)
                at org.apache.catalina.core.StandardServer.stop(StandardServer.java:2347)
                at org.apache.catalina.startup.Catalina.stop(Catalina.java:605)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:585)
                at org.apache.commons.modeler.BaseModelMBean.invoke(BaseModelMBean.java:503)
                at org.jboss.mx.server.RawDynamicInvoker.invoke(RawDynamicInvoker.java:149)
                at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
                at org.jboss.web.tomcat.tc5.Tomcat5.stopService(Tomcat5.java:398)
                at org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:319)
                at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:223)
                at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:585)
                at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
                at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
                at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
                at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
                at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:197)
                at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
                at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:884)
                at $Proxy17.stop(Unknown Source)
                at org.jboss.system.ServiceController.stop(ServiceController.java:496)
                at org.jboss.system.ServiceController.stop(ServiceController.java:487)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:585)
                at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
                at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
                at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
                at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
                at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:197)
                at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
                at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
                at $Proxy4.stop(Unknown Source)
                at org.jboss.deployment.SARDeployer.stop(SARDeployer.java:285)
                at org.jboss.deployment.MainDeployer.stop(MainDeployer.java:498)
                at org.jboss.deployment.MainDeployer.undeploy(MainDeployer.java:471)
                at org.jboss.deployment.MainDeployer.shutdown(MainDeployer.java:358)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:585)
                at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
                at org.jboss.mx.server.Invocation.dispatch(Invocation.java:62)
                at org.jboss.mx.server.Invocation.dispatch(Invocation.java:54)
                at org.jboss.mx.server.Invocation.invoke(Invocation.java:82)
                at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:197)
                at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
                at org.jboss.system.server.ServerImpl$ShutdownHook.shutdownDeployments(ServerImpl.java:903)
                at org.jboss.system.server.ServerImpl$ShutdownHook.shutdown(ServerImpl.java:878)
                at org.jboss.system.server.ServerImpl$ShutdownHook.run(ServerImpl.java:865)
                00:26:43,645 INFO [StandardService] Stopping service jboss.web
                00:26:43,646 INFO [StandardHost] Create Host deployer for direct deployment ( non-jmx )
                00:26:43,651 INFO [StandardHostDeployer] Removing web application at context path /jbossmq-httpil
                00:26:43,653 INFO [StandardHostDeployer] Removing web application at context path /invoker
                00:26:43,656 INFO [StandardHostDeployer] Removing web application at context path
                00:26:43,658 INFO [Http11Protocol] Stopping Coyote HTTP/1.1 on http-0.0.0.0-8080
                00:26:43,661 INFO [TomcatDeployer] undeploy, ctxPath=/invoker, warUrl=file:/usr/master/jboss-3.2.6/server/default/deploy/http-invoker.sar/invoker.war/
                00:26:43,662 INFO [TomcatDeployer] undeploy, ctxPath=/, warUrl=file:/usr/master/jboss-3.2.6/server/default/deploy/jbossweb-tomcat50.sar/ROOT.war/
                00:26:43,664 INFO [TomcatDeployer] undeploy, ctxPath=/jbossmq-httpil, warUrl=file:/usr/master/jboss-3.2.6/server/default/deploy/jms/jbossmq-httpil.sar/jbossmq-httpil.war/
                00:26:43,721 INFO [MailService] Mail service 'java:/Mail' removed from JNDI
                00:26:44,548 INFO [HypersonicDatabase] Database standalone closed clean
                Shutting down
                Shutdown complete
                Halting VM

                • 5. Re: Shutting down JBOSS hangs..
                  genman

                  I've had trouble on my machine with the firewall software. None of the internal RMI ports worked...



                  "main" prio=1 tid=0x0000000040115cd0 nid=0x7091 runnable [0x0000007fbfffb000..0x0000007fbfffd1a0]
                  at java.net.PlainSocketImpl.socketConnect(Native Method)
                  at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
                  - locked <0x0000002adf5c2520> (a java.net.SocksSocketImpl)
                  at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
                  at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
                  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
                  at java.net.Socket.connect(Socket.java:507)
                  at java.net.Socket.connect(Socket.java:457)
                  at java.net.Socket.(Socket.java:365)
                  at java.net.Socket.(Socket.java:178)
                  at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
                  at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
                  at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:569)
                  at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:185)
                  at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171)

                  is where it was stuck

                  • 6. Re: Shutting down JBOSS hangs..
                    svenkataraman_celequest

                    The issue was with the DNS resolution on the host. The problem went away when we fixed the DNS resolution.

                    - Shankar