11 Replies Latest reply on Jan 3, 2012 4:34 PM by pstackle

    JBoss AS 7.1CR1 hangs when deployments reference JMS objects

    Stephen Coy Master

      I was going to report this as a bug, but I read http://community.jboss.org/wiki/HelpBugReport first!

       

      If you build and deploy the attached application (required Maven 3.0.x), start the server with standalone-full.xml profile, and then try to terminate it with Ctrl-C, the server will hang:

      18:51:32,049 INFO  [org.jboss.ws.common.management.AbstractServerConfig] (MSC service thread 1-4) JBoss Web Services - Stack CXF Server 4.0.0.GA
      18:51:32,174 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-2) live server is starting with configuration HornetQ Configuration (clustered=false,backup=false,sharedStore=true,journalDirectory=/Users/steve/servers/jboss-as-7.1.0.CR1/standalone/data/messagingjournal,bindingsDirectory=/Users/steve/servers/jboss-as-7.1.0.CR1/standalone/data/messagingbindings,largeMessagesDirectory=/Users/steve/servers/jboss-as-7.1.0.CR1/standalone/data/messaginglargemessages,pagingDirectory=/Users/steve/servers/jboss-as-7.1.0.CR1/standalone/data/messagingpaging)
      18:51:32,343 WARNING [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-2) Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.
      18:51:32,366 INFO  [org.apache.tomcat.util.net.AprEndpoint] (MSC service thread 1-4) Failed to create poller with specified size of 32768
      18:51:32,367 WARN  [jacorb.codeset] (MSC service thread 1-5) Warning - unknown codeset (MacRoman) - defaulting to ISO-8859-1
      18:51:32,380 INFO  [org.apache.tomcat.util.net.AprEndpoint] (MSC service thread 1-4) Failed to create poller with specified size of 32768
      18:51:32,386 INFO  [org.apache.tomcat.util.net.AprEndpoint] (MSC service thread 1-4) Failed to create poller with specified size of 16384
      18:51:32,389 INFO  [org.apache.coyote.http11.Http11AprProtocol] (MSC service thread 1-4) Starting Coyote HTTP/1.1 on http-localhost-127.0.0.1-8080
      18:51:32,391 INFO  [org.jboss.as.jacorb] (MSC service thread 1-5) CORBA ORB Service Started
      18:51:32,569 INFO  [org.jboss.as.jacorb] (MSC service thread 1-3) CORBA Naming Service Started
      18:51:32,714 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-2) Started Netty Acceptor version 3.2.3.Final-r${buildNumber} localhost:5455 for CORE protocol
      18:51:32,716 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-2) Started Netty Acceptor version 3.2.3.Final-r${buildNumber} localhost:5445 for CORE protocol
      18:51:32,727 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-2) Server is now live
      18:51:32,728 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-2) HornetQ Server version 2.2.7.Final (HQ_2_2_7_FINAL_AS7, 121) [b50df0ad-3097-11e1-a495-9027e4faed49] started
      18:51:32,804 INFO  [org.jboss.as.messaging] (MSC service thread 1-2) JBAS011601: Bound messaging object to jndi name java:/ConnectionFactory
      18:51:32,808 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-4) trying to deploy queue jms.queue.testQueue
      18:51:32,851 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) JBAS010400: Bound data source [java:jboss/datasources/ExampleDS]
      18:51:32,862 INFO  [org.jboss.as.remoting] (MSC service thread 1-2) Listening on /127.0.0.1:9999
      18:51:32,901 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-5) JBAS015012: Started FileSystemDeploymentService for directory /Users/steve/servers/jboss-as-7.1.0.CR1/standalone/deployments
      18:51:32,920 INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-6) JBAS010406: Registered connection factory java:/JmsXA
      18:51:32,946 INFO  [org.hornetq.ra.HornetQResourceAdapter] (MSC service thread 1-6) HornetQ resource adaptor started
      18:51:32,946 INFO  [org.jboss.as.connector.services.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-6) IJ020002: Deployed: file://RaActivatorhornetq-ra
      18:51:32,948 INFO  [org.jboss.as.deployment.connector] (MSC service thread 1-1) JBAS010401: Bound JCA ConnectionFactory [java:/JmsXA]
      18:51:32,961 INFO  [org.jboss.as.messaging] (MSC service thread 1-4) JBAS011601: Bound messaging object to jndi name java:/queue/test
      18:51:32,963 INFO  [org.jboss.as.messaging] (MSC service thread 1-7) JBAS011601: Bound messaging object to jndi name java:/RemoteConnectionFactory
      18:51:32,964 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-8) trying to deploy queue jms.topic.testTopic
      18:51:32,995 INFO  [org.jboss.as.messaging] (MSC service thread 1-8) JBAS011601: Bound messaging object to jndi name java:/topic/test
      18:51:33,066 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-4) Starting deployment of "messaging-app-1.0-SNAPSHOT.ear"
      18:51:33,446 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-8) Starting deployment of "messaging-web-1.0-SNAPSHOT.war"
      18:51:33,446 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-6) Starting deployment of "messaging-ejb-1.0-SNAPSHOT.jar"
      18:51:33,909 INFO  [org.jboss.weld] (MSC service thread 1-7) Processing CDI deployment: messaging-app-1.0-SNAPSHOT.ear
      18:51:33,995 INFO  [org.jboss.weld] (MSC service thread 1-3) Processing CDI deployment: messaging-ejb-1.0-SNAPSHOT.jar
      18:51:33,998 INFO  [org.jboss.as.ejb3.deployment.processors.EjbJndiBindingsDeploymentUnitProcessor] (MSC service thread 1-3) JNDI bindings for session bean named Messenger in deployment unit subdeployment "messaging-ejb-1.0-SNAPSHOT.jar" of deployment "messaging-app-1.0-SNAPSHOT.ear" are as follows:
      
           java:global/messaging-app-1.0-SNAPSHOT/messaging-ejb-1.0-SNAPSHOT/Messenger!org.jboss.as7.demo.ejb.Messenger
           java:app/messaging-ejb-1.0-SNAPSHOT/Messenger!org.jboss.as7.demo.ejb.Messenger
           java:module/Messenger!org.jboss.as7.demo.ejb.Messenger
           java:global/messaging-app-1.0-SNAPSHOT/messaging-ejb-1.0-SNAPSHOT/Messenger
           java:app/messaging-ejb-1.0-SNAPSHOT/Messenger
           java:module/Messenger
      
      18:51:34,020 INFO  [org.jboss.weld] (MSC service thread 1-4) Processing CDI deployment: messaging-web-1.0-SNAPSHOT.war
      18:51:34,038 INFO  [org.jboss.weld] (MSC service thread 1-1) Starting Services for CDI deployment: messaging-app-1.0-SNAPSHOT.ear
      18:51:34,071 INFO  [org.jboss.weld.Version] (MSC service thread 1-1) WELD-000900 1.1.4 (Final)
      18:51:34,097 INFO  [org.jboss.weld] (MSC service thread 1-2) Starting weld service
      18:51:34,430 INFO  [org.jboss.solder.config.xml.bootstrap.XmlConfigExtension] (MSC service thread 1-2) Solder Config XML provider starting...
      18:51:34,431 INFO  [org.jboss.solder.config.xml.bootstrap.XmlConfigExtension] (MSC service thread 1-2) Loading XmlDocumentProvider: org.jboss.solder.config.xml.bootstrap.ResourceLoaderXmlDocumentProvider
      18:51:34,438 INFO  [org.jboss.solder.config.xml.bootstrap.XmlConfigExtension] (MSC service thread 1-2) Reading XML file: vfs:/content/messaging-app-1.0-SNAPSHOT.ear/messaging-ejb-1.0-SNAPSHOT.jar/META-INF/beans.xml
      18:51:34,444 INFO  [org.jboss.solder.Version] (MSC service thread 1-2) Solder null (build id: null)
      18:51:36,120 INFO  [javax.enterprise.resource.webcontainer.jsf.config] (MSC service thread 1-4) Initializing Mojarra 2.1.5 (SNAPSHOT 20111202) for context '/messaging-demo'
      18:51:36,924 INFO  [org.hibernate.validator.util.Version] (MSC service thread 1-4) Hibernate Validator 4.2.0.Final
      18:51:37,091 INFO  [org.jboss.web] (MSC service thread 1-4) registering web context: /messaging-demo
      18:51:37,092 INFO  [org.jboss.as] (MSC service thread 1-7) JBoss AS 7.1.0.CR1 "Flux Capacitor" started in 6963ms - Started 297 of 373 services (74 services are passive or on-demand)
      18:51:37,166 INFO  [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS018559: Deployed "messaging-app-1.0-SNAPSHOT.ear"
      ^C18:51:43,518 INFO  [org.jboss.as.messaging] (MSC service thread 1-4) JBAS011605: Unbound messaging object to jndi name java:/RemoteConnectionFactory
      18:51:43,519 INFO  [org.jboss.as.osgi] (MSC service thread 1-8) JBAS011921: Stopping OSGi Framework
      18:51:43,521 INFO  [org.jboss.as.messaging] (MSC service thread 1-6) JBAS011605: Unbound messaging object to jndi name java:/ConnectionFactory
      18:51:43,554 INFO  [org.jboss.as.logging] JBAS011503: Restored bootstrap log handlers
      18:51:43,560 INFO  [jacorb.poa] POA IRPOA destroyed
      18:51:43,570 INFO  [jacorb.poa] POA Naming destroyed
      18:51:43,576 INFO  [org.apache.coyote.http11.Http11AprProtocol] Pausing Coyote HTTP/1.1 on http-localhost-127.0.0.1-8080
      18:51:43,577 INFO  [org.apache.coyote.http11.Http11AprProtocol] Stopping Coyote HTTP/1.1 on http-localhost-127.0.0.1-8080
      18:51:43,579 ERROR [org.hornetq.core.server.impl.ServerSessionImpl] Cannot find consumer with id 0
      18:51:43,584 INFO  [jacorb.poa] POA RootPOA destroyed
      18:51:43,586 INFO  [jacorb.orb] prepare ORB for shutdown...
      18:51:43,586 INFO  [jacorb.orb] ORB going down...
      18:51:43,600 INFO  [jacorb.orb] ORB shutdown complete
      18:51:43,601 INFO  [jacorb.orb] ORB run, exit
      18:51:43,600 INFO  [jacorb.orb.iiop] Listener exited
      18:51:43,606 ERROR [org.hornetq.core.server.impl.ServerSessionImpl] Cannot find consumer with id 0
      18:51:43,604 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
      18:51:43,616 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
      18:51:43,616 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
      18:51:43,631 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
      18:51:43,637 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
      18:51:43,638 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
      18:51:43,643 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
      18:51:43,646 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
      18:51:43,649 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
      18:51:43,649 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
      18:51:43,652 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
      18:51:43,656 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
      18:51:43,658 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
      18:51:43,658 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
      18:51:43,660 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
      18:51:43,706 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] HornetQ Server version 2.2.7.Final (HQ_2_2_7_FINAL_AS7, 121) [b50df0ad-3097-11e1-a495-9027e4faed49] stopped
      
        • 1. Re: JBoss AS 7.1CR1 hangs when deployments reference JMS objects
          jaikiran pai Master

          Let me take a look. Which Java version (runtime) are you using by the way?

           

          P.S: There were a couple of bugs we fixed just immediately after CR1 was released. So we released a CR1b release the next day http://lists.jboss.org/pipermail/jboss-as7-dev/2011-December/004949.html. You might want to start using that http://www.jboss.org/jbossas/downloads.html. It most probably isn't related to the issue you are running into, but better start using that newer version.

          • 2. Re: JBoss AS 7.1CR1 hangs when deployments reference JMS objects
            Stephen Coy Master

            [steve@steves-mbp jboss]$ java -version

            java version "1.6.0_29"

            Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-11M3527)

            Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)

            • 3. Re: JBoss AS 7.1CR1 hangs when deployments reference JMS objects
              jaikiran pai Master

              Although I do see the WARN message about the connection failure, the server shuts down fine for me with that application deployed:

               

              13:46:26,290 INFO  [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS018559: Deployed "messaging-app-1.0-SNAPSHOT.ear"
              ^C13:46:40,971 INFO  [org.jboss.as.messaging] (MSC service thread 1-2) JBAS011605: Unbound messaging object to jndi name java:/RemoteConnectionFactory
              13:46:40,972 INFO  [org.jboss.as.osgi] (MSC service thread 1-2) JBAS011921: Stopping OSGi Framework
              13:46:40,980 INFO  [org.jboss.as.messaging] (MSC service thread 1-3) JBAS011605: Unbound messaging object to jndi name java:/ConnectionFactory
              13:46:41,082 INFO  [org.jboss.as.logging] JBAS011503: Restored bootstrap log handlers
              13:46:41,093 ERROR [org.hornetq.core.server.impl.ServerSessionImpl] Cannot find consumer with id 0
              13:46:41,095 ERROR [org.hornetq.core.server.impl.ServerSessionImpl] Cannot find consumer with id 0
              13:46:41,096 ERROR [org.hornetq.core.server.impl.ServerSessionImpl] Cannot find consumer with id 0
              13:46:41,097 ERROR [org.hornetq.core.server.impl.ServerSessionImpl] Cannot find consumer with id 0
              13:46:41,098 ERROR [org.hornetq.core.server.impl.ServerSessionImpl] Cannot find consumer with id 0
              13:46:41,099 ERROR [org.hornetq.core.server.impl.ServerSessionImpl] Cannot find consumer with id 0
              13:46:41,100 ERROR [org.hornetq.core.server.impl.ServerSessionImpl] Cannot find consumer with id 0
              13:46:41,101 ERROR [org.hornetq.core.server.impl.ServerSessionImpl] Cannot find consumer with id 0
              13:46:41,102 ERROR [org.hornetq.core.server.impl.ServerSessionImpl] Cannot find consumer with id 0
              13:46:41,145 INFO  [jacorb.poa] POA IRPOA destroyed
              13:46:41,148 INFO  [jacorb.poa] POA Naming destroyed
              13:46:41,156 INFO  [org.jboss.as.deployment.connector] JBAS010410: Unbound JCA ConnectionFactory [java:/JmsXA]
              13:46:41,164 INFO  [org.jboss.weld] Stopping weld service
              13:46:41,166 INFO  [jacorb.poa] POA RootPOA destroyed
              13:46:41,166 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
              13:46:41,171 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
              13:46:41,176 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
              13:46:41,191 INFO  [jacorb.orb] prepare ORB for shutdown...
              13:46:41,192 INFO  [jacorb.orb] ORB going down...
              13:46:41,236 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
              13:46:41,217 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
              13:46:41,262 INFO  [jacorb.orb] ORB shutdown complete
              13:46:41,263 INFO  [jacorb.orb.iiop] Listener exited
              13:46:41,262 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
              13:46:41,262 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
              13:46:41,286 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
              13:46:41,285 INFO  [com.arjuna.ats.jbossatx] ARJUNA032018: Destroying TransactionManagerService
              13:46:41,294 INFO  [com.arjuna.ats.jbossatx] ARJUNA032014: Stopping transaction recovery manager
              13:46:41,285 INFO  [jacorb.orb] ORB run, exit
              13:46:41,293 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
              13:46:41,313 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
              13:46:41,314 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
              13:46:41,322 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
              13:46:41,323 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
              13:46:41,347 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
              13:46:41,359 WARN  [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] Connection failure has been detected: The connection was disconnected because of server shutdown [code=4]
              13:46:41,374 INFO  [org.jboss.as.server.deployment] Stopped deployment messaging-ejb-1.0-SNAPSHOT.jar in 416ms
              13:46:41,383 INFO  [org.jboss.as.server.deployment] Stopped deployment messaging-web-1.0-SNAPSHOT.war in 423ms
              13:46:41,386 INFO  [org.jboss.as.server.deployment] Stopped deployment messaging-app-1.0-SNAPSHOT.ear in 391ms
              13:46:41,387 INFO  [org.hornetq.core.server.impl.HornetQServerImpl] HornetQ Server version 2.2.7.Final (HQ_2_2_7_FINAL_AS7, 121) [1637706a-2f95-11e1-a251-005056c00008] stopped
              13:46:41,400 INFO  [org.jboss.as] JBoss AS 7.1.0.CR1b "Flux Capacitor" stopped in 398ms
              
              • 5. Re: JBoss AS 7.1CR1 hangs when deployments reference JMS objects
                jaikiran pai Master

                If it still hangs on CR1b, then please take a few thread dumps when it hangs.

                • 6. Re: JBoss AS 7.1CR1 hangs when deployments reference JMS objects
                  Stephen Coy Master

                  A clean 7.1CR1b is still doing it:

                   
                  2011-12-29 20:58:09
                  Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.4-b02-402 mixed mode):
                  
                  "Thread-17 (group:HornetQ-client-global-threads-310083381)" daemon prio=5 tid=7fc5f1d39800 nid=0x117283000 waiting on condition [117282000]
                     java.lang.Thread.State: TIMED_WAITING (parking)
                       at sun.misc.Unsafe.park(Native Method)
                       - parking to wait for  <7d4588510> (a java.util.concurrent.SynchronousQueue$TransferStack)
                       at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
                       at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:424)
                       at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323)
                       at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
                       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Thread-16 (group:HornetQ-client-global-threads-310083381)" daemon prio=5 tid=7fc5f22ec800 nid=0x117180000 in Object.wait() [11717f000]
                     java.lang.Thread.State: TIMED_WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d5b20698> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:916)
                       - locked <7d5b20698> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:840)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:588)
                       - locked <7d5b20650> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:482)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$800(ClientSessionFactoryImpl.java:78)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1318)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1262)
                       at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Thread-15 (group:HornetQ-client-global-threads-310083381)" daemon prio=5 tid=7fc5eca86800 nid=0x116ac2000 in Object.wait() [116ac1000]
                     java.lang.Thread.State: TIMED_WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d5b1e058> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:916)
                       - locked <7d5b1e058> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:840)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:588)
                       - locked <7d5b1e010> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:482)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$800(ClientSessionFactoryImpl.java:78)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1318)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1262)
                       at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Thread-14 (group:HornetQ-client-global-threads-310083381)" daemon prio=5 tid=7fc5f082b800 nid=0x116856000 in Object.wait() [116855000]
                     java.lang.Thread.State: TIMED_WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d5b21358> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:916)
                       - locked <7d5b21358> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:840)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:588)
                       - locked <7d5b21310> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:482)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$800(ClientSessionFactoryImpl.java:78)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1318)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1262)
                       at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Thread-13 (group:HornetQ-client-global-threads-310083381)" daemon prio=5 tid=7fc5ecb96000 nid=0x116753000 in Object.wait() [116752000]
                     java.lang.Thread.State: TIMED_WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d5b28e20> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:916)
                       - locked <7d5b28e20> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:840)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:588)
                       - locked <7d5b28dd8> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:482)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$800(ClientSessionFactoryImpl.java:78)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1318)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1262)
                       at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Thread-11 (group:HornetQ-client-global-threads-310083381)" daemon prio=5 tid=7fc5eca31800 nid=0x116427000 in Object.wait() [116426000]
                     java.lang.Thread.State: TIMED_WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d5b2c120> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:916)
                       - locked <7d5b2c120> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:840)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:588)
                       - locked <7d5b2c0d8> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:482)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$800(ClientSessionFactoryImpl.java:78)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1318)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1262)
                       at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Thread-10 (group:HornetQ-client-global-threads-310083381)" daemon prio=5 tid=7fc5ec94c000 nid=0x116324000 in Object.wait() [116323000]
                     java.lang.Thread.State: TIMED_WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d5b1ed18> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:916)
                       - locked <7d5b1ed18> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:840)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:588)
                       - locked <7d5b1ecd0> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:482)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$800(ClientSessionFactoryImpl.java:78)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1318)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1262)
                       at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Thread-9 (group:HornetQ-client-global-threads-310083381)" daemon prio=5 tid=7fc5f22eb800 nid=0x116221000 in Object.wait() [116220000]
                     java.lang.Thread.State: TIMED_WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d5b29ae0> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:916)
                       - locked <7d5b29ae0> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:840)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:588)
                       - locked <7d5b29a98> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:482)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$800(ClientSessionFactoryImpl.java:78)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1318)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1262)
                       at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Thread-8 (group:HornetQ-client-global-threads-310083381)" daemon prio=5 tid=7fc5f22eb000 nid=0x115ffc000 in Object.wait() [115ffb000]
                     java.lang.Thread.State: TIMED_WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d5b1d398> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:916)
                       - locked <7d5b1d398> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:840)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:588)
                       - locked <7d5b1d350> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:482)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$800(ClientSessionFactoryImpl.java:78)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1318)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1262)
                       at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Thread-7 (group:HornetQ-client-global-threads-310083381)" daemon prio=5 tid=7fc5f22ea000 nid=0x115ef9000 in Object.wait() [115ef8000]
                     java.lang.Thread.State: TIMED_WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d5b14a80> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:916)
                       - locked <7d5b14a80> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:840)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:588)
                       - locked <7d5b14a20> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:482)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$800(ClientSessionFactoryImpl.java:78)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1318)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1262)
                       at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Thread-6 (group:HornetQ-client-global-threads-310083381)" daemon prio=5 tid=7fc5f22e9800 nid=0x115df6000 in Object.wait() [115df5000]
                     java.lang.Thread.State: TIMED_WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d5b1f9d8> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:916)
                       - locked <7d5b1f9d8> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:840)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:588)
                       - locked <7d5b1f990> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:482)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$800(ClientSessionFactoryImpl.java:78)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1318)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1262)
                       at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Thread-5 (group:HornetQ-client-global-threads-310083381)" daemon prio=5 tid=7fc5f22e8800 nid=0x115c27000 in Object.wait() [115c26000]
                     java.lang.Thread.State: TIMED_WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d5b274a0> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:916)
                       - locked <7d5b274a0> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:840)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:588)
                       - locked <7d5b27458> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:482)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$800(ClientSessionFactoryImpl.java:78)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1318)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1262)
                       at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Thread-4 (group:HornetQ-client-global-threads-310083381)" daemon prio=5 tid=7fc5eca2a800 nid=0x115b24000 in Object.wait() [115b23000]
                     java.lang.Thread.State: TIMED_WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d5af6070> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:916)
                       - locked <7d5af6070> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:840)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:588)
                       - locked <7d5af6028> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:482)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$800(ClientSessionFactoryImpl.java:78)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1318)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1262)
                       at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Thread-3 (group:HornetQ-client-global-threads-310083381)" daemon prio=5 tid=7fc5f22b6800 nid=0x11591b000 in Object.wait() [11591a000]
                     java.lang.Thread.State: TIMED_WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d5b2a7a0> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:916)
                       - locked <7d5b2a7a0> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:840)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:588)
                       - locked <7d5b2a758> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:482)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$800(ClientSessionFactoryImpl.java:78)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1318)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1262)
                       at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Thread-2 (group:HornetQ-client-global-threads-310083381)" daemon prio=5 tid=7fc5f2229000 nid=0x114f13000 in Object.wait() [114f12000]
                     java.lang.Thread.State: TIMED_WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d5b28160> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:916)
                       - locked <7d5b28160> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:840)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:588)
                       - locked <7d5b28118> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:482)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$800(ClientSessionFactoryImpl.java:78)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1318)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1262)
                       at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "MSC Shutdown Thread" prio=5 tid=7fc5f1d35000 nid=0x115818000 waiting on condition [115817000]
                     java.lang.Thread.State: WAITING (parking)
                       at sun.misc.Unsafe.park(Native Method)
                       - parking to wait for  <7d05dba10> (a java.util.concurrent.CountDownLatch$Sync)
                       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
                       at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
                       at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
                       at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
                       at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:207)
                       at org.jboss.msc.service.ServiceContainerImpl$ShutdownHookHolder$1$1.run(ServiceContainerImpl.java:129)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "SIGINT handler" daemon prio=9 tid=7fc5f0b23800 nid=0x115715000 in Object.wait() [115714000]
                     java.lang.Thread.State: WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d41e5d98> (a java.lang.Thread)
                       at java.lang.Thread.join(Thread.java:1210)
                       - locked <7d41e5d98> (a java.lang.Thread)
                       at java.lang.Thread.join(Thread.java:1263)
                       at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
                       at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
                       at java.lang.Shutdown.runHooks(Shutdown.java:79)
                       at java.lang.Shutdown.sequence(Shutdown.java:123)
                       at java.lang.Shutdown.exit(Shutdown.java:168)
                       - locked <7f019d288> (a java.lang.Class for java.lang.Shutdown)
                       at java.lang.Terminator$1.handle(Terminator.java:35)
                       at sun.misc.Signal$1.run(Signal.java:195)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Thread-1 (group:HornetQ-client-global-threads-310083381)" daemon prio=5 tid=7fc5f0801800 nid=0x11ccdb000 in Object.wait() [11ccda000]
                     java.lang.Thread.State: TIMED_WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d5b2b460> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:916)
                       - locked <7d5b2b460> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:840)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:588)
                       - locked <7d5b2b418> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:482)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$800(ClientSessionFactoryImpl.java:78)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1318)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528)
                       at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298)
                       at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1262)
                       at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Thread-4 (group:HornetQ-client-global-scheduled-threads-164700615)" daemon prio=5 tid=7fc5f11c6000 nid=0x11cbd8000 waiting on condition [11cbd7000]
                     java.lang.Thread.State: WAITING (parking)
                       at sun.misc.Unsafe.park(Native Method)
                       - parking to wait for  <7d5b147f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
                       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
                       at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
                       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Thread-3 (group:HornetQ-client-global-scheduled-threads-164700615)" daemon prio=5 tid=7fc5f11c5000 nid=0x11c6c9000 waiting on condition [11c6c8000]
                     java.lang.Thread.State: WAITING (parking)
                       at sun.misc.Unsafe.park(Native Method)
                       - parking to wait for  <7d5b147f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
                       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
                       at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
                       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Thread-2 (group:HornetQ-client-global-scheduled-threads-164700615)" daemon prio=5 tid=7fc5f093f800 nid=0x11c4c3000 waiting on condition [11c4c2000]
                     java.lang.Thread.State: WAITING (parking)
                       at sun.misc.Unsafe.park(Native Method)
                       - parking to wait for  <7d5b147f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
                       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
                       at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
                       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Thread-1 (group:HornetQ-client-global-scheduled-threads-164700615)" daemon prio=5 tid=7fc5f13f6000 nid=0x11bfb4000 waiting on condition [11bfb3000]
                     java.lang.Thread.State: WAITING (parking)
                       at sun.misc.Unsafe.park(Native Method)
                       - parking to wait for  <7d5b147f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
                       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
                       at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
                       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Thread-0 (group:HornetQ-client-global-scheduled-threads-164700615)" daemon prio=5 tid=7fc5ee43f000 nid=0x11a5c6000 waiting on condition [11a5c5000]
                     java.lang.Thread.State: WAITING (parking)
                       at sun.misc.Unsafe.park(Native Method)
                       - parking to wait for  <7d5b147f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
                       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
                       at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
                       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
                       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Transaction Reaper Worker 0" daemon prio=5 tid=7fc5ee178800 nid=0x11b390000 in Object.wait() [11b38f000]
                     java.lang.Thread.State: WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d509b2c0> (a java.util.LinkedList)
                       at java.lang.Object.wait(Object.java:485)
                       at com.arjuna.ats.arjuna.coordinator.TransactionReaper.waitForCancellations(TransactionReaper.java:318)
                       - locked <7d509b2c0> (a java.util.LinkedList)
                       at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:66)
                  
                  "Transaction Reaper" daemon prio=5 tid=7fc5ee177800 nid=0x11b28d000 in Object.wait() [11b28c000]
                     java.lang.Thread.State: TIMED_WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d4eab738> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
                       at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:91)
                       - locked <7d4eab738> (a com.arjuna.ats.arjuna.coordinator.TransactionReaper)
                  
                  "AOMRemoval" daemon prio=5 tid=7fc5eccac800 nid=0x11a97f000 waiting on condition [11a97e000]
                     java.lang.Thread.State: WAITING (parking)
                       at sun.misc.Unsafe.park(Native Method)
                       - parking to wait for  <7d509b3b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
                       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
                       at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
                       at org.jacorb.poa.AOM$1.run(AOM.java:120)
                  
                  "AOMRemoval" daemon prio=5 tid=7fc5ee1f1800 nid=0x119cba000 waiting on condition [119cb9000]
                     java.lang.Thread.State: WAITING (parking)
                       at sun.misc.Unsafe.park(Native Method)
                       - parking to wait for  <7d5093b30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
                       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
                       at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
                       at org.jacorb.poa.AOM$1.run(AOM.java:120)
                  
                  "AOMRemoval" daemon prio=5 tid=7fc5ecb11800 nid=0x11ad7e000 waiting on condition [11ad7d000]
                     java.lang.Thread.State: WAITING (parking)
                       at sun.misc.Unsafe.park(Native Method)
                       - parking to wait for  <7d5093b00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
                       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
                       at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
                       at org.jacorb.poa.AOM$1.run(AOM.java:120)
                  
                  "Periodic Recovery" prio=5 tid=7fc5ee26f800 nid=0x11ab75000 in Object.wait() [11ab74000]
                     java.lang.Thread.State: TIMED_WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d509b410> (a java.lang.Object)
                       at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doPeriodicWait(PeriodicRecovery.java:672)
                       at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:392)
                       - locked <7d509b410> (a java.lang.Object)
                  
                  "Transaction Expired Entry Monitor" daemon prio=5 tid=7fc5f0226000 nid=0x11a779000 in Object.wait() [11a778000]
                     java.lang.Thread.State: TIMED_WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d4f7b240> (a com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor)
                       at com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor.run(ExpiredEntryMonitor.java:191)
                       - locked <7d4f7b240> (a com.arjuna.ats.internal.arjuna.recovery.ExpiredEntryMonitor)
                  
                  "http-localhost-127.0.0.1-8080-Acceptor" daemon prio=5 tid=7fc5f0014800 nid=0x119bad000 waiting on condition [119bac000]
                     java.lang.Thread.State: TIMED_WAITING (sleeping)
                       at java.lang.Thread.sleep(Native Method)
                       at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1065)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "http-localhost-127.0.0.1-8080-Sendfile" daemon prio=5 tid=7fc5f0225000 nid=0x119aa8000 in Object.wait() [119aa7000]
                     java.lang.Thread.State: WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d4a50960> (a org.apache.tomcat.util.net.AprEndpoint$Sendfile)
                       at java.lang.Object.wait(Object.java:485)
                       at org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:2258)
                       - locked <7d4a50960> (a org.apache.tomcat.util.net.AprEndpoint$Sendfile)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "http-localhost-127.0.0.1-8080-EventPoller" daemon prio=5 tid=7fc5f009b000 nid=0x1199a3000 in Object.wait() [1199a2000]
                     java.lang.Thread.State: TIMED_WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d4ecdb50> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
                       at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1678)
                       - locked <7d4ecdb50> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Timer-0" prio=5 tid=7fc5f0927000 nid=0x1195df000 in Object.wait() [1195de000]
                     java.lang.Thread.State: WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d4a38cb0> (a java.util.TaskQueue)
                       at java.lang.Object.wait(Object.java:485)
                       at java.util.TimerThread.mainLoop(Timer.java:483)
                       - locked <7d4a38cb0> (a java.util.TaskQueue)
                       at java.util.TimerThread.run(Timer.java:462)
                  
                  "DestroyJavaVM" prio=5 tid=7fc5f00df000 nid=0x10ba5e000 waiting on condition [00000000]
                     java.lang.Thread.State: RUNNABLE
                  
                  "MSC service thread 1-8" prio=5 tid=7fc5ee0d5800 nid=0x114ddf000 waiting on condition [114dde000]
                     java.lang.Thread.State: TIMED_WAITING (parking)
                       at sun.misc.Unsafe.park(Native Method)
                       - parking to wait for  <7d5b2bf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                       at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
                       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2116)
                       at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:256)
                       - locked <7d5b2bf58> (a java.lang.Object)
                       at org.hornetq.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:857)
                       at org.hornetq.core.client.impl.DelegatingSession.close(DelegatingSession.java:152)
                       at org.hornetq.ra.inflow.HornetQMessageHandler.teardown(HornetQMessageHandler.java:242)
                       at org.hornetq.ra.inflow.HornetQActivation.teardown(HornetQActivation.java:319)
                       - locked <7d5aae0a0> (a org.hornetq.ra.inflow.HornetQActivation)
                       at org.hornetq.ra.inflow.HornetQActivation.stop(HornetQActivation.java:269)
                       at org.hornetq.ra.HornetQResourceAdapter.endpointDeactivation(HornetQResourceAdapter.java:175)
                       at org.jboss.as.ejb3.component.messagedriven.MessageDrivenComponent.stop(MessageDrivenComponent.java:167)
                       at org.jboss.as.ee.component.ComponentStartService.stop(ComponentStartService.java:49)
                       at org.jboss.msc.service.ServiceControllerImpl$StopTask.stopService(ServiceControllerImpl.java:1909)
                       at org.jboss.msc.service.ServiceControllerImpl$StopTask.run(ServiceControllerImpl.java:1872)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "MSC service thread 1-7" prio=5 tid=7fc5f0863800 nid=0x114cdc000 waiting on condition [114cdb000]
                     java.lang.Thread.State: WAITING (parking)
                       at sun.misc.Unsafe.park(Native Method)
                       - parking to wait for  <7d40292e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
                       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
                       at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
                       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "MSC service thread 1-6" prio=5 tid=7fc5ecbba000 nid=0x114bd9000 waiting on condition [114bd8000]
                     java.lang.Thread.State: WAITING (parking)
                       at sun.misc.Unsafe.park(Native Method)
                       - parking to wait for  <7d40292e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
                       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
                       at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
                       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "MSC service thread 1-5" prio=5 tid=7fc5ecbb9800 nid=0x114ad6000 waiting on condition [114ad5000]
                     java.lang.Thread.State: WAITING (parking)
                       at sun.misc.Unsafe.park(Native Method)
                       - parking to wait for  <7d40292e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
                       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
                       at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
                       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Poller SunPKCS11-Darwin" daemon prio=1 tid=7fc5ecbb8800 nid=0x1149d3000 waiting on condition [1149d2000]
                     java.lang.Thread.State: TIMED_WAITING (sleeping)
                       at java.lang.Thread.sleep(Native Method)
                       at sun.security.pkcs11.SunPKCS11$TokenPoller.run(SunPKCS11.java:692)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "MSC service thread 1-4" prio=5 tid=7fc5ecbb4000 nid=0x1148c3000 waiting on condition [1148c2000]
                     java.lang.Thread.State: WAITING (parking)
                       at sun.misc.Unsafe.park(Native Method)
                       - parking to wait for  <7d40292e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
                       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
                       at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
                       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "MSC service thread 1-3" prio=5 tid=7fc5ee0d4800 nid=0x1147c0000 waiting on condition [1147bf000]
                     java.lang.Thread.State: WAITING (parking)
                       at sun.misc.Unsafe.park(Native Method)
                       - parking to wait for  <7d40292e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
                       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
                       at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
                       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "MSC service thread 1-2" prio=5 tid=7fc5f08d7800 nid=0x1146bd000 waiting on condition [1146bc000]
                     java.lang.Thread.State: WAITING (parking)
                       at sun.misc.Unsafe.park(Native Method)
                       - parking to wait for  <7d40292e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
                       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
                       at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
                       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "MSC service thread 1-1" prio=5 tid=7fc5ee228000 nid=0x1145ba000 waiting on condition [1145b9000]
                     java.lang.Thread.State: WAITING (parking)
                       at sun.misc.Unsafe.park(Native Method)
                       - parking to wait for  <7d40292e8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
                       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
                       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
                       at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
                       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
                       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                       at java.lang.Thread.run(Thread.java:680)
                  
                  "Reference Reaper" daemon prio=5 tid=7fc5ee098800 nid=0x11403e000 in Object.wait() [11403d000]
                     java.lang.Thread.State: WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d4040d00> (a java.lang.ref.ReferenceQueue$Lock)
                       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
                       - locked <7d4040d00> (a java.lang.ref.ReferenceQueue$Lock)
                       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
                       at org.jboss.modules.ref.References$ReaperThread.run(References.java:68)
                  
                  "Low Memory Detector" daemon prio=5 tid=7fc5ee007000 nid=0x113be7000 runnable [00000000]
                     java.lang.Thread.State: RUNNABLE
                  
                  "C2 CompilerThread1" daemon prio=9 tid=7fc5ee006000 nid=0x113ae4000 waiting on condition [00000000]
                     java.lang.Thread.State: RUNNABLE
                  
                  "C2 CompilerThread0" daemon prio=9 tid=7fc5ee005800 nid=0x1139e1000 waiting on condition [00000000]
                     java.lang.Thread.State: RUNNABLE
                  
                  "Signal Dispatcher" daemon prio=9 tid=7fc5ee004800 nid=0x1138de000 waiting on condition [00000000]
                     java.lang.Thread.State: RUNNABLE
                  
                  "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7fc5ee003800 nid=0x1137db000 waiting on condition [00000000]
                     java.lang.Thread.State: RUNNABLE
                  
                  "Finalizer" daemon prio=8 tid=7fc5ee001000 nid=0x113517000 in Object.wait() [113516000]
                     java.lang.Thread.State: WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d40886e8> (a java.lang.ref.ReferenceQueue$Lock)
                       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
                       - locked <7d40886e8> (a java.lang.ref.ReferenceQueue$Lock)
                       at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
                       at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
                  
                  "Reference Handler" daemon prio=10 tid=7fc5ee000000 nid=0x113414000 in Object.wait() [113413000]
                     java.lang.Thread.State: WAITING (on object monitor)
                       at java.lang.Object.wait(Native Method)
                       - waiting on <7d4021ae0> (a java.lang.ref.Reference$Lock)
                       at java.lang.Object.wait(Object.java:485)
                       at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
                       - locked <7d4021ae0> (a java.lang.ref.Reference$Lock)
                  
                  "VM Thread" prio=9 tid=7fc5ec8eb800 nid=0x113311000 runnable 
                  
                  "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7fc5ec802800 nid=0x10ed99000 runnable 
                  
                  "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7fc5ec803800 nid=0x10ee9c000 runnable 
                  
                  "Gang worker#2 (Parallel GC Threads)" prio=9 tid=7fc5ec804000 nid=0x10ef9f000 runnable 
                  
                  "Gang worker#3 (Parallel GC Threads)" prio=9 tid=7fc5ec804800 nid=0x10f0a2000 runnable 
                  
                  "Concurrent Mark-Sweep GC Thread" prio=9 tid=7fc5ec87f800 nid=0x112fb7000 runnable 
                  "VM Periodic Task Thread" prio=10 tid=7fc5f0800000 nid=0x113cea000 waiting on condition 
                  
                  "Exception Catcher Thread" prio=10 tid=7fc5ec802000 nid=0x10bbc0000 runnable 
                  JNI global references: 1552
                  
                  Heap
                   par new generation   total 19136K, used 13050K [7d0000000, 7d14c0000, 7d4000000)
                    eden space 17024K,  65% used [7d0000000, 7d0ae7d60, 7d10a0000)
                    from space 2112K,  89% used [7d10a0000, 7d1276e88, 7d12b0000)
                    to   space 2112K,   0% used [7d12b0000, 7d12b0000, 7d14c0000)
                   concurrent mark-sweep generation total 44288K, used 28846K [7d4000000, 7d6b40000, 7f0000000)
                   concurrent-mark-sweep perm gen total 109064K, used 69234K [7f0000000, 7f6a82000, 800000000)
                  
                  

                   

                  This is the current Apple JVM by the way, so it may have different threading behaviour to other Oracle implementations.

                  • 7. Re: JBoss AS 7.1CR1 hangs when deployments reference JMS objects
                    jaikiran pai Master

                    Please file a JIRA. Looking at that HornetQ code, I am not sure what triggers that hang but that code does look susceptible to this kind of issue. For now file it in AS7 JIRA, we'll have to figure out whether it's the AS7 integration which triggers this or whether it's plain HornetQ issue.

                    • 8. Re: JBoss AS 7.1CR1 hangs when deployments reference JMS objects
                      Stephen Coy Master

                      Um, I just came back after dinner and found this:

                       
                      21:05:33,682 INFO  [org.jboss.weld] Stopping weld service
                      21:05:33,682 INFO  [org.jboss.as.deployment.connector] JBAS010410: Unbound JCA ConnectionFactory [java:/JmsXA]
                      21:05:33,696 INFO  [com.arjuna.ats.jbossatx] ARJUNA032018: Destroying TransactionManagerService
                      21:05:33,696 INFO  [com.arjuna.ats.jbossatx] ARJUNA032014: Stopping transaction recovery manager
                      21:05:33,710 INFO  [org.jboss.as.server.deployment] Stopped deployment messaging-ejb-1.0-SNAPSHOT.jar in 540189ms
                      21:05:33,714 INFO  [org.jboss.as.server.deployment] Stopped deployment messaging-web-1.0-SNAPSHOT.war in 540207ms
                      21:05:33,716 INFO  [org.jboss.as.server.deployment] Stopped deployment messaging-app-1.0-SNAPSHOT.ear in 540195ms
                      21:05:33,718 INFO  [org.jboss.as] JBoss AS 7.1.0.CR1b "Flux Capacitor" stopped in 540179ms

                      following the stack trace.

                       

                      9 minutes?

                      • 9. Re: JBoss AS 7.1CR1 hangs when deployments reference JMS objects
                        jaikiran pai Master

                        A 54 second shutdown time (for no real reason) isn't good either.

                         

                        Edit: It's actually 540 seconds - way worse!

                        • 11. Re: JBoss AS 7.1CR1 hangs when deployments reference JMS objects
                          pstackle Newbie

                          As a workaround, if you undeploy the deployment before calling shutdown, then it seems to shutdown cleanly/quickly.

                           

                          {code}

                          $ bin/jboss-admin.bat --connect "undeploy <ear-name>.ear, :shutdown"

                          {code}