1 2 Previous Next 15 Replies Latest reply on Oct 23, 2012 9:25 AM by michellegx11

    Failback reconnection problem

    michellegx11

      Hi all,

       

      I am using Hornetq version 2.2.5 Final and trying to establish a cluster with 2 node: node1 and node2. On each node, I have a Live machie and a Backup machine. But Standalone Hornetq on Node2 is the Backup of Live server on Node1; Standaloen on Node1 is the Backup of Live server on Node2.

       

      Node1: Live1 (embedded Hornetq) + Backup2 (standalone Hornetq)

      Node2: Live2 (embedded Hornetq) + Backup1 (standalone Hornetq)

       

      Live server is with embedded Hornetq of Jboss 6.1.0.Final with application software. Backup server is a Standalone Hornetq and is used only to treat unhandled Queue messages in case Live machine crashed.

       

      I tested by starting first 2 Live servers and then Backup server. After the starting up, I killed the Live1 machine, the Failover worked fine. Backup1 took the control and announced themselve as Live.

       

      Problem occured, when I restart the Live1 and try to Failback the traffic to the it. Sometimes (not all the time), I got the exception, indicating Live1 have problem to connect to other servers, when initializing Hornetq part during the starting up. But in the end, Live1 managed to take the control and Backup1 server announced to be Backup again.

       

      Logs at Live machine:

      20:47:23,418 INFO  [HornetQServerImpl] live server is starting with configuration HornetQ Configuration (clustered=true,backup=false,sharedStore=true,journalDirectory=/tmp/share/hornetq2/journal,bindingsDirectory=/tmp/share/hornetq2/bindings,largeMessagesDirectory=/tmp/share/hornetq2/largemessages,pagingDirectory=/tmp/share/hornetq2/paging)
      20:47:23,421 INFO  [HornetQServerImpl] Waiting to obtain live lock
      20:47:23,502 INFO  [JournalStorageManager] Using NIO Journal
      20:47:23,525 WARNING [HornetQServerImpl] Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.
      20:47:23,670 INFO  [ClusterManagerImpl] announcing backup
      20:47:26,955 WARN  [ClientSessionFactoryImpl] Tried 1 times to connect. Now giving up on reconnecting it.
      20:47:26,956 WARN  [ClusterManagerImpl] Unable to announce backup: HornetQException[errorCode=2 message=Cannot connect to server(s). Tried with all available servers.]
              at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:619) [:6.1.0.Final]
              at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:503) [:6.1.0.Final]
              at org.hornetq.core.server.cluster.impl.ClusterManagerImpl$1.run(ClusterManagerImpl.java:860) [:6.1.0.Final]
              at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:6.1.0.Final]
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_20]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_20]
              at java.lang.Thread.run(Thread.java:619) [:1.6.0_20]

      20:47:28,680 INFO  [FileLockNodeManager] Waiting to obtain live lock

      20:48:14,420 INFO  [FileLockNodeManager] Live Server Obtained live lock

      20:48:14,771 INFO  [NettyAcceptor] Started Netty Acceptor version 3.2.3.Final-r${buildNumber} 192.168.134.130:5445 for CORE protocol

      20:48:14,773 INFO  [NettyAcceptor] Started Netty Acceptor version 3.2.3.Final-r${buildNumber} 192.168.134.130:5455 for CORE protocol

      20:48:14,799 INFO  [HornetQServerImpl] Server is now live

      20:48:14,800 INFO  [HornetQServerImpl] HornetQ Server version 2.2.5.Final (HQ_2_2_5_FINAL_AS7, 121) [56d9023a-0d85-11e2-8d9a-000c299c9d74] startedBut

       

       

      Logs at Backup machine:

      [Thread-6] 20:48:13,925 INFO [org.hornetq.core.server.impl.FileLockNodeManager]  Waiting to become backup node
      [Thread-6] 20:48:13,925 INFO [org.hornetq.core.server.impl.FileLockNodeManager]  ** got backup lock
      [Thread-6] 20:48:13,926 INFO [org.hornetq.core.persistence.impl.journal.JournalStorageManager]  Using NIO Journal
      [Thread-6] 20:48:13,927 WARNING [org.hornetq.core.server.impl.HornetQServerImpl]  Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.
      [Thread-6] 20:48:13,997 INFO [org.hornetq.core.server.cluster.impl.ClusterManagerImpl]  announcing backup
      [Thread-6] 20:48:13,998 INFO [org.hornetq.core.server.impl.HornetQServerImpl]  HornetQ Backup Server version 2.2.5.Final (HQ_2_2_5_FINAL_AS7, 121) [56d9023a-0d85-11e2-8d9a-000c299c9d74] started, waiting live to fail before it gets active
      [Thread-0 (group:HornetQ-server-threads1963878523-955842370)] 20:48:17,069 INFO [org.hornetq.core.server.cluster.impl.ClusterManagerImpl]  backup announced

       

       

      With this situation, Live2 is not able to form the cluster with Live1 any more, but it kept form the cluster with the Backup1 server instead. Thus message going to the Live2 will not be send to Live1 but Backup1 instead. For Live1, it was able to form the cluster with Live2 and distributing messages between them.

       

      The only way to make Live2 discover Live1 again is by stopping Backup2 first, then Live2, then restart Live2 and finally Backup2. Stopping Live2 without stopping Backup2 first will not solve the problem, since restarting the Live2 without shutting down the Backup2 first, will result in the same exception while restarting Live2.

       

      I have tried to test in different ways, but not able to summarize in which situation the exception will occur. It seems to be generated "Randomly".

       

      Does anybody know the possible reason about this behaviour? Thanks a lot!

        • 1. Re: Failback reconnection problem
          clebert.suconic

          Can you try a newer version first?

          • 2. Re: Failback reconnection problem
            michellegx11

            Hi Clebert,

             

            We are considering of upgrade to the latest version in the next relase of our product. But with the current release, it was decided to use 2.2.5 already.

             

            I just want to find out, whether this exception is due to a bug of Hornetq or some configuration parameters. Do you have any suggestion on this point? Thanks!

             

            Kr,

             

            Xue

            • 3. Re: Failback reconnection problem
              michellegx11

              I have upgraded the Hornetq version to 2.2.14 and try to test the failover and failback. The failover is successful, but the failback is not working anymore. The Backup machine had announced "backup", and Live machine had announced to have "obtained the live lock". But the Live machine was freezed at this moment and didn't continue to deploy other packages any more.

               

              Failback related logs are shown as below:

               

              Live machine:

              11:48:03,764 INFO  [HornetQServerImpl] live server is starting with configuration HornetQ Configuration (clustered=true,backup=false,sharedStore=true,journalDirectory=/tmp/share/hornetq2/journal,bindingsDirectory=/tmp/share/hornetq2/bindings,largeMessagesDirectory=/tmp/share/hornetq2/largemessages,pagingDirectory=/tmp/share/hornetq2/paging)

              11:48:03,766 INFO  [HornetQServerImpl] Waiting to obtain live lock

              11:48:03,815 INFO  [JournalStorageManager] Using NIO Journal

              11:48:03,831 WARNING [HornetQServerImpl] Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.

              11:48:08,975 INFO  [FileLockNodeManager] Waiting to obtain live lock

              11:49:02,765 INFO  [FileLockNodeManager] Live Server Obtained live lock

               

              Backup machine:

               

              * [Activation for server HornetQServerImpl::serverUUID=a200ec8c-1900-11e2-b77e-bd8d4a8275c8] 18-Oct 11:49:8,257 INFO [FileLockNodeManager]  Waiting to become backup node

               

              * [Activation for server HornetQServerImpl::serverUUID=a200ec8c-1900-11e2-b77e-bd8d4a8275c8] 18-Oct 11:49:8,258 INFO [FileLockNodeManager]  ** got backup lock

               

              * [Activation for server HornetQServerImpl::serverUUID=a200ec8c-1900-11e2-b77e-bd8d4a8275c8] 18-Oct 11:49:8,259 INFO [JournalStorageManager]  Using NIO Journal

               

              * [Activation for server HornetQServerImpl::serverUUID=a200ec8c-1900-11e2-b77e-bd8d4a8275c8] 18-Oct 11:49:8,259 WARNING [HornetQServerImpl]  Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.

               

              * [Activation for server HornetQServerImpl::serverUUID=a200ec8c-1900-11e2-b77e-bd8d4a8275c8] 18-Oct 11:49:8,315 INFO [HornetQServerImpl]  HornetQ Backup Server version 2.2.14.Final (HQ_2_2_14_FINAL, 122) [a200ec8c-1900-11e2-b77e-bd8d4a8275c8] started, waiting live to fail before it gets active

               

              * [Thread-0 (HornetQ-server-HornetQServerImpl::serverUUID=a200ec8c-1900-11e2-b77e-bd8d4a8275c8-1681184607)] 18-Oct 11:49:8,317 INFO [ClusterConnectionImpl]  backup announced

              • 4. Re: Failback reconnection problem
                ataylor

                we cant really help you with out a thread dump to see where it is hanging, also make sure you clear down the journals just in case

                • 5. Re: Failback reconnection problem
                  michellegx11

                  Hi Andy,

                   

                  Thanks for your reply.

                   

                  I don't know whether it is important, when I take the thread dump, the Backup machine didn't announce Backup. Instead, it stopped in the step "waiting live to fail before it gets active".

                   

                  so I took the thread dump for both the Live machine and the Backup machine.

                   

                  Thread dump for Live machine:

                  2012-10-18 14:48:39

                  Full thread dump Java HotSpot(TM) 64-Bit Server VM (16.3-b01 mixed mode):

                   

                  "Attach Listener" daemon prio=10 tid=0x0000000044fa3000 nid=0x58f1 runnable [0x0000000000000000]

                     java.lang.Thread.State: RUNNABLE

                   

                  "Thread-12 (HornetQ-server-HornetQServerImpl::serverUUID=7a0af661-1921-11e2-9274-67c3f3743a8d-895089171)" prio=10 tid=0x0000000044569800 nid=0x58e0 waiting on condition [0x00000000473e9000]

                     java.lang.Thread.State: WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba8619618> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)

                          at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-11 (HornetQ-server-HornetQServerImpl::serverUUID=7a0af661-1921-11e2-9274-67c3f3743a8d-895089171)" prio=10 tid=0x0000000044567800 nid=0x58df waiting on condition [0x00000000472e8000]

                     java.lang.Thread.State: WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba8619618> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)

                          at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-10 (HornetQ-server-HornetQServerImpl::serverUUID=7a0af661-1921-11e2-9274-67c3f3743a8d-895089171)" prio=10 tid=0x0000000044565800 nid=0x58de waiting on condition [0x00000000471e7000]

                     java.lang.Thread.State: WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba8619618> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)

                          at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-9 (HornetQ-server-HornetQServerImpl::serverUUID=7a0af661-1921-11e2-9274-67c3f3743a8d-895089171)" prio=10 tid=0x00000000451bf800 nid=0x58dd waiting on condition [0x000000004030a000]

                     java.lang.Thread.State: WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba8619618> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)

                          at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-8 (HornetQ-server-HornetQServerImpl::serverUUID=7a0af661-1921-11e2-9274-67c3f3743a8d-895089171)" prio=10 tid=0x00000000451bb800 nid=0x58da waiting on condition [0x00000000470e6000]

                     java.lang.Thread.State: WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba8619618> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)

                          at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-7 (HornetQ-server-HornetQServerImpl::serverUUID=7a0af661-1921-11e2-9274-67c3f3743a8d-895089171)" prio=10 tid=0x00000000451b8800 nid=0x58d9 waiting on condition [0x0000000046fe5000]

                     java.lang.Thread.State: WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba8619618> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)

                          at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-6 (HornetQ-server-HornetQServerImpl::serverUUID=7a0af661-1921-11e2-9274-67c3f3743a8d-895089171)" prio=10 tid=0x00000000444f5000 nid=0x58d8 waiting on condition [0x0000000046ee4000]

                     java.lang.Thread.State: WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba8619618> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)

                          at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-5 (HornetQ-server-HornetQServerImpl::serverUUID=7a0af661-1921-11e2-9274-67c3f3743a8d-895089171)" prio=10 tid=0x00000000444f9000 nid=0x58d7 waiting on condition [0x0000000046de3000]

                     java.lang.Thread.State: WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba8619618> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)

                          at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "pool-9-thread-1" prio=10 tid=0x00000000444f1800 nid=0x58d6 waiting on condition [0x0000000046ce2000]

                     java.lang.Thread.State: TIMED_WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba2f96890> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)

                          at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "pool-8-thread-1" prio=10 tid=0x00000000444f0000 nid=0x58d5 waiting on condition [0x0000000046be1000]

                     java.lang.Thread.State: TIMED_WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba2f95c90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)

                          at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-4 (HornetQ-server-HornetQServerImpl::serverUUID=7a0af661-1921-11e2-9274-67c3f3743a8d-895089171)" prio=10 tid=0x00000000460f3800 nid=0x58d4 waiting on condition [0x0000000046ae0000]

                     java.lang.Thread.State: WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba8619618> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)

                          at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-3 (HornetQ-server-HornetQServerImpl::serverUUID=7a0af661-1921-11e2-9274-67c3f3743a8d-895089171)" prio=10 tid=0x00000000460f1000 nid=0x58d3 waiting on condition [0x00000000469df000]

                     java.lang.Thread.State: WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba8619618> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)

                          at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-2 (HornetQ-server-HornetQServerImpl::serverUUID=7a0af661-1921-11e2-9274-67c3f3743a8d-895089171)" prio=10 tid=0x00000000460ee000 nid=0x58d2 waiting on condition [0x00000000468de000]

                     java.lang.Thread.State: WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba8619618> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)

                          at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-1 (HornetQ-server-HornetQServerImpl::serverUUID=7a0af661-1921-11e2-9274-67c3f3743a8d-895089171)" prio=10 tid=0x00000000460ec800 nid=0x58d1 waiting on condition [0x00000000467dd000]

                     java.lang.Thread.State: WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba8619618> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)

                          at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-0 (HornetQ-Asynchronous-Persistent-Writes488997092-1123434088)" daemon prio=10 tid=0x00000000451dd000 nid=0x58d0 waiting on condition [0x00000000466dc000]

                     java.lang.Thread.State: WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba86ac090> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)

                          at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-4 (HornetQ-scheduled-threads-1132663502)" prio=10 tid=0x00000000451db000 nid=0x58cf waiting on condition [0x00000000465db000]

                     java.lang.Thread.State: TIMED_WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba8614ab0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)

                          at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-3 (HornetQ-scheduled-threads-1132663502)" prio=10 tid=0x00000000451e5000 nid=0x58ce waiting on condition [0x00000000464da000]

                     java.lang.Thread.State: TIMED_WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba8614ab0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)

                          at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-2 (HornetQ-scheduled-threads-1132663502)" prio=10 tid=0x00000000444fd000 nid=0x58cd waiting on condition [0x0000000043d32000]

                     java.lang.Thread.State: TIMED_WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba8614ab0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)

                          at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "hornetq-discovery-group-thread-dg-group1" daemon prio=10 tid=0x00000000460fc000 nid=0x58cc waiting for monitor entry [0x0000000043c31000]

                     java.lang.Thread.State: BLOCKED (on object monitor)

                          at org.hornetq.core.client.impl.ServerLocatorImpl.connectorsChanged(ServerLocatorImpl.java:1473)

                          - waiting to lock <0x00002aaba8619ed0> (a org.hornetq.core.client.impl.ServerLocatorImpl)

                          at org.hornetq.core.cluster.impl.DiscoveryGroupImpl.callListeners(DiscoveryGroupImpl.java:457)

                          at org.hornetq.core.cluster.impl.DiscoveryGroupImpl.run(DiscoveryGroupImpl.java:419)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-0 (HornetQ-server-HornetQServerImpl::serverUUID=7a0af661-1921-11e2-9274-67c3f3743a8d-895089171)" prio=10 tid=0x00002aabb495a000 nid=0x58cb in Object.wait() [0x0000000043b30000]

                     java.lang.Thread.State: TIMED_WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aaba8623ba8> (a java.lang.Object)

                          at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:1030)

                          - locked <0x00002aaba8623ba8> (a java.lang.Object)

                          at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:225)

                          at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:752)

                          - locked <0x00002aaba8619ed0> (a org.hornetq.core.client.impl.ServerLocatorImpl)

                          at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:593)

                          at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$2.run(ClusterConnectionImpl.java:485)

                          at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-1 (HornetQ-scheduled-threads-1132663502)" prio=10 tid=0x0000000044a37000 nid=0x58ca waiting on condition [0x0000000043a2f000]

                     java.lang.Thread.State: TIMED_WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba8614ab0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)

                          at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "hornetq-expiry-reaper-thread" prio=10 tid=0x0000000044a36800 nid=0x58c9 in Object.wait() [0x000000004392e000]

                     java.lang.Thread.State: TIMED_WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aaba85401a8> (a org.hornetq.core.postoffice.impl.PostOfficeImpl$Reaper)

                          at org.hornetq.core.postoffice.impl.PostOfficeImpl$Reaper.run(PostOfficeImpl.java:1319)

                          - locked <0x00002aaba85401a8> (a org.hornetq.core.postoffice.impl.PostOfficeImpl$Reaper)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Timer-1" daemon prio=10 tid=0x0000000044a3c000 nid=0x58c8 in Object.wait() [0x000000004382d000]

                     java.lang.Thread.State: TIMED_WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aaba874fa70> (a java.util.TaskQueue)

                          at java.util.TimerThread.mainLoop(Timer.java:509)

                          - locked <0x00002aaba874fa70> (a java.util.TaskQueue)

                          at java.util.TimerThread.run(Timer.java:462)

                   

                  "hornetq-buffer-timeout" prio=10 tid=0x0000000044d5d000 nid=0x58c7 waiting on condition [0x000000004372c000]

                     java.lang.Thread.State: WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba86999e0> (a java.util.concurrent.Semaphore$NonfairSync)

                          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)

                          at java.util.concurrent.Semaphore.acquire(Semaphore.java:286)

                          at org.hornetq.core.journal.impl.TimedBuffer$CheckTimer.run(TimedBuffer.java:438)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-0 (HornetQ-scheduled-threads-1132663502)" prio=10 tid=0x0000000045238000 nid=0x58c6 waiting on condition [0x00000000405f9000]

                     java.lang.Thread.State: TIMED_WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaba8614ab0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)

                          at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "AcceptorThread[ServerSocket[addr=/192.168.134.128,port=0,localport=4446]]" prio=10 tid=0x0000000043f13000 nid=0x58c5 runnable [0x0000000040d6e000]

                     java.lang.Thread.State: RUNNABLE

                          at java.net.PlainSocketImpl.socketAccept(Native Method)

                          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                          - locked <0x00002aaba8750260> (a java.net.SocksSocketImpl)

                          at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                          at java.net.ServerSocket.accept(ServerSocket.java:421)

                          at org.jboss.remoting.transport.socket.SocketServerInvoker.accept(SocketServerInvoker.java:1169)

                          at org.jboss.remoting.transport.socket.SocketServerInvoker.access$200(SocketServerInvoker.java:69)

                          at org.jboss.remoting.transport.socket.SocketServerInvoker$AcceptThread.run(SocketServerInvoker.java:1080)

                   

                  "ServerSocketRefresh" daemon prio=10 tid=0x0000000044650800 nid=0x58c4 in Object.wait() [0x000000004352a000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aaba87546a8> (a java.lang.Object)

                          at java.lang.Object.wait(Object.java:485)

                          at org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:904)

                          - locked <0x00002aaba87546a8> (a java.lang.Object)

                   

                  "RequestController-7" daemon prio=10 tid=0x00002aabb4400800 nid=0x58c2 in Object.wait() [0x000000004362b000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aaba853cd98> (a java.lang.Object)

                          at java.lang.Object.wait(Object.java:485)

                          at org.jacorb.poa.RequestController.waitForQueue(Unknown Source)

                          - locked <0x00002aaba853cd98> (a java.lang.Object)

                          at org.jacorb.poa.RequestController.run(Unknown Source)

                   

                  "RequestController-6" daemon prio=10 tid=0x00002aabb43ff800 nid=0x58bf in Object.wait() [0x0000000043429000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aaba853b790> (a java.lang.Object)

                          at java.lang.Object.wait(Object.java:485)

                          at org.jacorb.poa.RequestController.waitForQueue(Unknown Source)

                          - locked <0x00002aaba853b790> (a java.lang.Object)

                          at org.jacorb.poa.RequestController.run(Unknown Source)

                   

                  "RequestController-5" daemon prio=10 tid=0x00002aabb4420000 nid=0x58be in Object.wait() [0x0000000043328000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aaba853a818> (a java.lang.Object)

                          at java.lang.Object.wait(Object.java:485)

                          at org.jacorb.poa.RequestController.waitForQueue(Unknown Source)

                          - locked <0x00002aaba853a818> (a java.lang.Object)

                          at org.jacorb.poa.RequestController.run(Unknown Source)

                   

                  "pool-6-thread-1" prio=10 tid=0x00002aabb4aa1800 nid=0x58bd runnable [0x0000000043227000]

                     java.lang.Thread.State: RUNNABLE

                          at java.net.PlainSocketImpl.socketAccept(Native Method)

                          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                          - locked <0x00002aab7c01b410> (a java.net.SocksSocketImpl)

                          at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                          at java.net.ServerSocket.accept(ServerSocket.java:421)

                          at org.jnp.server.Main$AcceptHandler.run(Main.java:597)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "RMI TCP Accept-1098" daemon prio=10 tid=0x00002aabb4ec3000 nid=0x58bc runnable [0x0000000043126000]

                     java.lang.Thread.State: RUNNABLE

                          at java.net.PlainSocketImpl.socketAccept(Native Method)

                          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                          - locked <0x00002aab7c071b88> (a java.net.SocksSocketImpl)

                          at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                          at java.net.ServerSocket.accept(ServerSocket.java:421)

                          at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)

                          at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "RMI Reaper" prio=10 tid=0x00002aabb5948000 nid=0x58bb in Object.wait() [0x0000000043025000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aab7c78cc90> (a java.lang.ref.ReferenceQueue$Lock)

                          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

                          - locked <0x00002aab7c78cc90> (a java.lang.ref.ReferenceQueue$Lock)

                          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

                          at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "RMI TCP Accept-1091" daemon prio=10 tid=0x00002aabb4fa3000 nid=0x58ba runnable [0x0000000042c21000]

                     java.lang.Thread.State: RUNNABLE

                          at java.net.PlainSocketImpl.socketAccept(Native Method)

                          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                          - locked <0x00002aab7c07e3b0> (a java.net.SocksSocketImpl)

                          at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                          at java.net.ServerSocket.accept(ServerSocket.java:421)

                          at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)

                          at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "RMI TCP Accept-1090" daemon prio=10 tid=0x00002aabb4fa2000 nid=0x58b9 runnable [0x0000000042f24000]

                     java.lang.Thread.State: RUNNABLE

                          at java.net.PlainSocketImpl.socketAccept(Native Method)

                          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                          - locked <0x00002aab7c07e960> (a java.net.SocksSocketImpl)

                          at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                          at java.net.ServerSocket.accept(ServerSocket.java:421)

                          at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)

                          at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "RequestController-4" daemon prio=10 tid=0x00002aabb45e2800 nid=0x58b7 in Object.wait() [0x0000000042e23000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aab7bff8870> (a java.lang.Object)

                          at java.lang.Object.wait(Object.java:485)

                          at org.jacorb.poa.RequestController.waitForQueue(Unknown Source)

                          - locked <0x00002aab7bff8870> (a java.lang.Object)

                          at org.jacorb.poa.RequestController.run(Unknown Source)

                   

                  "ORB thread" prio=10 tid=0x00002aabb6894000 nid=0x58b6 in Object.wait() [0x0000000042d22000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aab7bfdbe30> (a java.lang.Object)

                          at java.lang.Object.wait(Object.java:485)

                          at org.jacorb.orb.ORB.run(Unknown Source)

                          - locked <0x00002aab7bfdbe30> (a java.lang.Object)

                          at org.jboss.iiop.CorbaORBService$1.run(CorbaORBService.java:319)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "RequestController-3" daemon prio=10 tid=0x00002aabb4fc7800 nid=0x58b4 in Object.wait() [0x0000000042b20000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aab7bff9640> (a java.lang.Object)

                          at java.lang.Object.wait(Object.java:485)

                          at org.jacorb.poa.RequestController.waitForQueue(Unknown Source)

                          - locked <0x00002aab7bff9640> (a java.lang.Object)

                          at org.jacorb.poa.RequestController.run(Unknown Source)

                   

                  "RequestController-2" daemon prio=10 tid=0x00002aabb4fc7000 nid=0x58b3 in Object.wait() [0x0000000042a1f000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aab7bff9fd0> (a java.lang.Object)

                          at java.lang.Object.wait(Object.java:485)

                          at org.jacorb.poa.RequestController.waitForQueue(Unknown Source)

                          - locked <0x00002aab7bff9fd0> (a java.lang.Object)

                          at org.jacorb.poa.RequestController.run(Unknown Source)

                   

                  "ServerSocketListener" daemon prio=10 tid=0x00002aabb45e9800 nid=0x58b2 runnable [0x000000004291e000]

                     java.lang.Thread.State: RUNNABLE

                          at java.net.PlainSocketImpl.socketAccept(Native Method)

                          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                          - locked <0x00002aab7bfdae48> (a java.net.SocksSocketImpl)

                          at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                          at java.net.ServerSocket.accept(ServerSocket.java:421)

                          at org.jacorb.orb.iiop.IIOPListener$Acceptor.run(Unknown Source)

                   

                  "RequestController-1" daemon prio=10 tid=0x0000000044fe7800 nid=0x58b1 in Object.wait() [0x000000004281d000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aab7bfec420> (a java.lang.Object)

                          at java.lang.Object.wait(Object.java:485)

                          at org.jacorb.poa.RequestController.waitForQueue(Unknown Source)

                          - locked <0x00002aab7bfec420> (a java.lang.Object)

                          at org.jacorb.poa.RequestController.run(Unknown Source)

                   

                  "AcceptorThread[ServerSocket[addr=/192.168.134.128,port=0,localport=3873]]" prio=10 tid=0x0000000046300000 nid=0x58b0 runnable [0x0000000040c6d000]

                     java.lang.Thread.State: RUNNABLE

                          at java.net.PlainSocketImpl.socketAccept(Native Method)

                          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                          - locked <0x00002aab7c0fb550> (a java.net.SocksSocketImpl)

                          at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                          at java.net.ServerSocket.accept(ServerSocket.java:421)

                          at org.jboss.remoting.transport.socket.SocketServerInvoker.accept(SocketServerInvoker.java:1169)

                          at org.jboss.remoting.transport.socket.SocketServerInvoker.access$200(SocketServerInvoker.java:69)

                          at org.jboss.remoting.transport.socket.SocketServerInvoker$AcceptThread.run(SocketServerInvoker.java:1080)

                   

                  "ServerSocketRefresh" daemon prio=10 tid=0x0000000044c11000 nid=0x58af in Object.wait() [0x0000000040209000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aab7c1037c8> (a java.lang.Object)

                          at java.lang.Object.wait(Object.java:485)

                          at org.jboss.remoting.transport.socket.SocketServerInvoker$ServerSocketRefresh.run(SocketServerInvoker.java:904)

                          - locked <0x00002aab7c1037c8> (a java.lang.Object)

                   

                  "Thread-2-FastReceiver" prio=10 tid=0x0000000046398000 nid=0x58ac runnable [0x000000004271c000]

                     java.lang.Thread.State: RUNNABLE

                          at java.net.PlainDatagramSocketImpl.receive0(Native Method)

                          - locked <0x00002aab7c10e0d0> (a java.net.PlainDatagramSocketImpl)

                          at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)

                          - locked <0x00002aab7c10e0d0> (a java.net.PlainDatagramSocketImpl)

                          at java.net.DatagramSocket.receive(DatagramSocket.java:725)

                          - locked <0x00002aaba20df9b0> (a java.net.DatagramPacket)

                          - locked <0x00002aab7c10e088> (a java.net.DatagramSocket)

                          at org.opennms.protocols.snmp.SnmpPortal$1.run(SnmpPortal.java:467)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "SnmpPortal--1" prio=10 tid=0x00000000463a0800 nid=0x58ab in Object.wait() [0x000000004261b000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aab7c10b1b8> (a java.util.LinkedList)

                          at java.lang.Object.wait(Object.java:485)

                          at org.opennms.protocols.snmp.SnmpPortal$Receiver.run(SnmpPortal.java:567)

                          - locked <0x00002aab7c10b1b8> (a java.util.LinkedList)

                   

                  "SnmpTimer" daemon prio=10 tid=0x0000000046395000 nid=0x58aa in Object.wait() [0x000000004251a000]

                     java.lang.Thread.State: TIMED_WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aab7c110540> (a java.util.TreeSet)

                          at org.opennms.protocols.snmp.SnmpTimer$TimerLogic.run(SnmpTimer.java:148)

                          - locked <0x00002aab7c110540> (a java.util.TreeSet)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-2-FastReceiver" prio=10 tid=0x0000000046388000 nid=0x58a9 runnable [0x0000000042419000]

                     java.lang.Thread.State: RUNNABLE

                          at java.net.PlainDatagramSocketImpl.receive0(Native Method)

                          - locked <0x00002aab7c110980> (a java.net.PlainDatagramSocketImpl)

                          at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)

                          - locked <0x00002aab7c110980> (a java.net.PlainDatagramSocketImpl)

                          at java.net.DatagramSocket.receive(DatagramSocket.java:725)

                          - locked <0x00002aaba1dba3c8> (a java.net.DatagramPacket)

                          - locked <0x00002aab7c110938> (a java.net.DatagramSocket)

                          at org.opennms.protocols.snmp.SnmpPortal$1.run(SnmpPortal.java:467)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "SnmpPortal--1" prio=10 tid=0x0000000046374000 nid=0x58a8 in Object.wait() [0x000000004126a000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aab7c112df8> (a java.util.LinkedList)

                          at java.lang.Object.wait(Object.java:485)

                          at org.opennms.protocols.snmp.SnmpPortal$Receiver.run(SnmpPortal.java:567)

                          - locked <0x00002aab7c112df8> (a java.util.LinkedList)

                   

                  "Timer-0" prio=10 tid=0x0000000046382000 nid=0x58a7 in Object.wait() [0x0000000042318000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aab7c1137b8> (a java.util.TaskQueue)

                          at java.lang.Object.wait(Object.java:485)

                          at java.util.TimerThread.mainLoop(Timer.java:483)

                          - locked <0x00002aab7c1137b8> (a java.util.TaskQueue)

                          at java.util.TimerThread.run(Timer.java:462)

                   

                  "Thread-2-FastReceiver" prio=10 tid=0x0000000046375800 nid=0x58a6 runnable [0x0000000041bb1000]

                     java.lang.Thread.State: RUNNABLE

                          at java.net.PlainDatagramSocketImpl.receive0(Native Method)

                          - locked <0x00002aab7bfe5c88> (a java.net.PlainDatagramSocketImpl)

                          at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:136)

                          - locked <0x00002aab7bfe5c88> (a java.net.PlainDatagramSocketImpl)

                          at java.net.DatagramSocket.receive(DatagramSocket.java:725)

                          - locked <0x00002aaba127ab10> (a java.net.DatagramPacket)

                          - locked <0x00002aab7bfe5c40> (a java.net.DatagramSocket)

                          at org.opennms.protocols.snmp.SnmpPortal$1.run(SnmpPortal.java:467)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "SnmpPortal--1" prio=10 tid=0x0000000046368000 nid=0x58a5 in Object.wait() [0x00000000404f8000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aab7bfe0260> (a java.util.LinkedList)

                          at java.lang.Object.wait(Object.java:485)

                          at org.opennms.protocols.snmp.SnmpPortal$Receiver.run(SnmpPortal.java:567)

                          - locked <0x00002aab7bfe0260> (a java.util.LinkedList)

                   

                  "GC Daemon" daemon prio=10 tid=0x00002aabb88fc000 nid=0x58a4 in Object.wait() [0x0000000041a98000]

                     java.lang.Thread.State: TIMED_WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aab74509008> (a sun.misc.GC$LatencyLock)

                          at sun.misc.GC$Daemon.run(GC.java:100)

                          - locked <0x00002aab74509008> (a sun.misc.GC$LatencyLock)

                   

                  "AuthenticationCacheFlushThread" prio=10 tid=0x00002aabb4a32000 nid=0x58a2 waiting on condition [0x0000000041997000]

                     java.lang.Thread.State: TIMED_WAITING (sleeping)

                          at java.lang.Thread.sleep(Native Method)

                          at org.jboss.security.auth.AuthenticationCacheFlushThread.run(AuthenticationCacheFlushThread.java:83)

                   

                  "Thread-2" prio=10 tid=0x00002aabb4260800 nid=0x58a0 waiting for monitor entry [0x0000000041167000]

                     java.lang.Thread.State: BLOCKED (on object monitor)

                          at org.hornetq.core.client.impl.ServerLocatorImpl.doClose(ServerLocatorImpl.java:1261)

                          - waiting to lock <0x00002aaba8619ed0> (a org.hornetq.core.client.impl.ServerLocatorImpl)

                          at org.hornetq.core.client.impl.ServerLocatorImpl.close(ServerLocatorImpl.java:1239)

                          at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl.activate(ClusterConnectionImpl.java:653)

                          - locked <0x00002aaba8623dd0> (a org.hornetq.core.server.cluster.impl.ClusterConnectionImpl)

                          at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl.start(ClusterConnectionImpl.java:385)

                          - locked <0x00002aaba8623dd0> (a org.hornetq.core.server.cluster.impl.ClusterConnectionImpl)

                          at org.hornetq.core.server.cluster.impl.ClusterManagerImpl.start(ClusterManagerImpl.java:205)

                          - locked <0x00002aaba8624a28> (a org.hornetq.core.server.cluster.impl.ClusterManagerImpl)

                          at org.hornetq.core.server.impl.HornetQServerImpl.initialisePart2(HornetQServerImpl.java:1493)

                          at org.hornetq.core.server.impl.HornetQServerImpl.access$1200(HornetQServerImpl.java:138)

                          at org.hornetq.core.server.impl.HornetQServerImpl$SharedStoreLiveActivation.run(HornetQServerImpl.java:1919)

                          at org.hornetq.core.server.impl.HornetQServerImpl.start(HornetQServerImpl.java:366)

                          - locked <0x00002aaba8540dd8> (a org.hornetq.core.server.impl.HornetQServerImpl)

                          at org.hornetq.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:269)

                          - locked <0x00002aaba8694300> (a org.hornetq.jms.server.impl.JMSServerManagerImpl)

                          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                          at java.lang.reflect.Method.invoke(Method.java:597)

                          at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:60)

                          at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:168)

                          at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)

                          at org.jboss.kernel.plugins.dependency.KernelControllerContextAction$JoinpointDispatchWrapper.execute(KernelControllerContextAction.java:257)

                          at org.jboss.kernel.plugins.dependency.ExecutionWrapper.execute(ExecutionWrapper.java:47)

                          at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerContextAction.java:125)

                          at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextAction.java:72)

                          at org.jboss.kernel.plugins.dependency.LifecycleAction.installActionInternal(LifecycleAction.java:202)

                          at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:54)

                          at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:42)

                          at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)

                          at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)

                          at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)

                          at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:379)

                          at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:2044)

                          at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:1083)

                          at org.jboss.dependency.plugins.AbstractController.executeOrIncrementStateDirectly(AbstractController.java:1322)

                          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1246)

                          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1139)

                          at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:894)

                          at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:641)

                          at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:182)

                          at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:58)

                          at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:63)

                          at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:55)

                          at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:179)

                          at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1832)

                          at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1550)

                          at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1571)

                          at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1491)

                          at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:379)

                          at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:2044)

                          at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:1083)

                          at org.jboss.dependency.plugins.AbstractController.executeOrIncrementStateDirectly(AbstractController.java:1322)

                          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1246)

                          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1139)

                          at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:939)

                          at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:654)

                          at org.jboss.deployers.plugins.deployers.DeployersImpl.change(DeployersImpl.java:1983)

                          at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:1076)

                          at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:679)

                          at org.jboss.system.server.profileservice.deployers.MainDeployerPlugin.process(MainDeployerPlugin.java:106)

                          at org.jboss.profileservice.dependency.ProfileControllerContext$DelegateDeployer.process(ProfileControllerContext.java:143)

                          at org.jboss.profileservice.dependency.ProfileDeployAction.deploy(ProfileDeployAction.java:151)

                          at org.jboss.profileservice.dependency.ProfileDeployAction.installActionInternal(ProfileDeployAction.java:94)

                          at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:54)

                          at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:42)

                          at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)

                          at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)

                          at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)

                          at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:379)

                          at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:2044)

                          at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:1083)

                          at org.jboss.dependency.plugins.AbstractController.executeOrIncrementStateDirectly(AbstractController.java:1322)

                          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1246)

                          at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1139)

                          at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:939)

                          at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:654)

                          at org.jboss.profileservice.dependency.ProfileActivationWrapper$BasicProfileActivation.start(ProfileActivationWrapper.java:190)

                          at org.jboss.profileservice.dependency.ProfileActivationWrapper.start(ProfileActivationWrapper.java:87)

                          at org.jboss.profileservice.dependency.ProfileActivationService.activateProfile(ProfileActivationService.java:215)

                          at org.jboss.profileservice.dependency.ProfileActivationService.activate(ProfileActivationService.java:159)

                          at org.jboss.profileservice.bootstrap.AbstractProfileServiceBootstrap.activate(AbstractProfileServiceBootstrap.java:112)

                          at org.jboss.profileservice.resolver.BasicResolverFactory$ProfileResolverFacade.deploy(BasicResolverFactory.java:87)

                          at org.jboss.profileservice.bootstrap.AbstractProfileServiceBootstrap.start(AbstractProfileServiceBootstrap.java:91)

                          at org.jboss.system.server.profileservice.bootstrap.BasicProfileServiceBootstrap.start(BasicProfileServiceBootstrap.java:132)

                          at org.jboss.system.server.profileservice.bootstrap.BasicProfileServiceBootstrap.start(BasicProfileServiceBootstrap.java:56)

                          at org.jboss.bootstrap.impl.base.server.AbstractServer.startBootstraps(AbstractServer.java:827)

                          at org.jboss.bootstrap.impl.base.server.AbstractServer$StartServerTask.run(AbstractServer.java:417)

                          - locked <0x00002aab74837168> (a org.jboss.bootstrap.impl.as.server.JBossASServerImpl)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "DestroyJavaVM" prio=10 tid=0x0000000043d84000 nid=0x5890 waiting on condition [0x0000000000000000]

                     java.lang.Thread.State: RUNNABLE

                   

                  "main" prio=10 tid=0x0000000043e5c800 nid=0x589f in Object.wait() [0x0000000040b6c000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aab72f321a8> (a java.lang.Thread)

                          at java.lang.Thread.join(Thread.java:1143)

                          - locked <0x00002aab72f321a8> (a java.lang.Thread)

                          at java.lang.Thread.join(Thread.java:1196)

                          at org.jboss.bootstrap.impl.base.server.AbstractServer.start(AbstractServer.java:348)

                          at org.jboss.bootstrap.impl.mc.server.AbstractMCServerBase.start(AbstractMCServerBase.java:273)

                          at org.jboss.bootstrap.impl.as.server.AbstractJBossASServerBase.start(AbstractJBossASServerBase.java:402)

                          at org.jboss.Main.boot(Main.java:472)

                          at org.jboss.Main$1.run(Main.java:816)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Low Memory Detector" daemon prio=10 tid=0x0000000043e39000 nid=0x589d runnable [0x0000000000000000]

                     java.lang.Thread.State: RUNNABLE

                   

                  "CompilerThread1" daemon prio=10 tid=0x0000000043e37000 nid=0x589c waiting on condition [0x0000000000000000]

                     java.lang.Thread.State: RUNNABLE

                   

                  "CompilerThread0" daemon prio=10 tid=0x0000000043e31800 nid=0x589b waiting on condition [0x0000000000000000]

                     java.lang.Thread.State: RUNNABLE

                   

                  "JDWP Event Helper Thread" daemon prio=10 tid=0x0000000043e1c000 nid=0x589a runnable [0x0000000000000000]

                     java.lang.Thread.State: RUNNABLE

                   

                  "JDWP Transport Listener: dt_socket" daemon prio=10 tid=0x0000000043e18800 nid=0x5899 runnable [0x0000000000000000]

                     java.lang.Thread.State: RUNNABLE

                   

                  "Signal Dispatcher" daemon prio=10 tid=0x0000000043e0d000 nid=0x5898 runnable [0x0000000000000000]

                     java.lang.Thread.State: RUNNABLE

                   

                  "Finalizer" daemon prio=10 tid=0x0000000043de9000 nid=0x5897 in Object.wait() [0x0000000041e13000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aab7339f810> (a java.lang.ref.ReferenceQueue$Lock)

                          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

                          - locked <0x00002aab7339f810> (a java.lang.ref.ReferenceQueue$Lock)

                          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

                          at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

                   

                  "Reference Handler" daemon prio=10 tid=0x0000000043de7000 nid=0x5896 in Object.wait() [0x0000000041d12000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aab7339f788> (a java.lang.ref.Reference$Lock)

                          at java.lang.Object.wait(Object.java:485)

                          at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

                          - locked <0x00002aab7339f788> (a java.lang.ref.Reference$Lock)

                   

                  "VM Thread" prio=10 tid=0x0000000043de2800 nid=0x5895 runnable

                   

                  "GC task thread#0 (ParallelGC)" prio=10 tid=0x0000000043d97800 nid=0x5891 runnable

                   

                  "GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000043d99000 nid=0x5892 runnable

                   

                  "GC task thread#2 (ParallelGC)" prio=10 tid=0x0000000043d9b000 nid=0x5893 runnable

                   

                  "GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000043d9d000 nid=0x5894 runnable

                   

                  "VM Periodic Task Thread" prio=10 tid=0x0000000043e3b800 nid=0x589e waiting on condition

                   

                  JNI global references: 13260

                   

                   

                  Thread dump for Backup machine:

                   

                  2012-10-18 14:57:41

                  Full thread dump Java HotSpot(TM) 64-Bit Server VM (16.3-b01 mixed mode):

                   

                  "Attach Listener" daemon prio=10 tid=0x0000000052922800 nid=0x61ca waiting on condition [0x0000000000000000]

                     java.lang.Thread.State: RUNNABLE

                   

                  "Thread-0 (HornetQ-server-HornetQServerImpl::serverUUID=7a0af661-1921-11e2-9274-67c3f3743a8d-72981851)" prio=10 tid=0x00002aaaf833a000 nid=0x6172 waiting on condition [0x0000000044a30000]

                     java.lang.Thread.State: TIMED_WAITING (sleeping)

                          at java.lang.Thread.sleep(Native Method)

                          at org.hornetq.core.client.impl.ServerLocatorImpl$StaticConnector.connect(ServerLocatorImpl.java:1659)

                          at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:587)

                          - locked <0x00002aaae3455ef0> (a org.hornetq.core.client.impl.ServerLocatorImpl)

                          at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl$2.run(ClusterConnectionImpl.java:485)

                          at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-1 (HornetQ-scheduled-threads-392807320)" prio=10 tid=0x00002aaaf816b800 nid=0x6171 waiting on condition [0x0000000042a10000]

                     java.lang.Thread.State: TIMED_WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaae3442c58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)

                          at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "hornetq-expiry-reaper-thread" prio=10 tid=0x00002aaaf825c800 nid=0x6170 in Object.wait() [0x0000000044026000]

                     java.lang.Thread.State: TIMED_WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aaae344e680> (a org.hornetq.core.postoffice.impl.PostOfficeImpl$Reaper)

                          at org.hornetq.core.postoffice.impl.PostOfficeImpl$Reaper.run(PostOfficeImpl.java:1319)

                          - locked <0x00002aaae344e680> (a org.hornetq.core.postoffice.impl.PostOfficeImpl$Reaper)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "hornetq-buffer-timeout" prio=10 tid=0x00002aaaf82d0800 nid=0x616f waiting on condition [0x0000000044127000]

                     java.lang.Thread.State: WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaae3448690> (a java.util.concurrent.Semaphore$NonfairSync)

                          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)

                          at java.util.concurrent.Semaphore.acquire(Semaphore.java:286)

                          at org.hornetq.core.journal.impl.TimedBuffer$CheckTimer.run(TimedBuffer.java:438)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-0 (HornetQ-scheduled-threads-392807320)" prio=10 tid=0x0000000052b1c000 nid=0x616e waiting on condition [0x000000004270d000]

                     java.lang.Thread.State: TIMED_WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaae3442c58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)

                          at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Activation for server HornetQServerImpl::serverUUID=7a0af661-1921-11e2-9274-67c3f3743a8d" prio=10 tid=0x00000000529ee000 nid=0x616d waiting on condition [0x000000004381e000]

                     java.lang.Thread.State: TIMED_WAITING (sleeping)

                          at java.lang.Thread.sleep(Native Method)

                          at org.hornetq.core.server.impl.FileLockNodeManager.lock(FileLockNodeManager.java:394)

                          at org.hornetq.core.server.impl.FileLockNodeManager.awaitLiveNode(FileLockNodeManager.java:184)

                          at org.hornetq.core.server.impl.HornetQServerImpl$SharedStoreBackupActivation.run(HornetQServerImpl.java:1961)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-1 (HornetQ-client-global-scheduled-threads-395237242)" daemon prio=10 tid=0x00000000529cb000 nid=0x611c waiting on condition [0x000000004391f000]

                     java.lang.Thread.State: WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaae763ea40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)

                          at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "pool-2-thread-1" prio=10 tid=0x0000000052d1f000 nid=0x6114 waiting on condition [0x0000000043218000]

                     java.lang.Thread.State: TIMED_WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaae00def50> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)

                          at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Thread-0 (HornetQ-client-global-scheduled-threads-395237242)" daemon prio=10 tid=0x00002aaaf830c000 nid=0x60dc waiting on condition [0x000000004280e000]

                     java.lang.Thread.State: WAITING (parking)

                          at sun.misc.Unsafe.park(Native Method)

                          - parking to wait for  <0x00002aaae763ea40> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

                          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)

                          at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)

                          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)

                          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "DestroyJavaVM" prio=10 tid=0x00002aaaf81c5800 nid=0x60b1 waiting on condition [0x0000000000000000]

                     java.lang.Thread.State: RUNNABLE

                   

                  "HornetQ Server Shutdown Timer" daemon prio=10 tid=0x00002aaaf822d000 nid=0x60d5 in Object.wait() [0x0000000040980000]

                     java.lang.Thread.State: TIMED_WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aaae76a4dd0> (a java.util.TaskQueue)

                          at java.util.TimerThread.mainLoop(Timer.java:509)

                          - locked <0x00002aaae76a4dd0> (a java.util.TaskQueue)

                          at java.util.TimerThread.run(Timer.java:462)

                   

                  "Naming Bootstrap#1" prio=10 tid=0x00002aaaf81c9000 nid=0x60d3 runnable [0x000000004250b000]

                     java.lang.Thread.State: RUNNABLE

                          at java.net.PlainSocketImpl.socketAccept(Native Method)

                          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                          - locked <0x00002aaae778b958> (a java.net.SocksSocketImpl)

                          at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                          at java.net.ServerSocket.accept(ServerSocket.java:421)

                          at org.jnp.server.Main$AcceptHandler.run(Main.java:597)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "GC Daemon" daemon prio=10 tid=0x00002aaaf81e8000 nid=0x60d2 in Object.wait() [0x000000004240a000]

                     java.lang.Thread.State: TIMED_WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aaab360e480> (a sun.misc.GC$LatencyLock)

                          at sun.misc.GC$Daemon.run(GC.java:100)

                          - locked <0x00002aaab360e480> (a sun.misc.GC$LatencyLock)

                   

                  "RMI Reaper" prio=10 tid=0x00002aaaf8198000 nid=0x60d1 in Object.wait() [0x0000000042309000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aaab3606570> (a java.lang.ref.ReferenceQueue$Lock)

                          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

                          - locked <0x00002aaab3606570> (a java.lang.ref.ReferenceQueue$Lock)

                          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

                          at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:333)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "RMI TCP Accept-2098" daemon prio=10 tid=0x00002aaaf815b000 nid=0x60d0 runnable [0x0000000041dce000]

                     java.lang.Thread.State: RUNNABLE

                          at java.net.PlainSocketImpl.socketAccept(Native Method)

                          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                          - locked <0x00002aaab3616460> (a java.net.SocksSocketImpl)

                          at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                          at java.net.ServerSocket.accept(ServerSocket.java:421)

                          at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)

                          at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "RMI TCP Accept-0" daemon prio=10 tid=0x00002aaaf8152000 nid=0x60ce runnable [0x000000004077e000]

                     java.lang.Thread.State: RUNNABLE

                          at java.net.PlainSocketImpl.socketAccept(Native Method)

                          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                          - locked <0x00002aaab360e550> (a java.net.SocksSocketImpl)

                          at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                          at java.net.ServerSocket.accept(ServerSocket.java:421)

                          at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)

                          at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)

                          at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "RMI TCP Accept-3000" daemon prio=10 tid=0x00002aaaf814c000 nid=0x60cd runnable [0x0000000041ccd000]

                     java.lang.Thread.State: RUNNABLE

                          at java.net.PlainSocketImpl.socketAccept(Native Method)

                          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                          - locked <0x00002aaab3616710> (a java.net.SocksSocketImpl)

                          at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                          at java.net.ServerSocket.accept(ServerSocket.java:421)

                          at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)

                          at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "RMI TCP Accept-0" daemon prio=10 tid=0x00002aaaf813c000 nid=0x60cc runnable [0x00000000418a5000]

                     java.lang.Thread.State: RUNNABLE

                          at java.net.PlainSocketImpl.socketAccept(Native Method)

                          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

                          - locked <0x00002aaab360e808> (a java.net.SocksSocketImpl)

                          at java.net.ServerSocket.implAccept(ServerSocket.java:453)

                          at java.net.ServerSocket.accept(ServerSocket.java:421)

                          at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)

                          at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)

                          at java.lang.Thread.run(Thread.java:619)

                   

                  "Low Memory Detector" daemon prio=10 tid=0x0000000052808000 nid=0x60bc runnable [0x0000000000000000]

                     java.lang.Thread.State: RUNNABLE

                   

                  "CompilerThread1" daemon prio=10 tid=0x0000000052805800 nid=0x60bb waiting on condition [0x0000000000000000]

                     java.lang.Thread.State: RUNNABLE

                   

                  "CompilerThread0" daemon prio=10 tid=0x0000000052800800 nid=0x60ba waiting on condition [0x0000000000000000]

                     java.lang.Thread.State: RUNNABLE

                   

                  "Signal Dispatcher" daemon prio=10 tid=0x00000000527fe800 nid=0x60b9 runnable [0x0000000000000000]

                     java.lang.Thread.State: RUNNABLE

                   

                  "Finalizer" daemon prio=10 tid=0x00000000527d9800 nid=0x60b8 in Object.wait() [0x00000000411b6000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aaab3616c48> (a java.lang.ref.ReferenceQueue$Lock)

                          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

                          - locked <0x00002aaab3616c48> (a java.lang.ref.ReferenceQueue$Lock)

                          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

                          at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

                   

                  "Reference Handler" daemon prio=10 tid=0x00000000527d7800 nid=0x60b7 in Object.wait() [0x00000000410b5000]

                     java.lang.Thread.State: WAITING (on object monitor)

                          at java.lang.Object.wait(Native Method)

                          - waiting on <0x00002aaab360e470> (a java.lang.ref.Reference$Lock)

                          at java.lang.Object.wait(Object.java:485)

                          at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

                          - locked <0x00002aaab360e470> (a java.lang.ref.Reference$Lock)

                   

                  "VM Thread" prio=10 tid=0x00000000527d3000 nid=0x60b6 runnable

                   

                  "GC task thread#0 (ParallelGC)" prio=10 tid=0x000000005278a000 nid=0x60b2 runnable

                   

                  "GC task thread#1 (ParallelGC)" prio=10 tid=0x000000005278c000 nid=0x60b3 runnable

                   

                  "GC task thread#2 (ParallelGC)" prio=10 tid=0x000000005278d800 nid=0x60b4 runnable

                   

                  "GC task thread#3 (ParallelGC)" prio=10 tid=0x000000005278f800 nid=0x60b5 runnable

                   

                  "VM Periodic Task Thread" prio=10 tid=0x00002aaaf8154800 nid=0x60cf waiting on condition

                   

                  JNI global references: 842

                  • 6. Re: Failback reconnection problem
                    ataylor

                    Im confused now, your previous logs implied that this was happening at initial startup not after live server restarted.

                     

                    2 things i would do

                     

                    1. upgrade to the latest version

                    2. make sure the failback example works and go from there

                    • 7. Re: Failback reconnection problem
                      michellegx11

                      Hi Andy,

                       

                      I could try with the latest version 2.3.0.Beta1. But the failback worked well with my configuration with 2.2.5Final. Do you think the error could still due to the hornetq config file?

                      • 8. Re: Failback reconnection problem
                        ataylor

                        its hard to say, try the example in 2.2.5 see if that works then

                        • 9. Re: Failback reconnection problem
                          michellegx11

                          ok, I will have a try. Thanks!

                          • 10. Re: Failback reconnection problem
                            michellegx11

                            Hi Andy,

                             

                            I have found the reason for my problem. The <journal-max-io> value was not consistent for my Live and Backup machine. I think I have updated one by mistake while doing other test.

                             

                            May I ask you another question? Is the Failover structure support journal-type ASYNCIO? We wanted to improve the performance by set the journal-type to ASYNCIO with journal-max-io = 4, but after the change, the Backup machine become alive directly, with the detailed log shown below. Thanks for the answer in advance!

                             

                            * [main] 19-Oct 16:18:5,904 INFO [HornetQBootstrapServer]  Starting HornetQ Server

                             

                            * [main] 19-Oct 16:18:6,662 INFO [HornetQServerImpl]  backup server is starting with configuration HornetQ Configuration (clustered=true,backup=true,sharedStore=true,journalDirectory=/tmp/share/hornetq1/journal,bindingsDirectory=/tmp/share/hornetq1/bindings,largeMessagesDirectory=/tmp/share/hornetq1/largemessages,pagingDirectory=/tmp/share/hornetq1/paging)

                             

                            * [Activation for server HornetQServerImpl::serverUUID=8dc83167-19f7-11e2-818f-611cb5cae5df] 19-Oct 16:18:6,663 INFO [AIOFileLockNodeManager]  Waiting to become backup node

                             

                            * [Activation for server HornetQServerImpl::serverUUID=8dc83167-19f7-11e2-818f-611cb5cae5df] 19-Oct 16:18:6,664 INFO [AIOFileLockNodeManager]  ** got backup lock

                             

                            * [Activation for server HornetQServerImpl::serverUUID=8dc83167-19f7-11e2-818f-611cb5cae5df] 19-Oct 16:18:6,684 INFO [JournalStorageManager]  Using AIO Journal

                             

                            * [Activation for server HornetQServerImpl::serverUUID=8dc83167-19f7-11e2-818f-611cb5cae5df] 19-Oct 16:18:6,696 WARNING [HornetQServerImpl]  Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.

                             

                            * [Activation for server HornetQServerImpl::serverUUID=8dc83167-19f7-11e2-818f-611cb5cae5df] 19-Oct 16:18:6,858 INFO [HornetQServerImpl]  HornetQ Backup Server version 2.2.14.Final (HQ_2_2_14_FINAL, 122) [8dc83167-19f7-11e2-818f-611cb5cae5df] started, waiting live to fail before it gets active

                             

                            * [Thread-0 (HornetQ-server-HornetQServerImpl::serverUUID=8dc83167-19f7-11e2-818f-611cb5cae5df-1589377628)] 19-Oct 16:18:6,923 INFO [ClusterConnectionImpl]  backup announced

                             

                            * [Activation for server HornetQServerImpl::serverUUID=8dc83167-19f7-11e2-818f-611cb5cae5df] 19-Oct 16:18:7,678 INFO [HornetQServerImpl]  trying to deploy queue jms.queue.DLQ

                             

                            * [Activation for server HornetQServerImpl::serverUUID=8dc83167-19f7-11e2-818f-611cb5cae5df] 19-Oct 16:18:7,698 INFO [HornetQServerImpl]  trying to deploy queue jms.queue.ExpiryQueue

                             

                            * [Activation for server HornetQServerImpl::serverUUID=8dc83167-19f7-11e2-818f-611cb5cae5df] 19-Oct 16:18:7,704 INFO [HornetQServerImpl]  trying to deploy queue jms.queue.bms-tdr-queue

                             

                            * [Activation for server HornetQServerImpl::serverUUID=8dc83167-19f7-11e2-818f-611cb5cae5df] 19-Oct 16:18:7,754 INFO [NettyAcceptor]  Started Netty Acceptor version 3.2.5.Final-a96d88c localhost:5446 for CORE protocol

                             

                            * [Activation for server HornetQServerImpl::serverUUID=8dc83167-19f7-11e2-818f-611cb5cae5df] 19-Oct 16:18:7,755 INFO [NettyAcceptor]  Started Netty Acceptor version 3.2.5.Final-a96d88c localhost:5456 for CORE protocol

                             

                            * [Activation for server HornetQServerImpl::serverUUID=8dc83167-19f7-11e2-818f-611cb5cae5df] 19-Oct 16:18:7,768 INFO [HornetQServerImpl]  Backup Server is now live

                            • 11. Re: Failback reconnection problem
                              michellegx11

                              I was wrong saying that the Live restarting problem was solved. It came back again now, I don't know why.

                               

                              I tried rolled back to Hornetq 2.2.5, and with the configuration file it worked well. With 2.2.14Final, the Live machine get stuck during the restarting.

                               

                              I have made more tests, I have found out, if I install the 2 pairs of Live/Backup on the same server, as shown below, it worked as well:

                              Node1: Live1 (embedded Hornetq) + Backup1 (standalone Hornetq)

                              Node2: Live2 (embedded Hornetq) + Backup2 (standalone Hornetq)

                               

                              With the current structural we have in our project, shown as below, the Live machine get blocked. But if I stop the other Live machine, the restarting Live machine will start to deploy again.  

                              Node1: Live1 (embedded Hornetq) + Backup2 (standalone Hornetq)

                              Node2: Live2 (embedded Hornetq) + Backup1 (standalone Hornetq)

                               

                              I am using a nfs to store hornetq data files for Live/Backup pairs. I don't know whether it could due to the type of the file system of the shared store?

                              • 12. Re: Failback reconnection problem
                                ataylor

                                if it did work then didn't i can only assume something has changed in your config or env, did you try the example like i suggested

                                • 13. Re: Failback reconnection problem
                                  michellegx11

                                  Hi Andy,

                                   

                                  Yes, I have tried to use the configuration files from the example multiple-failover-failback directly. But adding the directory related settings (doesn't exist on config file of server0), otherwise Backup can not recognize the Live machine; also adding the journal-type settings as NIO, because the default is AIO which causes "same node id" Warnings. I got even more wierd behavior: Backup didn't announce itself as Backup, but will become alive when Live is killed, as shown below:

                                   

                                  * [main] 23-Oct 11:55:58,115 INFO [HornetQBootstrapServer]  Starting HornetQ Server

                                   

                                  * [main] 23-Oct 11:55:58,984 INFO [HornetQServerImpl]  backup server is starting with configuration HornetQ Configuration (clustered=true,backup=true,sharedStore=true,journalDirectory=/tmp/share/hornetq2/journal,bindingsDirectory=/tmp/share/hornetq2/bindings,largeMessagesDirectory=/tmp/share/hornetq2/largemessages,pagingDirectory=/tmp/share/hornetq2/paging)

                                   

                                  * [Activation for server HornetQServerImpl::serverUUID=8c1759f5-1cf2-11e2-b5fd-07a219693ecb] 23-Oct 11:55:58,987 INFO [FileLockNodeManager]  Waiting to become backup node

                                   

                                  * [Activation for server HornetQServerImpl::serverUUID=8c1759f5-1cf2-11e2-b5fd-07a219693ecb] 23-Oct 11:56:2,997 INFO [FileLockNodeManager]  ** got backup lock

                                   

                                  * [Activation for server HornetQServerImpl::serverUUID=8c1759f5-1cf2-11e2-b5fd-07a219693ecb] 23-Oct 11:56:3,20 INFO [JournalStorageManager]  Using NIO Journal

                                   

                                  * [Activation for server HornetQServerImpl::serverUUID=8c1759f5-1cf2-11e2-b5fd-07a219693ecb] 23-Oct 11:56:3,31 WARNING [HornetQServerImpl]  Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.

                                   

                                  * [Activation for server HornetQServerImpl::serverUUID=8c1759f5-1cf2-11e2-b5fd-07a219693ecb] 23-Oct 11:56:3,186 INFO [HornetQServerImpl]  HornetQ Backup Server version 2.2.14.Final (HQ_2_2_14_FINAL, 122) [8c1759f5-1cf2-11e2-b5fd-07a219693ecb] started, waiting live to fail before it gets active

                                   

                                  * [Activation for server HornetQServerImpl::serverUUID=8c1759f5-1cf2-11e2-b5fd-07a219693ecb] 23-Oct 11:59:25,775 INFO [HornetQServerImpl]  trying to deploy queue jms.queue.exampleQueue

                                   

                                  * [Activation for server HornetQServerImpl::serverUUID=8c1759f5-1cf2-11e2-b5fd-07a219693ecb] 23-Oct 11:59:25,830 INFO [NettyAcceptor]  Started Netty Acceptor version 3.2.5.Final-a96d88c localhost:5446 for CORE protocol

                                   

                                  If I install Live/Backup on the same node, again, everything goes well.

                                   

                                   

                                  The first time the problem solve, was because I have noticed for one of my Live machine had the journal-max-io set to 4 instead of 1. After I updated it, the Live machine got started, so I thought the problem was due to that settings. But later, with the same environment, same settings, the problem came back again mysterily.

                                   

                                  The problem seems only occured, when I use NFS as the shared folder for Live/Backup folder. So I don't know whether it was the reason for the whole problem.

                                  • 14. Re: Failback reconnection problem
                                    ataylor

                                    as per the user manual you cant use NFS for shared store

                                    1 2 Previous Next