1 Reply Latest reply on Aug 13, 2012 4:56 PM by clebert.suconic

    Thread dump org.hornetq.tests.integration.cluster.failover.ReplicatedPagedFailoverTest::#test testFailBack

    clebert.suconic
    clebert.suconic Aug 10, 2012 9:55 PM

      I have seen this same hang in other failback case. I dont think it's related to paging.. but on the failback mechanism.

       

      It's Friday night and I won't have time to evaluate this now.. just wanted to register here. Francisco asked me for it earlier. If anyone gets to it before me I will take a look on monday:

       

      Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.2-b06 mixed mode):

       

       

      "Thread-110" prio=10 tid=0x00007f1e05418000 nid=0x539 runnable [0x00007f1e5110f000]

         java.lang.Thread.State: RUNNABLE

                at java.io.FileInputStream.readBytes(Native Method)

                at java.io.FileInputStream.read(FileInputStream.java:220)

                at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)

                at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)

                at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)

                - locked <0x00000007eca41278> (a java.io.InputStreamReader)

                at java.io.InputStreamReader.read(InputStreamReader.java:167)

                at java.io.BufferedReader.fill(BufferedReader.java:136)

                at java.io.BufferedReader.readLine(BufferedReader.java:299)

                - locked <0x00000007eca41278> (a java.io.InputStreamReader)

                at java.io.BufferedReader.readLine(BufferedReader.java:362)

                at org.codehaus.plexus.util.cli.StreamPumper.run(StreamPumper.java:129)

       

       

      "Thread-109" prio=10 tid=0x00007f1e04ffe800 nid=0x538 runnable [0x00007f1e5135a000]

         java.lang.Thread.State: RUNNABLE

                at java.io.FileInputStream.readBytes(Native Method)

                at java.io.FileInputStream.read(FileInputStream.java:220)

                at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)

                at java.io.BufferedInputStream.read(BufferedInputStream.java:317)

                - locked <0x00000007ed1fd910> (a java.io.BufferedInputStream)

                at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)

                at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)

                at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)

                - locked <0x00000007eca3e6d8> (a java.io.InputStreamReader)

                at java.io.InputStreamReader.read(InputStreamReader.java:167)

                at java.io.BufferedReader.fill(BufferedReader.java:136)

                at java.io.BufferedReader.readLine(BufferedReader.java:299)

                - locked <0x00000007eca3e6d8> (a java.io.InputStreamReader)

                at java.io.BufferedReader.readLine(BufferedReader.java:362)

                at org.codehaus.plexus.util.cli.StreamPumper.run(StreamPumper.java:129)

       

       

      "process reaper" daemon prio=10 tid=0x00007f1e041ea800 nid=0x536 runnable [0x00007f1e5100e000]

         java.lang.Thread.State: RUNNABLE

                at java.lang.UNIXProcess.waitForProcessExit(Native Method)

                at java.lang.UNIXProcess.access$900(UNIXProcess.java:20)

                at java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)

       

       

      "ThreadedStreamConsumer" prio=10 tid=0x00007f1e05042800 nid=0x535 runnable [0x00007f1e5145b000]

         java.lang.Thread.State: RUNNABLE

                at java.io.FileOutputStream.writeBytes(Native Method)

                at java.io.FileOutputStream.write(FileOutputStream.java:282)

                at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)

                at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)

                - locked <0x0000000744911898> (a java.io.BufferedOutputStream)

                at java.io.PrintStream.write(PrintStream.java:432)

                - locked <0x0000000744911878> (a java.io.PrintStream)

                at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)

                at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)

                at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:85)

                - locked <0x00000007449119c8> (a java.io.OutputStreamWriter)

                at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:168)

                at java.io.PrintStream.newLine(PrintStream.java:496)

                - locked <0x0000000744911878> (a java.io.PrintStream)

                at java.io.PrintStream.println(PrintStream.java:757)

                - locked <0x0000000744911878> (a java.io.PrintStream)

                at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:80)

                at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:67)

                at java.lang.Thread.run(Thread.java:662)

       

       

      "resolver-5" daemon prio=10 tid=0x00007f1e04e85000 nid=0x6f85 waiting on condition [0x00007f1e51766000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x0000000744587c30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

       

       

      "Pipe writer thread: channel" prio=10 tid=0x00007f1e00006800 nid=0x6f66 waiting on condition [0x00007f1e51867000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x0000000744939a38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

       

       

      "resolver-4" daemon prio=10 tid=0x00007f1e04a3d800 nid=0x6f4e waiting on condition [0x00007f1e51fcd000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x0000000744587c30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

       

       

      "resolver-3" daemon prio=10 tid=0x00007f1e04d26800 nid=0x6f4d waiting on condition [0x00007f1e5197a000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x0000000744587c30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

       

       

      "resolver-2" daemon prio=10 tid=0x00007f1e041dd800 nid=0x6f4c waiting on condition [0x00007f1e51b7c000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x0000000744587c30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

       

       

      "resolver-1" daemon prio=10 tid=0x00007f1e04d5b800 nid=0x6f4b waiting on condition [0x00007f1e51a7b000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x0000000744587c30> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

       

       

      "pool-1-thread-1" prio=10 tid=0x00007f1e00004000 nid=0x6f29 in Object.wait() [0x00007f1e523f5000]

         java.lang.Thread.State: WAITING (on object monitor)

                at java.lang.Object.wait(Native Method)

                - waiting on <0x00000007eca3de68> (a java.lang.UNIXProcess)

                at java.lang.Object.wait(Object.java:485)

                at java.lang.UNIXProcess.waitFor(UNIXProcess.java:165)

                - locked <0x00000007eca3de68> (a java.lang.UNIXProcess)

                at org.codehaus.plexus.util.cli.CommandLineUtils$1.call(CommandLineUtils.java:172)

                at org.codehaus.plexus.util.cli.CommandLineUtils.executeCommandLine(CommandLineUtils.java:113)

                at org.codehaus.plexus.util.cli.CommandLineUtils.executeCommandLine(CommandLineUtils.java:86)

                at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:276)

                at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:113)

                at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:199)

                at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAllProviders(AbstractSurefireMojo.java:176)

                at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:135)

                at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:98)

                at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:107)

                at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)

                at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)

                at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)

                at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)

                at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)

                at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)

                at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)

                at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:319)

                at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)

                at org.jvnet.hudson.maven3.launcher.Maven3Launcher.main(Maven3Launcher.java:79)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                at java.lang.reflect.Method.invoke(Method.java:597)

                at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)

                at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)

                at org.jvnet.hudson.maven3.agent.Maven3Main.launch(Maven3Main.java:158)

                at hudson.maven.Maven3Builder.call(Maven3Builder.java:122)

                at hudson.maven.Maven3Builder.call(Maven3Builder.java:74)

                at hudson.remoting.UserRequest.perform(UserRequest.java:118)

                at hudson.remoting.UserRequest.perform(UserRequest.java:48)

                at hudson.remoting.Request$2.run(Request.java:287)

                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

                at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

                at java.util.concurrent.FutureTask.run(FutureTask.java:138)

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                at java.lang.Thread.run(Thread.java:662)

       

       

      "Channel reader thread: channel" prio=10 tid=0x00007f1e5811c800 nid=0x6f28 runnable [0x00007f1e524f8000]

         java.lang.Thread.State: RUNNABLE

                at java.net.SocketInputStream.socketRead0(Native Method)

                at java.net.SocketInputStream.read(SocketInputStream.java:129)

                at java.io.FilterInputStream.read(FilterInputStream.java:116)

                at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

                at java.io.BufferedInputStream.read(BufferedInputStream.java:237)

                - locked <0x000000074493a2b0> (a java.io.BufferedInputStream)

                at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2248)

                at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2541)

                at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2551)

                at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1296)

                at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)

                at hudson.remoting.Channel$ReaderThread.run(Channel.java:1087)

       

       

      "Low Memory Detector" daemon prio=10 tid=0x00007f1e58098800 nid=0x6f25 runnable [0x0000000000000000]

         java.lang.Thread.State: RUNNABLE

       

       

      "C2 CompilerThread1" daemon prio=10 tid=0x00007f1e58096000 nid=0x6f24 waiting on condition [0x0000000000000000]

         java.lang.Thread.State: RUNNABLE

       

       

      "C2 CompilerThread0" daemon prio=10 tid=0x00007f1e58093800 nid=0x6f23 waiting on condition [0x0000000000000000]

         java.lang.Thread.State: RUNNABLE

       

       

      "Signal Dispatcher" daemon prio=10 tid=0x00007f1e58091000 nid=0x6f22 waiting on condition [0x0000000000000000]

         java.lang.Thread.State: RUNNABLE

       

       

      "Finalizer" daemon prio=10 tid=0x00007f1e58074800 nid=0x6f21 in Object.wait() [0x00007f1e52c50000]

         java.lang.Thread.State: WAITING (on object monitor)

                at java.lang.Object.wait(Native Method)

                - waiting on <0x0000000744c6e0c0> (a java.lang.ref.ReferenceQueue$Lock)

                at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

                - locked <0x0000000744c6e0c0> (a java.lang.ref.ReferenceQueue$Lock)

                at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

                at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

       

       

      "Reference Handler" daemon prio=10 tid=0x00007f1e58072800 nid=0x6f20 in Object.wait() [0x00007f1e52d51000]

         java.lang.Thread.State: WAITING (on object monitor)

                at java.lang.Object.wait(Native Method)

                - waiting on <0x0000000744c6e080> (a java.lang.ref.Reference$Lock)

                at java.lang.Object.wait(Object.java:485)

                at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

                - locked <0x0000000744c6e080> (a java.lang.ref.Reference$Lock)

       

       

      "main" prio=10 tid=0x00007f1e58006800 nid=0x6f16 in Object.wait() [0x00007f1e5c2a0000]

         java.lang.Thread.State: WAITING (on object monitor)

                at java.lang.Object.wait(Native Method)

                - waiting on <0x0000000744939b80> (a hudson.remoting.Channel)

                at java.lang.Object.wait(Object.java:485)

                at hudson.remoting.Channel.join(Channel.java:780)

                - locked <0x0000000744939b80> (a hudson.remoting.Channel)

                at hudson.remoting.Launcher.main(Launcher.java:420)

                at hudson.remoting.Launcher.main(Launcher.java:400)

                at hudson.remoting.Launcher.main(Launcher.java:396)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                at java.lang.reflect.Method.invoke(Method.java:597)

                at org.jvnet.hudson.maven3.agent.Maven3Main.main(Maven3Main.java:131)

                at org.jvnet.hudson.maven3.agent.Maven3Main.main(Maven3Main.java:63)

       

       

      "VM Thread" prio=10 tid=0x00007f1e5806c000 nid=0x6f1f runnable

       

       

      "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f1e58019800 nid=0x6f17 runnable

       

       

      "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f1e5801b800 nid=0x6f18 runnable

       

       

      "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f1e5801d000 nid=0x6f19 runnable

       

       

      "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f1e5801f000 nid=0x6f1a runnable

       

       

      "GC task thread#4 (ParallelGC)" prio=10 tid=0x00007f1e58021000 nid=0x6f1b runnable

       

       

      "GC task thread#5 (ParallelGC)" prio=10 tid=0x00007f1e58022800 nid=0x6f1c runnable

       

       

      "GC task thread#6 (ParallelGC)" prio=10 tid=0x00007f1e58024800 nid=0x6f1d runnable

       

       

      "GC task thread#7 (ParallelGC)" prio=10 tid=0x00007f1e58026800 nid=0x6f1e runnable

       

       

      "VM Periodic Task Thread" prio=10 tid=0x00007f1e580a3000 nid=0x6f26 waiting on condition

       

       

       

       

      "Thread-10 (HornetQ-server-HornetQServerImpl::ReplicatedPagedFailoverTest/liveServer-77892085)" prio=10 tid=0x00007f07e0df4000 nid=0x7df8 waiting on condition [0x00007f0854f74000]

         java.lang.Thread.State: WAITING (parking)

      JNI global references: 1392

       

       

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff440070> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)Heap

       

       

      PSYoungGen               at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) total 894720K, used 769197K

      [0x00000007c1600000, 0x00000007fcc80000, 0x0000000800000000)

        eden          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) space 826176K, 93% used [0x00000007c1600000,0x00000007f052b650,0x00000007f3cd0000)

       

       

        from space 68544K, 0% used [0x00000007f82c0000,0x00000007f82c0000,0x00000007fc5b0000)

        to            at java.lang.Thread.run(Thread.java:662) space 71616K, 0% used [0x00000007f3cd0000,0x00000007f3cd0000,0x00000007f82c0000)

       

       

      PSOldGen        total 395904K, used 65225K [0x0000000744200000, 0x000000075c4a0000, 0x00000007c1600000)

      "Thread-9 (HornetQ-server-HornetQServerImpl::ReplicatedPagedFailoverTest/liveServer-77892085)" prio=10 tid=0x00007f07e03c2000 nid=0x7df7 waiting on condition [0x00007f082cece000]  object space 395904K, 16% used [0x0000000744200000,0x00000007481b2420,0x000000075c4a0000)

       

       

      PSPermGen       total 83968K, used 73969K [0x000000073f000000, 0x0000000744200000, 0x0000000744200000)

         java.lang.Thread.State: WAITING (parking)  object space 83968K, 88% used

      [0x000000073f000000,0x000000074383c480,0x0000000744200000)

       

       

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff440070> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-8 (HornetQ-server-HornetQServerImpl::ReplicatedPagedFailoverTest/liveServer-77892085)" prio=10 tid=0x00007f07e0e0b800 nid=0x7df6 waiting on condition [0x00007f075eff6000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff440070> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-7 (HornetQ-server-HornetQServerImpl::ReplicatedPagedFailoverTest/liveServer-77892085)" prio=10 tid=0x00007f07e033a000 nid=0x7df4 waiting on condition [0x00007f082c3c3000]

         java.lang.Thread.State: TIMED_WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff447c88> (a java.util.concurrent.CountDownLatch$Sync)

                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)

                at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253)

                at org.hornetq.core.client.impl.ServerLocatorImpl$StaticConnector.connect(ServerLocatorImpl.java:1743)

                at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:609)

                - locked <0x00000007ff458668> (a org.hornetq.core.client.impl.ServerLocatorImpl)

                at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:599)

                at org.hornetq.core.client.impl.ServerLocatorImpl$3.run(ServerLocatorImpl.java:572)

                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:99)

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-6 (HornetQ-server-HornetQServerImpl::ReplicatedPagedFailoverTest/liveServer-77892085)" prio=10 tid=0x00007f07e020e000 nid=0x7df3 waiting on condition [0x00007f082d2d2000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff440070> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

      "hornetq-failure-check-thread" prio=10 tid=0x00007f07e0140800 nid=0x7df2 waiting on condition [0x00007f075fefd000]

         java.lang.Thread.State: TIMED_WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff4502b0> (a java.util.concurrent.CountDownLatch$Sync)

                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)

                at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253)

                at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread.run(RemotingServiceImpl.java:626)

      "Thread-5 (HornetQ-server-HornetQServerImpl::ReplicatedPagedFailoverTest/liveServer-77892085)" prio=10 tid=0x00007f07e0d8b800 nid=0x7df1 waiting on condition [0x00007f075f0f7000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff440070> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-0 (HornetQ-Asynchronous-Persistent-Writes493508994-1449528699)" daemon prio=10 tid=0x00007f075982c000 nid=0x7df0 waiting on condition [0x00007f082cfcf000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff478350> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-1 (HornetQ-scheduled-threads-2110692774)" prio=10 tid=0x00007f075905d800 nid=0x7dee waiting on condition [0x00007f082cbcb000]

         java.lang.Thread.State: TIMED_WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff468128> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)

                at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)

                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)

                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

      "hornetq-buffer-timeout" prio=10 tid=0x00007f07e0df2800 nid=0x7ded waiting on condition [0x00007f082c1c1000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff440510> (a java.util.concurrent.Semaphore$NonfairSync)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)

                at java.util.concurrent.Semaphore.acquire(Semaphore.java:286)

                at org.hornetq.core.journal.impl.TimedBuffer$CheckTimer.run(TimedBuffer.java:429)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-4 (HornetQ-server-HornetQServerImpl::ReplicatedPagedFailoverTest/liveServer-77892085)" prio=10 tid=0x00007f07e0980800 nid=0x7dec waiting on condition [0x00007f0856085000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff440070> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-3 (HornetQ-server-HornetQServerImpl::ReplicatedPagedFailoverTest/liveServer-77892085)" prio=10 tid=0x00007f07e0dbd000 nid=0x7deb waiting on condition [0x00007f075f2f9000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff440070> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-2 (HornetQ-server-HornetQServerImpl::ReplicatedPagedFailoverTest/liveServer-77892085)" prio=10 tid=0x00007f07e020f800 nid=0x7dea waiting on condition [0x00007f075fdfc000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff440070> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-1 (HornetQ-server-HornetQServerImpl::ReplicatedPagedFailoverTest/liveServer-77892085)" prio=10 tid=0x00007f07e0d54800 nid=0x7de8 waiting on condition [0x00007f082d3d3000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff440070> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-0 (HornetQ-server-HornetQServerImpl::ReplicatedPagedFailoverTest/liveServer-77892085)" prio=10 tid=0x00007f07e0917000 nid=0x7dd8 waiting on condition [0x00007f0855c81000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff440070> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-0 (HornetQ-scheduled-threads-2110692774)" prio=10 tid=0x00007f07e0e00800 nid=0x7dd7 waiting on condition [0x00007f085496e000]

         java.lang.Thread.State: TIMED_WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff468128> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)

                at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)

                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)

                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

      "hornetq-expiry-reaper-thread" prio=10 tid=0x00007f075988a800 nid=0x7dd6 waiting on condition [0x00007f085557a000]

         java.lang.Thread.State: TIMED_WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff478a20> (a java.util.concurrent.CountDownLatch$Sync)

                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)

                at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253)

                at org.hornetq.core.postoffice.impl.PostOfficeImpl$Reaper.run(PostOfficeImpl.java:1305)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-4 (HornetQ-client-global-scheduled-threads-803215332)" daemon prio=10 tid=0x00007f07e0db5000 nid=0x7dc2 waiting on condition [0x00007f082cdcd000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff4701e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)

                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)

                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-4 (HornetQ-client-global-threads-1539842247)" daemon prio=10 tid=0x00007f07e0d6d800 nid=0x7db5 waiting on condition [0x00007f0855e83000]

         java.lang.Thread.State: TIMED_WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff468328> (a java.util.concurrent.CountDownLatch$Sync)

                at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)

                at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:253)

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:1031)

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.reconnectSessions(ClientSessionFactoryImpl.java:945)

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:669)

                - locked <0x00000007ff440af8> (a java.lang.Object)

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:547)

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$000(ClientSessionFactoryImpl.java:81)

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1603)

                at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:573)

                at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:329)

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl$CloseRunnable.run(ClientSessionFactoryImpl.java:1562)

                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:99)

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-3 (HornetQ-client-global-scheduled-threads-803215332)" daemon prio=10 tid=0x00007f07e033f800 nid=0x7d9e waiting on condition [0x00007f082c8c8000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff4701e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)

                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)

                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-2 (HornetQ-client-global-scheduled-threads-803215332)" daemon prio=10 tid=0x00007f08686fc800 nid=0x7d8e waiting on condition [0x00007f0855b80000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff4701e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)

                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)

                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-1 (HornetQ-client-global-scheduled-threads-803215332)" daemon prio=10 tid=0x00007f07e0963800 nid=0x7d87 waiting on condition [0x00007f082d4d4000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff4701e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)

                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)

                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-0 (HornetQ-client-global-scheduled-threads-803215332)" daemon prio=10 tid=0x00007f07e03be800 nid=0x7d7e waiting on condition [0x00007f0854166000]

         java.lang.Thread.State: WAITING (parking)

                at sun.misc.Unsafe.park(Native Method)

                - parking to wait for  <0x00000007ff4701e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

                at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

                at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)

                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)

                at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)

                at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-0 (HornetQ-AIO-poller-pool1845305042-1447645537)" daemon prio=10 tid=0x00007f08684f2000 nid=0x4ff1 runnable [0x00007f0854e73000]

         java.lang.Thread.State: RUNNABLE

                at org.hornetq.core.asyncio.impl.AsynchronousFileImpl.internalPollEvents(Native Method)

                at org.hornetq.core.asyncio.impl.AsynchronousFileImpl.pollEvents(AsynchronousFileImpl.java:599)

                at org.hornetq.core.asyncio.impl.AsynchronousFileImpl.access$400(AsynchronousFileImpl.java:45)

                at org.hornetq.core.asyncio.impl.AsynchronousFileImpl$PollerRunnable.run(AsynchronousFileImpl.java:822)

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-0 (HornetQ-AIO-poller-pool1816169064-1229658869)" daemon prio=10 tid=0x00007f0868717800 nid=0x3cc3 runnable [0x00007f075cbd2000]

         java.lang.Thread.State: RUNNABLE

                at org.hornetq.core.asyncio.impl.AsynchronousFileImpl.internalPollEvents(Native Method)

                at org.hornetq.core.asyncio.impl.AsynchronousFileImpl.pollEvents(AsynchronousFileImpl.java:599)

                at org.hornetq.core.asyncio.impl.AsynchronousFileImpl.access$400(AsynchronousFileImpl.java:45)

                at org.hornetq.core.asyncio.impl.AsynchronousFileImpl$PollerRunnable.run(AsynchronousFileImpl.java:822)

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                at java.lang.Thread.run(Thread.java:662)

      "Thread-0 (HornetQ-AIO-poller-pool679943859-589245674)" daemon prio=10 tid=0x00007f08686f8800 nid=0x2e72 runnable [0x00007f075f1f8000]

         java.lang.Thread.State: RUNNABLE

                at org.hornetq.core.asyncio.impl.AsynchronousFileImpl.internalPollEvents(Native Method)

                at org.hornetq.core.asyncio.impl.AsynchronousFileImpl.pollEvents(AsynchronousFileImpl.java:599)

                at org.hornetq.core.asyncio.impl.AsynchronousFileImpl.access$400(AsynchronousFileImpl.java:45)

                at org.hornetq.core.asyncio.impl.AsynchronousFileImpl$PollerRunnable.run(AsynchronousFileImpl.java:822)

                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                at java.lang.Thread.run(Thread.java:662)

      "Low Memory Detector" daemon prio=10 tid=0x00007f08680c0000 nid=0x54a runnable [0x0000000000000000]

         java.lang.Thread.State: RUNNABLE

      "C2 CompilerThread1" daemon prio=10 tid=0x00007f08680bd800 nid=0x549 waiting on condition [0x0000000000000000]

         java.lang.Thread.State: RUNNABLE

      "C2 CompilerThread0" daemon prio=10 tid=0x00007f08680bb000 nid=0x548 waiting on condition [0x0000000000000000]

         java.lang.Thread.State: RUNNABLE

      "Signal Dispatcher" daemon prio=10 tid=0x00007f08680b8800 nid=0x547 waiting on condition [0x0000000000000000]

         java.lang.Thread.State: RUNNABLE

      "Finalizer" daemon prio=10 tid=0x00007f086809c000 nid=0x546 in Object.wait() [0x00007f0856f57000]

         java.lang.Thread.State: WAITING (on object monitor)

                at java.lang.Object.wait(Native Method)

                at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

                - locked <0x0000000744296b08> (a java.lang.ref.ReferenceQueue$Lock)

                at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

                at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

      "Reference Handler" daemon prio=10 tid=0x00007f086809a000 nid=0x545 in Object.wait() [0x00007f0857058000]

         java.lang.Thread.State: WAITING (on object monitor)

                at java.lang.Object.wait(Native Method)

                at java.lang.Object.wait(Object.java:485)

                at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

                - locked <0x0000000744296ac8> (a java.lang.ref.Reference$Lock)

      "main" prio=10 tid=0x00007f0868007000 nid=0x53b waiting for monitor entry [0x00007f086c632000]

         java.lang.Thread.State: BLOCKED (on object monitor)

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:749)

                - waiting to lock <0x00000007ff440af8> (a java.lang.Object)

                - locked <0x00000007ff440b08> (a java.lang.Object)

                at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:343)

                at org.hornetq.tests.integration.cluster.failover.FailoverTest.createSession(FailoverTest.java:89)

                at org.hornetq.tests.integration.cluster.failover.FailoverTest.simpleReplication(FailoverTest.java:715)

                at org.hornetq.tests.integration.cluster.failover.FailoverTest.testFailBack(FailoverTest.java:561)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                at java.lang.reflect.Method.invoke(Method.java:597)

                at junit.framework.TestCase.runTest(TestCase.java:164)

                at junit.framework.TestCase.runBare(TestCase.java:130)

                at junit.framework.TestResult$1.protect(TestResult.java:106)

                at junit.framework.TestResult.runProtected(TestResult.java:124)

                at junit.framework.TestResult.run(TestResult.java:109)

                at junit.framework.TestCase.run(TestCase.java:120)

                at junit.framework.TestSuite.runTest(TestSuite.java:230)

                at junit.framework.TestSuite.run(TestSuite.java:225)

                at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                at java.lang.reflect.Method.invoke(Method.java:597)

                at org.apache.maven.surefire.junit.JUnitTestSet.execute(JUnitTestSet.java:95)

                at org.apache.maven.surefire.junit.JUnit3Provider.executeTestSet(JUnit3Provider.java:121)

                at org.apache.maven.surefire.junit.JUnit3Provider.invoke(JUnit3Provider.java:98)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

                at java.lang.reflect.Method.invoke(Method.java:597)

                at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)

                at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)

                at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)

                at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)

                at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)

      "VM Thread" prio=10 tid=0x00007f0868093800 nid=0x544 runnable

      "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f086801a000 nid=0x53c runnable

      "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f086801c000 nid=0x53d runnable

      "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f086801d800 nid=0x53e runnable

      "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f086801f800 nid=0x53f runnable

      "GC task thread#4 (ParallelGC)" prio=10 tid=0x00007f0868021000 nid=0x540 runnable

      "GC task thread#5 (ParallelGC)" prio=10 tid=0x00007f0868023000 nid=0x541 runnable

      "GC task thread#6 (ParallelGC)" prio=10 tid=0x00007f0868025000 nid=0x542 runnable

      "GC task thread#7 (ParallelGC)" prio=10 tid=0x00007f0868026800 nid=0x543 runnable

      "VM Periodic Task Thread" prio=10 tid=0x00007f08680ca800 nid=0x54b waiting on condition

      JNI global references: 1611

      Heap

      PSYoungGen      total 1014592K, used 85657K [0x00000007c1600000, 0x0000000800000000, 0x0000000800000000)

        eden space 1004288K, 8% used [0x00000007c1600000,0x00000007c675ee68,0x00000007feac0000)

        from space 10304K, 22% used [0x00000007ff440000,0x00000007ff687830,0x00000007ffe50000)

        to   space 9728K, 0% used [0x00000007feac0000,0x00000007feac0000,0x00000007ff440000)

      PSOldGen        total 211200K, used 14434K [0x0000000744200000, 0x0000000751040000, 0x00000007c1600000)

        object space 211200K, 6% used [0x0000000744200000,0x0000000745018a10,0x0000000751040000)

      PSPermGen       total 25024K, used 24757K [0x000000073f000000, 0x0000000740870000, 0x0000000744200000)

        object space 25024K, 98% used [0x000000073f000000,0x000000074082d7b8,0x0000000740870000)

      • 1039 Views
      • Tags:
        • 1. Re: Thread dump org.hornetq.tests.integration.cluster.failover.ReplicatedPagedFailoverTest::#test testFailBack
          clebert.suconic
          clebert.suconic Aug 13, 2012 4:56 PM (in response to clebert.suconic)

          The issue here is that if you stop the server as soon as the announcement is done by the backup, (making it an instantenous fail-back) you may endup doing it so fast that the clients weren't informed about the new server, and they keep retrying on the old backup.

           

           

          I could easily reproduce this by doing a simple loop on testFailBack (on any Netty test, e.g NettyReplicatedFailoverTest).

           

          testLoop() throws Throwable

          {

              for (int i = 0 ; i < 100; i++)

             {

               System.out.println("Test #" + i);

                testFailBack(); tearDown(); setUp();

             }

          }

           

          I can think of two possible fixes:

          - Place a sleep back

          - Wait the clients to get notified somehow. (Perhaps by changing where the server is notified)

           

           

          Follows a temporary hack that I'm placing to my branch so I can run tests without blocking them:

           

           

          diff --git hornetq-core/src/main/java/org/hornetq/core/server/impl/HornetQServerImpl.java hornetq-core/src/main/java/org/hornetq/core/server/impl/HornetQServerImpl.java

          index aebca78..c9fbb5f 100644

          --- hornetq-core/src/main/java/org/hornetq/core/server/impl/HornetQServerImpl.java

          +++ hornetq-core/src/main/java/org/hornetq/core/server/impl/HornetQServerImpl.java

          @@ -2642,6 +2642,8 @@ public class HornetQServerImpl implements HornetQServer

                                clusterConnection.addClusterTopologyListener(listener);

                                if (listener.waitForBackup())

                                {

          +                        // Hack to temporarily fix the test

          +                        Thread.sleep(1000);

                                   stop(true);

                                }

                                else

            • Actions