4 Replies Latest reply on Jul 3, 2013 12:59 AM by Viet Bui

    Backup cannot startup when Live server is down ( version 2.3_final with Data Replication )

    gojoke Newbie

      2 servers are configured as a Live - Backup Groups.

      The hornetq version : 2.3_final

       

      I do the test as below.

      1. Started Live server successfully.

      2. Started Backup server successfully. And the [backup announced ] is OK.

          The journal file is aslo synchronized with the live server.

      3. Stop the live server.

      4  Error occured at Backup server and the backup server is stopped.

          See the detailed infomation at the log attached below. 

       

       

      [aaa@hornetq_backup bin]$ ./run.sh
      ***********************************************************************************
      java  -XX:+UseParallelGC -XX:+AggressiveOpts -XX:+UseFastAccessorMethods -Xms512M -Xmx1024M -Dhornetq.config.dir=/home/aaa/hornetq-2.3.0.Final_Backup/config/stand-alone/replicated -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Dlogging.configuration=file:///home/lyonmainte/hornetq-2.3.0.Final_Backup/config/stand-alone/replicated/logging.properties -Djava.library.path=./lib/linux-i686:./lib/linux-x86_64 -classpath ../lib/netty.jar:../lib/jnpserver.jar:../lib/jnp-client.jar:../lib/jboss-mc.jar:../lib/jboss-jms-api.jar:../lib/hornetq-twitter-integration.jar:../lib/hornetq-spring-integration.jar:../lib/hornetq-service-sar.jar:../lib/hornetq-server.jar:../lib/hornetq-rest.jar:../lib/hornetq-ra.jar:../lib/hornetq-journal.jar:../lib/hornetq-jms-server.jar:../lib/hornetq-jms-client.jar:../lib/hornetq-jboss-as-integration.jar:../lib/hornetq-core-client.jar:../lib/hornetq-commons.jar:../lib/hornetq-bootstrap.jar:/home/lyonmainte/hornetq-2.3.0.Final_Backup/config/stand-alone/replicated:../schemas/ org.hornetq.integration.bootstrap.HornetQBootstrapServer hornetq-beans.xml
      ***********************************************************************************
      17:03:47,172 INFO  [org.hornetq.integration.bootstrap] HQ101000: Starting HornetQ Server
      17:03:49,183 INFO  [org.hornetq.core.server] HQ221000: backup server is starting with configuration HornetQ Configuration (clustered=true,backup=true,sharedStore=false,journalDirectory=../data/journal,bindingsDirectory=../data/bindings,largeMessagesDirectory=../data/large-messages,pagingDirectory=../data/paging)
      17:03:49,207 WARN  [org.hornetq.core.server] HQ222162: Moving data directory ../data/bindings to ../data/bindings2
      17:03:49,208 WARN  [org.hornetq.core.server] HQ222162: Moving data directory ../data/journal to ../data/journal2
      17:03:49,209 WARN  [org.hornetq.core.server] HQ222162: Moving data directory ../data/paging to ../data/paging2
      17:03:49,211 WARN  [org.hornetq.core.server] HQ222162: Moving data directory ../data/large-messages to ../data/large-messages2
      17:03:49,429 INFO  [org.hornetq.core.server] HQ221012: Using AIO Journal
      17:03:49,456 WARN  [org.hornetq.core.server] HQ222007: Security risk! HornetQ is running with the default cluster admin user and default password. Please see the HornetQ user guide, cluster chapter, for instructions on how to change this.
      17:04:08,037 INFO  [org.hornetq.core.server] HQ221109: HornetQ Backup Server version 2.3.0.SNAPSHOT (colonizer, 123) [null] started, waiting live to fail before it gets active
      17:04:09,872 INFO  [org.hornetq.core.server] HQ221024: Backup server HornetQServerImpl::serverUUID=a4c5c230-d2a0-11e2-934b-93316aeaeb81 is synchronized with live-server.
      17:04:12,911 INFO  [org.hornetq.core.server] HQ221031: backup announced
      17:04:35,220 WARN  [org.hornetq.core.client] HQ212037: Connection failure has been detected: HQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
      17:04:35,220 WARN  [org.hornetq.core.client] HQ212037: Connection failure has been detected: HQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
      HornetQException[errorType=IO_ERROR message=null]
              at org.hornetq.core.journal.impl.NIOSequentialFile.open(NIOSequentialFile.java:109)
              at org.hornetq.core.journal.impl.NIOSequentialFile.open(NIOSequentialFile.java:94)
              at org.hornetq.core.paging.impl.PagingStoreImpl.createPage(PagingStoreImpl.java:591)
              at org.hornetq.core.replication.ReplicationEndpoint.newPage(ReplicationEndpoint.java:871)
              at org.hornetq.core.replication.ReplicationEndpoint.getPage(ReplicationEndpoint.java:852)
              at org.hornetq.core.replication.ReplicationEndpoint.handlePageEvent(ReplicationEndpoint.java:795)
              at org.hornetq.core.replication.ReplicationEndpoint.handlePacket(ReplicationEndpoint.java:197)
              at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:631)
              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547)
              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523)
              at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1656)
              at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:72)
              at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281)
              at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169)
              at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134)
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
              at org.jboss.netty.channel.socket.oio.OioWorker.process(OioWorker.java:71)
              at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73)
              at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51)
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
              at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
              at java.lang.Thread.run(Thread.java:662)
      Caused by: java.nio.channels.ClosedByInterruptException
              at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
              at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:304)
              at org.hornetq.core.journal.impl.NIOSequentialFile.open(NIOSequentialFile.java:105)
              ... 31 more
      Exception happened, criticalErrorListener = org.hornetq.core.server.impl.HornetQServerImpl$ShutdownOnCriticalErrorListener@4d74f02c
      Critical ErrorListener, failedAlready=false
      17:04:35,262 WARN  [org.hornetq.core.server] HQ222010: Critical IO Error, shutting down the server. file=NIOSequentialFile ../data/paging/3a837ad5-d2a8-11e2-8d1e-2fceae3b5bf0/000000002.page, message=null: HornetQException[errorType=IO_ERROR message=null]
              at org.hornetq.core.journal.impl.NIOSequentialFile.open(NIOSequentialFile.java:109) [hornetq-journal.jar:]
              at org.hornetq.core.journal.impl.NIOSequentialFile.open(NIOSequentialFile.java:94) [hornetq-journal.jar:]
              at org.hornetq.core.paging.impl.PagingStoreImpl.createPage(PagingStoreImpl.java:591) [hornetq-server.jar:]
              at org.hornetq.core.replication.ReplicationEndpoint.newPage(ReplicationEndpoint.java:871) [hornetq-server.jar:]
              at org.hornetq.core.replication.ReplicationEndpoint.getPage(ReplicationEndpoint.java:852) [hornetq-server.jar:]
              at org.hornetq.core.replication.ReplicationEndpoint.handlePageEvent(ReplicationEndpoint.java:795) [hornetq-server.jar:]
              at org.hornetq.core.replication.ReplicationEndpoint.handlePacket(ReplicationEndpoint.java:197) [hornetq-server.jar:]
              at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:631) [hornetq-core-client.jar:]
              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547) [hornetq-core-client.jar:]
              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523) [hornetq-core-client.jar:]
              at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1656) [hornetq-core-client.jar:]
              at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:72) [hornetq-core-client.jar:]
              at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty.jar:]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty.jar:]
              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) [netty.jar:]
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) [netty.jar:]
              at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169) [hornetq-core-client.jar:]
              at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134) [hornetq-core-client.jar:]
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty.jar:]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty.jar:]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) [netty.jar:]
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty.jar:]
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty.jar:]
              at org.jboss.netty.channel.socket.oio.OioWorker.process(OioWorker.java:71) [netty.jar:]
              at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73) [netty.jar:]
              at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51) [netty.jar:]
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty.jar:]
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty.jar:]
              at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175) [netty.jar:]
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_43]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_43]
              at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]
      Caused by: java.nio.channels.ClosedByInterruptException
              at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184) [rt.jar:1.6.0_43]
              at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:304) [rt.jar:1.6.0_43]
              at org.hornetq.core.journal.impl.NIOSequentialFile.open(NIOSequentialFile.java:105) [hornetq-journal.jar:]
              ... 31 more

      17:04:35,291 WARN  [org.hornetq.core.server] HQ222086: error handling packet ReplicationPageEventMessage(channel=2, isDelete=false, storeName=jms.queue.ExpiryQueue, pageNumber=2) for replication: java.nio.channels.ClosedByInterruptException
              at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184) [rt.jar:1.6.0_43]
              at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:304) [rt.jar:1.6.0_43]
              at org.hornetq.core.journal.impl.NIOSequentialFile.open(NIOSequentialFile.java:105) [hornetq-journal.jar:]
              at org.hornetq.core.journal.impl.NIOSequentialFile.open(NIOSequentialFile.java:94) [hornetq-journal.jar:]
              at org.hornetq.core.paging.impl.PagingStoreImpl.createPage(PagingStoreImpl.java:591) [hornetq-server.jar:]
              at org.hornetq.core.replication.ReplicationEndpoint.newPage(ReplicationEndpoint.java:871) [hornetq-server.jar:]
              at org.hornetq.core.replication.ReplicationEndpoint.getPage(ReplicationEndpoint.java:852) [hornetq-server.jar:]
              at org.hornetq.core.replication.ReplicationEndpoint.handlePageEvent(ReplicationEndpoint.java:795) [hornetq-server.jar:]
              at org.hornetq.core.replication.ReplicationEndpoint.handlePacket(ReplicationEndpoint.java:197) [hornetq-server.jar:]
              at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:631) [hornetq-core-client.jar:]
              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547) [hornetq-core-client.jar:]
              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523) [hornetq-core-client.jar:]
              at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1656) [hornetq-core-client.jar:]
              at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:72) [hornetq-core-client.jar:]
              at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty.jar:]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty.jar:]
              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) [netty.jar:]
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) [netty.jar:]
              at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169) [hornetq-core-client.jar:]
              at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134) [hornetq-core-client.jar:]
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty.jar:]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty.jar:]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) [netty.jar:]
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty.jar:]
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty.jar:]
              at org.jboss.netty.channel.socket.oio.OioWorker.process(OioWorker.java:71) [netty.jar:]
              at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73) [netty.jar:]
              at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51) [netty.jar:]
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty.jar:]
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty.jar:]
              at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175) [netty.jar:]
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_43]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_43]
              at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]

      17:04:35,316 ERROR [org.hornetq.core.client] HQ214013: Failed to decode packet: org.hornetq.api.core.HornetQInterruptedException: java.lang.InterruptedException
              at org.hornetq.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:244) [hornetq-core-client.jar:]
              at org.hornetq.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:267) [hornetq-core-client.jar:]
              at org.hornetq.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:194) [hornetq-core-client.jar:]
              at org.hornetq.core.replication.ReplicationEndpoint.handlePacket(ReplicationEndpoint.java:243) [hornetq-server.jar:]
              at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:631) [hornetq-core-client.jar:]
              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547) [hornetq-core-client.jar:]
              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523) [hornetq-core-client.jar:]
              at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1656) [hornetq-core-client.jar:]
              at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:72) [hornetq-core-client.jar:]
              at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty.jar:]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty.jar:]
              at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) [netty.jar:]
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) [netty.jar:]
              at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169) [hornetq-core-client.jar:]
              at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134) [hornetq-core-client.jar:]
              at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty.jar:]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty.jar:]
              at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) [netty.jar:]
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty.jar:]
              at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty.jar:]
              at org.jboss.netty.channel.socket.oio.OioWorker.process(OioWorker.java:71) [netty.jar:]
              at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73) [netty.jar:]
              at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51) [netty.jar:]
              at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty.jar:]
              at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty.jar:]
              at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175) [netty.jar:]
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_43]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_43]
              at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]
      Caused by: java.lang.InterruptedException
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1279) [rt.jar:1.6.0_43]
              at java.util.concurrent.Semaphore.acquire(Semaphore.java:286) [rt.jar:1.6.0_43]
              at org.hornetq.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:175) [hornetq-core-client.jar:]
              ... 28 more

      17:04:35,343 ERROR [org.hornetq.core.client] HQ214002: Failed to execute failure listener: java.util.concurrent.RejectedExecutionException
              at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1774) [rt.jar:1.6.0_43]
              at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:768) [rt.jar:1.6.0_43]
              at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:656) [rt.jar:1.6.0_43]
              at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:78) [rt.jar:1.6.0_43]
              at org.hornetq.core.server.impl.QuorumManager.isLiveDown(QuorumManager.java:146) [hornetq-server.jar:]
              at org.hornetq.core.server.impl.QuorumManager.decideOnAction(QuorumManager.java:298) [hornetq-server.jar:]
              at org.hornetq.core.server.impl.QuorumManager.connectionFailed(QuorumManager.java:286) [hornetq-server.jar:]
              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:961) [hornetq-core-client.jar:]
              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:740) [hornetq-core-client.jar:]
              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:580) [hornetq-core-client.jar:]
              at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionException(ClientSessionFactoryImpl.java:434) [hornetq-core-client.jar:]
              at org.hornetq.core.remoting.impl.netty.NettyConnector$Listener$2.run(NettyConnector.java:895) [hornetq-core-client.jar:]
              at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106) [hornetq-core-client.jar:]
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_43]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_43]
              at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]

      17:04:36,348 INFO  [org.hornetq.core.server] HQ221002: HornetQ Server version 2.3.0.SNAPSHOT (colonizer, 123) [a4c5c230-d2a0-11e2-934b-93316aeaeb81] stopped

        • 1. Re: Backup cannot startup when Live server is down ( version 2.3_final with Data Replication )
          borges Novice

          Hi,

           

          can you tell us more about how you are running this? I would like to see the configuration of both servers and to know how you are starting them and stopping the live.

           

           

           

          From the trace, we can see the backup getting stopped by getting an Interrupt signal which interrupts an IO thread, this is an IOError, and stuff like this will always shutdown a server.

           

          Notice the IO exception (right at the start of your logs) is actually caused by

          ({code}Caused by: java.nio.channels.ClosedByInterruptException{code}). An interrupt signal is also getting the Netty thread channel out of its write.

           

          The thread that would be taking the necessary actions for fail-over is right at the end, and it fails to execute as the server is already shutting down, I'm assuming that this is due to the IOError right at the beginning)

           

          So the fail-over procedure has not even started and the (backup) server is already getting stopped. From the looks of it, it looks like you are sending an interrupt to the backup while stopping the live.

          • 2. Re: Backup cannot startup when Live server is down ( version 2.3_final with Data Replication )
            gojoke Newbie

            Thank you for your reply.

             

            The attached files are configuations for both server.

            • 3. Re: Backup cannot startup when Live server is down ( version 2.3_final with Data Replication )
              gojoke Newbie

              The live server 's trace as below

               

              There is a warning :

              On ManagementService stop, there are 1 unexpected registered MBeans: [core.acceptor.netty]

              You mean is sigal sending to backup server and make it stop?

               

               

              --------------------------------------------------------------- trace ------------------------------------------------------------

              [aaa@hornetq_live bin]$ ./run.sh
              ***********************************************************************************
              java  -XX:+UseParallelGC -XX:+AggressiveOpts -XX:+UseFastAccessorMethods -Xms512M -Xmx1024M -Dhornetq.config.dir=/home/aaa/hornetq-2.3.0.Final_Live/config/stand-alone/replicated -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Dlogging.configuration=file:///home/lyonmainte/hornetq-2.3.0.Final_Live/config/stand-alone/replicated/logging.properties -Djava.library.path=./lib/linux-i686:./lib/linux-x86_64 -classpath ../lib/netty.jar:../lib/jnpserver.jar:../lib/jnp-client.jar:../lib/jboss-mc.jar:../lib/jboss-jms-api.jar:../lib/hornetq-twitter-integration.jar:../lib/hornetq-spring-integration.jar:../lib/hornetq-service-sar.jar:../lib/hornetq-server.jar:../lib/hornetq-rest.jar:../lib/hornetq-ra.jar:../lib/hornetq-journal.jar:../lib/hornetq-jms-server.jar:../lib/hornetq-jms-client.jar:../lib/hornetq-jboss-as-integration.jar:../lib/hornetq-core-client.jar:../lib/hornetq-commons.jar:../lib/hornetq-bootstrap.jar:/home/lyonmainte/hornetq-2.3.0.Final_Live/config/stand-alone/replicated:../schemas/ org.hornetq.integration.bootstrap.HornetQBootstrapServer hornetq-beans.xml
              ***********************************************************************************
              11:09:25,168 INFO  [org.hornetq.integration.bootstrap] HQ101000: Starting HornetQ Server
              11:09:27,185 INFO  [org.hornetq.core.server] HQ221000: live server is starting with configuration HornetQ Configuration (clustered=true,backup=false,sharedStore=false,journalDirectory=../data/journal,bindingsDirectory=../data/bindings,largeMessagesDirectory=../data/large-messages,pagingDirectory=../data/paging)
              11:09:27,380 INFO  [org.hornetq.core.server] HQ221012: Using AIO Journal
              11:09:27,405 WARN  [org.hornetq.core.server] HQ222007: Security risk! HornetQ is running with the default cluster admin user and default password. Please see the HornetQ user guide, cluster chapter, for instructions on how to change this.
              11:09:29,227 INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.DLQ
              11:09:29,262 INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.ExpiryQueue
              11:09:29,273 INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.topic.TempleTopic
              11:09:29,433 INFO  [org.hornetq.core.server] HQ221020: Started Netty Acceptor version 3.6.2.Final-c0d783c 192.168.0.7:5445 for CORE protocol
              11:09:29,454 INFO  [org.hornetq.core.server] HQ221007: Server is now live
              11:09:29,455 INFO  [org.hornetq.core.server] HQ221001: HornetQ Server version 2.3.0.SNAPSHOT (colonizer, 123) [a4c5c230-d2a0-11e2-934b-93316aeaeb81]

              11:12:04,805 INFO  [org.hornetq.core.server] HQ221025: Replication: sending JournalFileImpl: (hornetq-data-4.hq id = 54, recordID = 54) (size=10,485,760) to backup. AIOSequentialFile:/home/lyonmainte/hornetq-2.3.0.Final_Live/bin/../data/journal/hornetq-data-4.hq
              11:12:05,094 INFO  [org.hornetq.core.server] HQ221025: Replication: sending JournalFileImpl: (hornetq-bindings-16.bindings id = 1, recordID = 1) (size=1,048,576) to backup. NIOSequentialFile ../data/bindings/hornetq-bindings-16.bindings
              11:12:05,114 INFO  [org.hornetq.core.server] HQ221025: Replication: sending JournalFileImpl: (hornetq-bindings-17.bindings id = 2, recordID = 2) (size=1,048,576) to backup. NIOSequentialFile ../data/bindings/hornetq-bindings-17.bindings
              11:12:05,129 INFO  [org.hornetq.core.server] HQ221025: Replication: sending JournalFileImpl: (hornetq-bindings-2.bindings id = 14, recordID = 14) (size=1,048,576) to backup. NIOSequentialFile ../data/bindings/hornetq-bindings-2.bindings
              ^C11:12:47,531 INFO  [org.hornetq.integration.bootstrap] HQ101001: Stopping HornetQ Server
              11:12:47,583 WARN  [org.hornetq.core.server] HQ222113: On ManagementService stop, there are 1 unexpected registered MBeans: [core.acceptor.netty]
              11:12:47,624 INFO  [org.hornetq.core.server] HQ221002: HornetQ Server version 2.3.0.SNAPSHOT (colonizer, 123) [a4c5c230-d2a0-11e2-934b-93316aeaeb81] stopped

               

              • 4. Re: Backup cannot startup when Live server is down ( version 2.3_final with Data Replication )
                Viet Bui Newbie

                Hi there,

                 

                I'm having exactly the same problem. Does anyone have a solution yet?

                 

                A bit of background information, I'm evaluating HornetQ as the selected messaging system. We are attempting to deploy this into AWS with clustering enabled. As AWS does not support shared store, data replication must be used.

                 

                Hope to hear from you guys soon.

                 

                Regards.