Backup cannot startup when Live server is down ( version 2.3_final with Data Replication )

[gojoke]

2 servers are configured as a Live - Backup Groups.

The hornetq version : 2.3_final

I do the test as below.

1. Started Live server successfully.

2. Started Backup server successfully. And the [backup announced ] is OK.

    The journal file is aslo synchronized with the live server.

3. Stop the live server.

4  Error occured at Backup server and the backup server is stopped.

    See the detailed infomation at the log attached below. 

[aaa@hornetq_backup bin]$ ./run.sh
***********************************************************************************
java  -XX:+UseParallelGC -XX:+AggressiveOpts -XX:+UseFastAccessorMethods -Xms512M -Xmx1024M -Dhornetq.config.dir=/home/aaa/hornetq-2.3.0.Final_Backup/config/stand-alone/replicated -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Dlogging.configuration=file:///home/lyonmainte/hornetq-2.3.0.Final_Backup/config/stand-alone/replicated/logging.properties -Djava.library.path=./lib/linux-i686:./lib/linux-x86_64 -classpath ../lib/netty.jar:../lib/jnpserver.jar:../lib/jnp-client.jar:../lib/jboss-mc.jar:../lib/jboss-jms-api.jar:../lib/hornetq-twitter-integration.jar:../lib/hornetq-spring-integration.jar:../lib/hornetq-service-sar.jar:../lib/hornetq-server.jar:../lib/hornetq-rest.jar:../lib/hornetq-ra.jar:../lib/hornetq-journal.jar:../lib/hornetq-jms-server.jar:../lib/hornetq-jms-client.jar:../lib/hornetq-jboss-as-integration.jar:../lib/hornetq-core-client.jar:../lib/hornetq-commons.jar:../lib/hornetq-bootstrap.jar:/home/lyonmainte/hornetq-2.3.0.Final_Backup/config/stand-alone/replicated:../schemas/ org.hornetq.integration.bootstrap.HornetQBootstrapServer hornetq-beans.xml
***********************************************************************************
17:03:47,172 INFO  [org.hornetq.integration.bootstrap] HQ101000: Starting HornetQ Server
17:03:49,183 INFO  [org.hornetq.core.server] HQ221000: backup server is starting with configuration HornetQ Configuration (clustered=true,backup=true,sharedStore=false,journalDirectory=../data/journal,bindingsDirectory=../data/bindings,largeMessagesDirectory=../data/large-messages,pagingDirectory=../data/paging)
17:03:49,207 WARN  [org.hornetq.core.server] HQ222162: Moving data directory ../data/bindings to ../data/bindings2
17:03:49,208 WARN  [org.hornetq.core.server] HQ222162: Moving data directory ../data/journal to ../data/journal2
17:03:49,209 WARN  [org.hornetq.core.server] HQ222162: Moving data directory ../data/paging to ../data/paging2
17:03:49,211 WARN  [org.hornetq.core.server] HQ222162: Moving data directory ../data/large-messages to ../data/large-messages2
17:03:49,429 INFO  [org.hornetq.core.server] HQ221012: Using AIO Journal
17:03:49,456 WARN  [org.hornetq.core.server] HQ222007: Security risk! HornetQ is running with the default cluster admin user and default password. Please see the HornetQ user guide, cluster chapter, for instructions on how to change this.
17:04:08,037 INFO  [org.hornetq.core.server] HQ221109: HornetQ Backup Server version 2.3.0.SNAPSHOT (colonizer, 123) [null] started, waiting live to fail before it gets active
17:04:09,872 INFO  [org.hornetq.core.server] HQ221024: Backup server HornetQServerImpl::serverUUID=a4c5c230-d2a0-11e2-934b-93316aeaeb81 is synchronized with live-server.
17:04:12,911 INFO  [org.hornetq.core.server] HQ221031: backup announced
17:04:35,220 WARN  [org.hornetq.core.client] HQ212037: Connection failure has been detected: HQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
17:04:35,220 WARN  [org.hornetq.core.client] HQ212037: Connection failure has been detected: HQ119015: The connection was disconnected because of server shutdown [code=DISCONNECTED]
HornetQException[errorType=IO_ERROR message=null]
        at org.hornetq.core.journal.impl.NIOSequentialFile.open(NIOSequentialFile.java:109)
        at org.hornetq.core.journal.impl.NIOSequentialFile.open(NIOSequentialFile.java:94)
        at org.hornetq.core.paging.impl.PagingStoreImpl.createPage(PagingStoreImpl.java:591)
        at org.hornetq.core.replication.ReplicationEndpoint.newPage(ReplicationEndpoint.java:871)
        at org.hornetq.core.replication.ReplicationEndpoint.getPage(ReplicationEndpoint.java:852)
        at org.hornetq.core.replication.ReplicationEndpoint.handlePageEvent(ReplicationEndpoint.java:795)
        at org.hornetq.core.replication.ReplicationEndpoint.handlePacket(ReplicationEndpoint.java:197)
        at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:631)
        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547)
        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523)
        at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1656)
        at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:72)
        at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281)
        at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169)
        at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.oio.OioWorker.process(OioWorker.java:71)
        at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73)
        at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51)
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.nio.channels.ClosedByInterruptException
        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184)
        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:304)
        at org.hornetq.core.journal.impl.NIOSequentialFile.open(NIOSequentialFile.java:105)
        ... 31 more
Exception happened, criticalErrorListener = org.hornetq.core.server.impl.HornetQServerImpl$ShutdownOnCriticalErrorListener@4d74f02c
Critical ErrorListener, failedAlready=false
17:04:35,262 WARN  [org.hornetq.core.server] HQ222010: Critical IO Error, shutting down the server. file=NIOSequentialFile ../data/paging/3a837ad5-d2a8-11e2-8d1e-2fceae3b5bf0/000000002.page, message=null: HornetQException[errorType=IO_ERROR message=null]
        at org.hornetq.core.journal.impl.NIOSequentialFile.open(NIOSequentialFile.java:109) [hornetq-journal.jar:]
        at org.hornetq.core.journal.impl.NIOSequentialFile.open(NIOSequentialFile.java:94) [hornetq-journal.jar:]
        at org.hornetq.core.paging.impl.PagingStoreImpl.createPage(PagingStoreImpl.java:591) [hornetq-server.jar:]
        at org.hornetq.core.replication.ReplicationEndpoint.newPage(ReplicationEndpoint.java:871) [hornetq-server.jar:]
        at org.hornetq.core.replication.ReplicationEndpoint.getPage(ReplicationEndpoint.java:852) [hornetq-server.jar:]
        at org.hornetq.core.replication.ReplicationEndpoint.handlePageEvent(ReplicationEndpoint.java:795) [hornetq-server.jar:]
        at org.hornetq.core.replication.ReplicationEndpoint.handlePacket(ReplicationEndpoint.java:197) [hornetq-server.jar:]
        at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:631) [hornetq-core-client.jar:]
        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547) [hornetq-core-client.jar:]
        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523) [hornetq-core-client.jar:]
        at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1656) [hornetq-core-client.jar:]
        at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:72) [hornetq-core-client.jar:]
        at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty.jar:]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty.jar:]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) [netty.jar:]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) [netty.jar:]
        at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169) [hornetq-core-client.jar:]
        at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134) [hornetq-core-client.jar:]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty.jar:]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty.jar:]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) [netty.jar:]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty.jar:]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty.jar:]
        at org.jboss.netty.channel.socket.oio.OioWorker.process(OioWorker.java:71) [netty.jar:]
        at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73) [netty.jar:]
        at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51) [netty.jar:]
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty.jar:]
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty.jar:]
        at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175) [netty.jar:]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_43]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_43]
        at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]
Caused by: java.nio.channels.ClosedByInterruptException
        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184) [rt.jar:1.6.0_43]
        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:304) [rt.jar:1.6.0_43]
        at org.hornetq.core.journal.impl.NIOSequentialFile.open(NIOSequentialFile.java:105) [hornetq-journal.jar:]
        ... 31 more

17:04:35,291 WARN  [org.hornetq.core.server] HQ222086: error handling packet ReplicationPageEventMessage(channel=2, isDelete=false, storeName=jms.queue.ExpiryQueue, pageNumber=2) for replication: java.nio.channels.ClosedByInterruptException
        at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:184) [rt.jar:1.6.0_43]
        at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:304) [rt.jar:1.6.0_43]
        at org.hornetq.core.journal.impl.NIOSequentialFile.open(NIOSequentialFile.java:105) [hornetq-journal.jar:]
        at org.hornetq.core.journal.impl.NIOSequentialFile.open(NIOSequentialFile.java:94) [hornetq-journal.jar:]
        at org.hornetq.core.paging.impl.PagingStoreImpl.createPage(PagingStoreImpl.java:591) [hornetq-server.jar:]
        at org.hornetq.core.replication.ReplicationEndpoint.newPage(ReplicationEndpoint.java:871) [hornetq-server.jar:]
        at org.hornetq.core.replication.ReplicationEndpoint.getPage(ReplicationEndpoint.java:852) [hornetq-server.jar:]
        at org.hornetq.core.replication.ReplicationEndpoint.handlePageEvent(ReplicationEndpoint.java:795) [hornetq-server.jar:]
        at org.hornetq.core.replication.ReplicationEndpoint.handlePacket(ReplicationEndpoint.java:197) [hornetq-server.jar:]
        at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:631) [hornetq-core-client.jar:]
        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547) [hornetq-core-client.jar:]
        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523) [hornetq-core-client.jar:]
        at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1656) [hornetq-core-client.jar:]
        at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:72) [hornetq-core-client.jar:]
        at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty.jar:]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty.jar:]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) [netty.jar:]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) [netty.jar:]
        at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169) [hornetq-core-client.jar:]
        at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134) [hornetq-core-client.jar:]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty.jar:]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty.jar:]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) [netty.jar:]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty.jar:]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty.jar:]
        at org.jboss.netty.channel.socket.oio.OioWorker.process(OioWorker.java:71) [netty.jar:]
        at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73) [netty.jar:]
        at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51) [netty.jar:]
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty.jar:]
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty.jar:]
        at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175) [netty.jar:]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_43]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_43]
        at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]

17:04:35,316 ERROR [org.hornetq.core.client] HQ214013: Failed to decode packet: org.hornetq.api.core.HornetQInterruptedException: java.lang.InterruptedException
        at org.hornetq.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:244) [hornetq-core-client.jar:]
        at org.hornetq.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:267) [hornetq-core-client.jar:]
        at org.hornetq.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:194) [hornetq-core-client.jar:]
        at org.hornetq.core.replication.ReplicationEndpoint.handlePacket(ReplicationEndpoint.java:243) [hornetq-server.jar:]
        at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:631) [hornetq-core-client.jar:]
        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547) [hornetq-core-client.jar:]
        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523) [hornetq-core-client.jar:]
        at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1656) [hornetq-core-client.jar:]
        at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:72) [hornetq-core-client.jar:]
        at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88) [netty.jar:]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty.jar:]
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787) [netty.jar:]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:281) [netty.jar:]
        at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169) [hornetq-core-client.jar:]
        at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134) [hornetq-core-client.jar:]
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) [netty.jar:]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560) [netty.jar:]
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555) [netty.jar:]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) [netty.jar:]
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) [netty.jar:]
        at org.jboss.netty.channel.socket.oio.OioWorker.process(OioWorker.java:71) [netty.jar:]
        at org.jboss.netty.channel.socket.oio.AbstractOioWorker.run(AbstractOioWorker.java:73) [netty.jar:]
        at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:51) [netty.jar:]
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [netty.jar:]
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [netty.jar:]
        at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175) [netty.jar:]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_43]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_43]
        at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]
Caused by: java.lang.InterruptedException
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1279) [rt.jar:1.6.0_43]
        at java.util.concurrent.Semaphore.acquire(Semaphore.java:286) [rt.jar:1.6.0_43]
        at org.hornetq.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:175) [hornetq-core-client.jar:]
        ... 28 more

17:04:35,343 ERROR [org.hornetq.core.client] HQ214002: Failed to execute failure listener: java.util.concurrent.RejectedExecutionException
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1774) [rt.jar:1.6.0_43]
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:768) [rt.jar:1.6.0_43]
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:656) [rt.jar:1.6.0_43]
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:78) [rt.jar:1.6.0_43]
        at org.hornetq.core.server.impl.QuorumManager.isLiveDown(QuorumManager.java:146) [hornetq-server.jar:]
        at org.hornetq.core.server.impl.QuorumManager.decideOnAction(QuorumManager.java:298) [hornetq-server.jar:]
        at org.hornetq.core.server.impl.QuorumManager.connectionFailed(QuorumManager.java:286) [hornetq-server.jar:]
        at org.hornetq.core.client.impl.ClientSessionFactoryImpl.callSessionFailureListeners(ClientSessionFactoryImpl.java:961) [hornetq-core-client.jar:]
        at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:740) [hornetq-core-client.jar:]
        at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:580) [hornetq-core-client.jar:]
        at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionException(ClientSessionFactoryImpl.java:434) [hornetq-core-client.jar:]
        at org.hornetq.core.remoting.impl.netty.NettyConnector$Listener$2.run(NettyConnector.java:895) [hornetq-core-client.jar:]
        at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106) [hornetq-core-client.jar:]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) [rt.jar:1.6.0_43]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) [rt.jar:1.6.0_43]
        at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_43]

17:04:36,348 INFO  [org.hornetq.core.server] HQ221002: HornetQ Server version 2.3.0.SNAPSHOT (colonizer, 123) [a4c5c230-d2a0-11e2-934b-93316aeaeb81] stopped

[borges]

Hi,

can you tell us more about how you are running this? I would like to see the configuration of both servers and to know how you are starting them and stopping the live.

From the trace, we can see the backup getting stopped by getting an Interrupt signal which interrupts an IO thread, this is an IOError, and stuff like this will always shutdown a server.
Notice the IO exception (right at the start of your logs) is actually caused by 
({code}Caused by: java.nio.channels.ClosedByInterruptException{code}). An interrupt signal is also getting the Netty thread channel out of its write. 
The thread that would be taking the necessary actions for fail-over is right at the end, and it fails to execute as the server is already shutting down, I'm assuming that this is due to the IOError right at the beginning)
So the fail-over procedure has not even started and the (backup) server is already getting stopped. From the looks of it, it looks like you are sending an interrupt to the backup while stopping the live.

[gojoke]

Thank you for your reply.

The attached files are configuations for both server.

[gojoke]

The live server 's trace as below

There is a warning :

On ManagementService stop, there are 1 unexpected registered MBeans: [core.acceptor.netty]

You mean is sigal sending to backup server and make it stop?

--------------------------------------------------------------- trace ------------------------------------------------------------

[aaa@hornetq_live bin]$ ./run.sh
***********************************************************************************
java  -XX:+UseParallelGC -XX:+AggressiveOpts -XX:+UseFastAccessorMethods -Xms512M -Xmx1024M -Dhornetq.config.dir=/home/aaa/hornetq-2.3.0.Final_Live/config/stand-alone/replicated -Djava.util.logging.manager=org.jboss.logmanager.LogManager -Dlogging.configuration=file:///home/lyonmainte/hornetq-2.3.0.Final_Live/config/stand-alone/replicated/logging.properties -Djava.library.path=./lib/linux-i686:./lib/linux-x86_64 -classpath ../lib/netty.jar:../lib/jnpserver.jar:../lib/jnp-client.jar:../lib/jboss-mc.jar:../lib/jboss-jms-api.jar:../lib/hornetq-twitter-integration.jar:../lib/hornetq-spring-integration.jar:../lib/hornetq-service-sar.jar:../lib/hornetq-server.jar:../lib/hornetq-rest.jar:../lib/hornetq-ra.jar:../lib/hornetq-journal.jar:../lib/hornetq-jms-server.jar:../lib/hornetq-jms-client.jar:../lib/hornetq-jboss-as-integration.jar:../lib/hornetq-core-client.jar:../lib/hornetq-commons.jar:../lib/hornetq-bootstrap.jar:/home/lyonmainte/hornetq-2.3.0.Final_Live/config/stand-alone/replicated:../schemas/ org.hornetq.integration.bootstrap.HornetQBootstrapServer hornetq-beans.xml
***********************************************************************************
11:09:25,168 INFO  [org.hornetq.integration.bootstrap] HQ101000: Starting HornetQ Server
11:09:27,185 INFO  [org.hornetq.core.server] HQ221000: live server is starting with configuration HornetQ Configuration (clustered=true,backup=false,sharedStore=false,journalDirectory=../data/journal,bindingsDirectory=../data/bindings,largeMessagesDirectory=../data/large-messages,pagingDirectory=../data/paging)
11:09:27,380 INFO  [org.hornetq.core.server] HQ221012: Using AIO Journal
11:09:27,405 WARN  [org.hornetq.core.server] HQ222007: Security risk! HornetQ is running with the default cluster admin user and default password. Please see the HornetQ user guide, cluster chapter, for instructions on how to change this.
11:09:29,227 INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.DLQ
11:09:29,262 INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.queue.ExpiryQueue
11:09:29,273 INFO  [org.hornetq.core.server] HQ221003: trying to deploy queue jms.topic.TempleTopic
11:09:29,433 INFO  [org.hornetq.core.server] HQ221020: Started Netty Acceptor version 3.6.2.Final-c0d783c 192.168.0.7:5445 for CORE protocol
11:09:29,454 INFO  [org.hornetq.core.server] HQ221007: Server is now live
11:09:29,455 INFO  [org.hornetq.core.server] HQ221001: HornetQ Server version 2.3.0.SNAPSHOT (colonizer, 123) [a4c5c230-d2a0-11e2-934b-93316aeaeb81]

11:12:04,805 INFO  [org.hornetq.core.server] HQ221025: Replication: sending JournalFileImpl: (hornetq-data-4.hq id = 54, recordID = 54) (size=10,485,760) to backup. AIOSequentialFile:/home/lyonmainte/hornetq-2.3.0.Final_Live/bin/../data/journal/hornetq-data-4.hq
11:12:05,094 INFO  [org.hornetq.core.server] HQ221025: Replication: sending JournalFileImpl: (hornetq-bindings-16.bindings id = 1, recordID = 1) (size=1,048,576) to backup. NIOSequentialFile ../data/bindings/hornetq-bindings-16.bindings
11:12:05,114 INFO  [org.hornetq.core.server] HQ221025: Replication: sending JournalFileImpl: (hornetq-bindings-17.bindings id = 2, recordID = 2) (size=1,048,576) to backup. NIOSequentialFile ../data/bindings/hornetq-bindings-17.bindings
11:12:05,129 INFO  [org.hornetq.core.server] HQ221025: Replication: sending JournalFileImpl: (hornetq-bindings-2.bindings id = 14, recordID = 14) (size=1,048,576) to backup. NIOSequentialFile ../data/bindings/hornetq-bindings-2.bindings
^C11:12:47,531 INFO  [org.hornetq.integration.bootstrap] HQ101001: Stopping HornetQ Server
11:12:47,583 WARN  [org.hornetq.core.server] HQ222113: On ManagementService stop, there are 1 unexpected registered MBeans: [core.acceptor.netty]
11:12:47,624 INFO  [org.hornetq.core.server] HQ221002: HornetQ Server version 2.3.0.SNAPSHOT (colonizer, 123) [a4c5c230-d2a0-11e2-934b-93316aeaeb81] stopped

[viet.bui.mit]

Hi there,

I'm having exactly the same problem. Does anyone have a solution yet?

A bit of background information, I'm evaluating HornetQ as the selected messaging system. We are attempting to deploy this into AWS with clustering enabled. As AWS does not support shared store, data replication must be used.

Hope to hear from you guys soon.

Regards.