JBossDeveloper

Thanks you. I'll give it a try.

Hi,

I tried to simplify your test case using standalone hornetq servers. It shows that the fail-back works on my environment. I tried both the latest 2.3.21 and 2.3.0.CR1 and they all work as expected. Note for replication mode it is expected that the backup will stop after failback.

The periodical warnings are caused by the XA recovery manager. You need to remove the pooled-connection-factory from your backup config:

If you are using community version i strongly recommend you to upgrade hornetq to its latest. There are a lot bug fixes since CR1.

Howard

I only tested a simple version of the case. It's not easy for me to get the real test environment like yours. I didn't test the network failure but I think to a replication mode backup the behavior is the same. The only difference could be that in case of network failure and backup becomes live, if your old live is still running, you need manually shutdown it and let the clients failover to the new live server. You can restart the live after the network is restored.

The backup has to stop after fail back, because (here I quote the user manual)

"otherwise the live server has no means to know whether there was a fail-over or not, and if there was if the server that took its duties is still running or not"

So I'm curious that why you keep getting "same node id messages" when your network is having a problem, but not if you shutdown the live? Either way the backup always gets the connection failure error. Can you upload the server logs of the two scenarios for comparison?

Regarding the last question I don't think you need update jboss version.

Thanks for the logs. I'll take a look at them. Meanwhile can you tell me during Network Failure scenario, have you ever stopped the live server or you just let the live server running all the time?

Howard

Unfortunately in such a situation you have to manually shutdown the live and restart it. Currently what hornetq live does in case of network failure is just stop the replication and start it again when network failure is recovered.

One way to deal with it is to avoid such a split brain situation by setting a proper (longer) connection ttl and failure check period (ping interval) on the cluster connection. (However there is a bug regarding this so you need to use the latest to have it work).

Howard

Thanks Howard for helping us out. Are you suggesting that HornetQ/JBoss is designed this way or this is a bug? I assume that if someone is setting up live-backup for high availability then it should continue to work despite multiple network failures/glitches and it should not require manual intervention to restart one of the node.

Please point us to JBoss version which have all these fixes so that it work seamlessly.

Thanks,

Ravi

Yes this is what is implemented so far. I wouldn't say it's a bug here but I think it's possible we could improve this in the future. There seems to be no simple way to handle such a 'split brain' situation fully automatically and HornetQ is trying to best deal with it, for example using a quorum algorithm. As I suggested you'd better set up a connection ttl and ping to a value suitable for your actual network situation (for example if the network is slow and has long glitches you probably use a bigger connection ttl/ping values).

If you are using community releases for jboss and hornetq I suggest you update both to its latest stable releases. Only EAP releases comes with a specific version combination because they have strictly procedures for the integration.

Good luck

There are ways to mitigate "split brain" situations, but that requires a larger cluster.  See the paragraph regarding "split brain" at the end of section 39.1.2.

Your's is the pathological case where there is only 1 live and 1 backup and they get disconnected from each other long enough that the connection-ttl elapses.  You may consider switching to shared-storage if you find the network connection between the live and backup server is not reliable.