-
1. Re: Cluster connection on failover
ataylor Aug 3, 2011 3:40 AM (in response to huma)Yes this is expected behaviour, the messages are held in a storeandforwardqueue waiying for the second server (or its backup) to come back. SInce you have forward-when-no-consumers=true messages will still be load balanced.
-
2. Re: Cluster connection on failover
jnelas Aug 3, 2011 8:48 AM (in response to ataylor)Just to clarify, on a cluster, if one of the machines goes down, the messages that would normally be delivered to that machine will wait indefinitely for that machine to recover, the other machines will not receive them, correct?
Will this behaviour change as the discussion on http://community.jboss.org/message/584510#584510 seems to imply?
-
3. Re: Cluster connection on failover
ataylor Aug 3, 2011 8:54 AM (in response to jnelas)Just to clarify, on a cluster, if one of the machines goes down, the messages that would normally be delivered to that machine will wait indefinitely for that machine to recover, the other machines will not receive them, correct?
correct, but like i say, if forward when no consumers=false then load balancing would stop
Will this behaviour change as the discussion on http://community.jboss.org/message/584510#584510 seems to imply?
Im not sure i see where its implied?
-
4. Re: Cluster connection on failover
jnelas Aug 3, 2011 8:58 AM (in response to ataylor)
Will this behaviour change as the discussion on http://community.jboss.org/message/584510#584510 seems to imply?Im not sure i see where its implied?
If the cluster is considered dead after a while, the messages should be treated by the remaining server.
-
5. Re: Cluster connection on failover
huma Aug 3, 2011 9:03 AM (in response to ataylor)Hi Andy,
correct, but like i say, if forward when no consumers=false then load balancing would stop
I've tried this but the behaviour keeps the same. The messages are still stuck.
The client only receives half of the messages.
-
6. Re: Cluster connection on failover
ataylor Aug 3, 2011 9:04 AM (in response to huma)thats probably because they have already beed load balanced. just bring back the server and delivery will continue.
-
7. Re: Cluster connection on failover
ataylor Aug 3, 2011 9:05 AM (in response to ataylor)If the cluster is considered dead after a while, the messages should be treated by the remaining server.
you sure your pointing me at the right link, i cant see where this is mentioned at all
-
8. Re: Cluster connection on failover
jnelas Aug 3, 2011 9:14 AM (in response to ataylor)Andy Taylor wrote:
If the cluster is considered dead after a while, the messages should be treated by the remaining server.
you sure your pointing me at the right link, i cant see where this is mentioned at all
On last message on the thread by Clebert Suconic he says:
"However this is being changed now. I'm introducing a reconnect attempt# on the cluster connection. That means the cluster will be considered dead and the remote binding will be removed on that case."
The conclusion that the messages would be treated by the other machines is mine.
-
9. Re: Cluster connection on failover
huma Aug 3, 2011 9:21 AM (in response to ataylor)Andy Taylor wrote:
thats probably because they have already beed load balanced. just bring back the server and delivery will continue.
In the test that I did, the messages were only sent after I killed the second server. They still got stuck.
-
10. Re: Cluster connection on failover
ataylor Aug 3, 2011 9:22 AM (in response to jnelas)The binding and the queue are different things, once the binding is removed the queue is no longer available to forward messages too, however the queue will still exist and may have messages in it. Im not saying that at some point we wont change this to allow a user to configure what will happen to messages in a queue however, but arbitrarily just sending them back to the server to redistribute is something we won't do. The reason why we have a redistribute setting on queues is that when you do this you can (and probably will) break ordering gaurantees.
-
11. Re: Cluster connection on failover
jnelas Aug 3, 2011 9:35 AM (in response to ataylor)But will the redistribute setting apply to messages that are on the storeandforwardqueue for delivery to a dead cluster member?
-
12. Re: Cluster connection on failover
ataylor Aug 3, 2011 9:38 AM (in response to jnelas)But will the redistribute setting apply to messages that are on the storeandforwardqueue for delivery to a dead cluster member?
The redistribute setting is on live nodes that have no consumers, i was just using it to illustrate a scenario.
If you dont have a consistent cluster, i.e. where there a possibilities that servers will crash, then you should configure backup servers. remember, what if the messages were sent to the server just before it crashed, they are still lost.
-
13. Re: Cluster connection on failover
jnelas Aug 3, 2011 10:03 AM (in response to ataylor)If you dont have a consistent cluster, i.e. where there a possibilities that servers will crash, then you should configure backup servers. remember, what if the messages were sent to the server just before it crashed, they are still lost.
True, I get your point.
On my specific case the servers didn't crash, for some unknown reason the cluster happears to get disconnected at some point and the messages get stuck on the storeandforwardqueue. The cluster eventually reconnects, but the messages are stuck forever.
Newer messages pass though the bridge, but the old ones never get delivered to none of the servers.
I realize that the connection problems between the cluster are a different issue, but one that would be minimized if there was a way to deal with the messages that are on the storeandforwardqueue.
-
14. Re: Cluster connection on failover
ataylor Aug 3, 2011 10:17 AM (in response to jnelas)On my specific case the servers didn't crash, for some unknown reason the cluster happears to get disconnected at some point and the messages get stuck on the storeandforwardqueue. The cluster eventually reconnects, but the messages are stuck forever.
Newer messages pass though the bridge, but the old ones never get delivered to none of the servers.
Ok, thats slightly different to what you are saying, if the server reconnects then the messages should be sent. can you verify thet they are in the store and forward queue and that the cluster connection is remade.
Also what version are you using, if there is an issue it may well havebeen fiixed in the last release