-
30. Re: Client failover redeliveries discussion
"Tim" wrote:
How can you ensure that different clients that use the same queue fail over on to the same node?
Based on what we talked before, a client connection to a nodeA will always have to faile to a nodeB. A server rule will determine this."Tim" wrote:
Non clustered queues don't need to be reconnected. If queues want to benefit from HA then they should be made clustered.
What we should do then if a connectionState has a non clustered queue on its hierarchy during a connection failure event then? Throw an exception? Disconnect the Consumer?
So far I think we could (don't know if we should) reconnect non clustered topics on the new node. We won't be able to guarantee deliveries, but at least we wouldn't interrupt the client. Now if the requirement is to fail over only on ClusteredQueues we should know what to do then."Tim" wrote:
Personally I don't think you can drive the failover fully from the client side.
I will be able to refactor anything if we get some problem. I'm constructing this piece by piece, we will be able to refactor later. -
31. Re: Client failover redeliveries discussion
"Tim" wrote:
There are other issues here. When a durable subscription is attached to a topic, then the subscription must retain all messages *even if it isn't active* - this is a requirement of the JMS spec
Wouldn't a queue.load take care of this? Maybe I didn't understand your point here. -
32. Re: Client failover redeliveries discussion
"clebert.suconic@jboss.com" wrote:
Based on what we talked before, a client connection to a nodeA will always have to faile to a nodeB. A server rule will determine this.
So it's not totally driven from the client side then?
What we should do then if a connectionState has a non clustered queue on its hierarchy during a connection failure event then? Throw an exception? Disconnect the Consumer?
I would say disconnect it and any further operations (e.g. receive()) on that consumer should throw an exception.
So far I think we could (don't know if we should) reconnect non clustered topics on the new node. We won't be able to guarantee deliveries, but at least we wouldn't interrupt the client. Now if the requirement is to fail over only on ClusteredQueues we should know what to do then.
A non clustered destination is, by definition, non clustered. Therefore there is no expectation of HA for that destination.
I will be able to refactor anything if we get some problem. I'm constructing this piece by piece, we will be able to refactor later.
That's cool.
I'd just be careful of going too far down the wrong path. Otherwise you may have a *lot* of refactoring to do :) -
33. Re: Client failover redeliveries discussion
"clebert.suconic@jboss.com" wrote:
Wouldn't a queue.load take care of this? Maybe I didn't understand your point here.
If you have 2 nodes, node A and node B.
There exists durable sub S1 on node A for topic T
Messages are being sent via node A to topic T.
They are routed to node B where they go into S1.
node B then fails
More messages are now sent to node A, it is a requirement of the JMS spec that S1 still receives the messages sent.
If you don't reload S1 until the client requests it, then S1 won't get the message and we have broken the JMS spec.
This isn't such an issue when using a shared db since we persist before send, but when doing in memory replication it simply won't work.
A similar reasoning applies to queues. We're not permitted to lose any messages destined for a queue while failover is occurring. We must ensure that all messages end up in the queue. -
34. Re: Client failover redeliveries discussion
"Tim" wrote:
This isn't such an issue when using a shared db since we persist before send, but when doing in memory replication it simply won't work.
My understand was we would guarantee deliveries on PersistentQueues only. We will have to test these conditions later. -
-
36. Re: Client failover redeliveries discussion
"ovidiu.feodorov@jboss.com" wrote:
I've checked in a simple failover test that fails. It's too late to look too deep into it now, but I'll get back tomorrow. In the mean time, you could take a look.
It's ManualClusteringTest.testSimpleFailover()
The test is valid. I will look into that.
I have tested with durable subscriptions only so far. -
37. Re: Client failover redeliveries discussion
This is a thread created to discuss client-side fail-over. How about keeping it focused on this? Tim created a different thread for server-side fail-over discussions: http://www.jboss.org/index.html?module=bb&op=viewtopic&t=92225
Clebert, would you kindly summarize the server-side situation so far, post it there, and continue the discussion on the other side?
Until then, I have a whole bunch of questions/suggestions related to the client-side. -
38. Re: Client failover redeliveries discussion
Clebert wrote:
If the consumer receives a message from CallBack but if it didn't send an ACK yet, after the failover, the server not knowing the message might throw an exception (messageId not known).
There are a couple of use cases we have to consider.
- Persistent Messages. (how to treat a redelivery).
- Should we send the list of previously ACKs to the server?
- Should we ignore ACKs for non existent messages on the server?
Let's systematize a bit the cases we need to deal with on fail-over. In order to do that, we need to understand what is maintained as transitory state (in flight messages and acknowledgments) by the client side delegate hierarchy.
A client may happen to be sending a message when the failure occurs. If the message is sent individually (not in the context of a transaction), then the on-going synchronous invocation is going to fail, the client code will catch the exception, and most likely will re-try to send the message, hopefully over a correctly failed-over connection. So there is nothing else to do here at this stage. (There may be something, though: for a truly seamless client fail-over, we need to handle this situation and retry sending the message by ourselves without client code actually noticing anything unusual, maybe just a send call that takes longer than usual).
If the client happens to send message in the context of a transaction when the failure occurs, we could either throw an exception, and discard everything, or go for the more elegant solution of transparently copying the transactional state (the corresponding TxState instance) into the new ResourceManager and send the messages over the new connection when transaction commits. We're probably not doing this right now, but this is what should be doing.
It is interesting to consider also what happens when the failure occurs right in the middle of "sentTransaction()" invocation.
These are example of use cases we need to deal with, when sending and failover are concerned. Each such use case must be captured in its own test case. I will start working towards that.
The series of use cases to be considered continues with situations in which messages being received are affected by failure.
Messages arrive to the MessageCallbackHandler instance, which is an consumer delegate accessory (there is an one-to-one relationship between the MessageCallbackHandler instances and the consumer delegate instances). At this stage, it really doesn't make any difference if they are persistent or non-persistent messages, they are treated all the same.
If the failure occurs while a message is being sent to the MessageCallbackHandler, the in-flight message instance is simply lost (there will be some error log statements on the server-side, if it's only a network failure, or there won't be anything at all, if the server box goes down). For a non persistent message, this is the expected outcome in case of failure, and for a persistent message, it will be eventually recovered from the database, so everything will end up fine.
Nothing to do here from a client-failover perspective.
What is more interesting is what happens with the messages that are already in the MessageCallbackHandler's buffer.
For a seamless fail-over, they will need to be transferred in the new MessageCallbackHandler's buffer. Also important, immediately after the failover condition is detected, any in-progress read should be completed, and no further reads should be accepted until the client-side fail-over is complete ("client side failover lockdown"). The next post-failover read should be done from the new MessagingCallbackHandler's buffer.
Contrary to what has been discussed so far on this thread, I think we can also salvage non-persistent messages, with minimum of effort. I'll address this issue again later. The acknowledgments for these messages (persistent and non-persistent) will be sent by the new Connection Delegate.
We also have the acknowledgments accumulated in a transaction on the client-side. The case should be dealt with similarly with the way we handle transacted messages (copy the TxState instance).
Again, for all these cases we should have tests.Clebert wrote:
Second point also:
What to do when a durable subscriber gets the queue refilled?
- The client will probably receive the message again. I would just ignore redeliveries.Tim wrote:
Yes - we should send the ids of every persistent message as part of the failover protocol - the server then repopulates the delivery list in the server consumer endpoint
This will help us avoid situation when a "failed-over" message is consumed on the client-side, the consumer delegate sends the acknowledgment and the new server consumer endpoint doesn't know what the client is talking about, because there's no active delivery for that message. I think we can go a step further and also send the IDs of non-persistent messages that have been "failed-over" on the client side. This way, the client will continue to receive (and successfully acknowledge) non-persistent messages that otherwise would have been lost.Tim wrote:
Clebert wrote:
- Should we ignore ACKs for non existent messages on the server?
Non existent messages on the server will be non persistent messages that didn't survive the failover.
They should be removed from the client side list on failover so the acks will never get sent.
Not necessarily. See my above comment. We could also include the ids of non-persistent messages with the list of message ID sent to the server as part of the failover protocol, and thus be able to "salvage" those messages as well. I don't see any problem if we do that. We get better fault tolerance.Clebert wrote:
failoever receives a new ClientConnectionDelegate as the parameter. The idea is to get a new connection, but keep the actual delegates we are using.
We need to make sure that the "new" connection is then properly discarded so it will eventually garbage collected. Minor thing, anyway, just an observation.Clebert wrote:
Creating a new connection on the new server will create a new server Object, consequently a new ServerId.
You probably want to say that creating a new connection will subsequently create a new server-side connection endpoint instance, and because we're connecting to a different server node, we need to use its serverID instead of the "dead node" serverID, right?Clebert wrote:
Tim wrote:
Clebert wrote:
Second point also:
What to do when a durable subscriber gets the queue refilled?
- The client will probably receive the message again. I would just ignore redeliveries.
I don't understand the issue here. Can you explain more?
The consumer is going to be recreated the same way the connection on the previous example. (creating a new consumer / replacing the IDs on the old objects).
On that case, the new server will think it's a new client connecting and it will resend non committed messages from a durable subscription. (So I hope).
On that case, I'm considering to ignore message previously sent, and on the list of CurrentTransaction.ACKs(). I just want to know if this is everybody's expected behavior.
What about the "fail-over protocol"? Your statement above seem to assume that the new server node is called into without any "preparation", as would a completely new client that creates a new connection, session and consumer endpoints. This is not going to work, those server-side objects need to undergo a "post-failover" preparation phase, where deliveries for the client-side failed over messages are created and so forth.Tim wrote:
So to summarise:
1) Detect failover
2) Find the "correct" failover server. (This may take several hops)
3) Let the server "stall" you until server failover has completed
4) Recreate the conections, sessions, consumers, producers and browsers. (Swapping ids here sounds fine)
5) Delete any non persistent messages from the client list of unacked messages in any sessions in the failed connection.
6) Send a list of the ids of the peristent messages for each consumer that failed to the server. For each list recreate the ServerConsumerEndpoint delivery list by removing the refs from the channel and creating deliveries and putting in the list.
7) The connection is now ready to be usedClebert wrote:
At this point I'm not detecting the failure at remoting level yet.
At this point, given the way Remoting discussion evolve, we can safely assume we'll use Remoting connection failure facilities to detect our connection failure.Tim wrote:
So to summarise:
...
3) Let the server "stall" you until server failover has completed
...
What exactly does this mean?Tim wrote:
So to summarise:
...
5) Delete any non persistent messages from the client list of unacked messages in any sessions in the failed connection.
...
Why? See my comment above. Why do you think "salvaging" non-persistent messages too isn't going to work?Tim wrote:
So to summarise:
...
6) Send a list of the ids of the peristent messages for each consumer that failed to the server. For each list recreate the ServerConsumerEndpoint delivery list by removing the refs from the channel and creating deliveries and putting in the list.
...
Non-persistent message ids too.
We also need to insure that the client code is safely prevented from using the connection(s) in a transitory state during the fail over. This would be the "client-side failover lockdown" ....Tim wrote:
Note: We must also ensure that no new connections are created on the failover node while old connections are being recreated, otherwise we can have a situation where the new connections grab the messages which have already been delivered to consumers in the failed connecton!
... and this would be the corresponding "server-side failover lockdown".Clebert wrote:
Tim wrote:
I don't see why anything would be received twice (apart from non persistent
messages since we lose the acks for them but that's fine)
There are some scenarios I'm thinking about, all of them under incomplete transactions.
I will create few testcases and will come up with them to the discussion later.
I have started a list of scenarios at the beginning of this post. The way should be deal with this situation is to create test cases for each of use cases we are going to support. This way we make sure that if we refactor the code later, it still behaves how we originally wanted it to. No new functionality should ideally be added if there is no test case for it.Clebert wrote:
Every time a new serverObject is created [...]
Every time a new server-side endpoint is created ... :) -
39. Re: Client failover redeliveries discussion
Ovidiu wrote:
Let's systematize a bit the cases we need to deal with on fail-over.
......
Again, for all these cases we should have tests.
I havent thought much about retries and stuff like that. I agree we should create test for these scenarios.
At this point I'm concentrating in transfer state from one connection to another in conjunction with server side failover.
I will start treating connection events problems after I have finished this state transfer.Ovidiu wrote:
This will help us avoid situation when a "failed-over" message is consumed on the client-side, the consumer delegate sends the acknowledgment and the new server consumer endpoint doesn't know what the client is talking about, because there's no active delivery for that message. I think we can go a step further and also send the IDs of non-persistent messages that have been "failed-over" on the client side. This way, the client will continue to receive (and successfully acknowledge) non-persistent messages that otherwise would have been lost.
Let me make some progress with the server side failover, then we will decide what to do with these IDs. We should do it next week.Ovidiu wrote:
We need to make sure that the "new" connection is then properly discarded so it will eventually garbage collected. Minor thing, anyway, just an observation.
agreed.Ovidiu wrote:
Clebert wrote:
Creating a new connection on the new server will create a new server Object, consequently a new ServerId.
You probably want to say that creating a new connection will subsequently create a new server-side connection endpoint instance, and because we're connecting to a different server node, we need to use its serverID instead of the "dead node" serverID, right?
Yep... I have recently tried to improved my terminology. That was sent before your request ;-)Ovidiu wrote:
What about the "fail-over protocol"? Your statement above seem to assume that the new server node is called into without any "preparation", as would a completely new client that creates a new connection, session and consumer endpoints. This is not going to work, those server-side objects need to undergo a "post-failover" preparation phase, where deliveries for the client-side failed over messages are created and so forth.
I'm working on the preparation/failover protocol right now. I'm trying to drive it from the client as the first connection fails to a node appears."Ovidiu" wrote:
At this point, given the way Remoting discussion evolve, we can safely assume we'll use Remoting connection failure facilities to detect our connection failure."Ovidiu" wrote:
I have started a list of scenarios at the beginning of this post. The way should be deal with this situation is to create test cases for each of use cases we are going to support. This way we make sure that if we refactor the code later, it still behaves how we originally wanted it to. No new functionality should ideally be added if there is no test case for it.
Yep!
I will let Tim comment on the list. -
40. Re: Client failover redeliveries discussion
I thought we had gone over all this already, but here goes again....
"ovidiu.feodorov@jboss.com" wrote:
A client may happen to be sending a message when the failure occurs. If the message is sent individually (not in the context of a transaction), then the on-going synchronous invocation is going to fail, the client code will catch the exception, and most likely will re-try to send the message,
What do you mean the "client code" will catch the exception?
Do you mean the application code?
If so, this is incorrect - JBoss Messaging failover is supposed to be automatic - this is one of our selling points.
Applications shouldn't have to catch connection exceptions and retry like in JBossMQ."Ovidiu" wrote:
If the client happens to send message in the context of a transaction when the failure occurs, we could either throw an exception, and discard everything, or go for the more elegant solution of transparently copying the transactional state (the corresponding TxState instance) into the new ResourceManager and send the messages over the new connection when transaction commits. We're probably not doing this right now, but this is what should be doing.
Same reasoning applies as previous comment. It should be transparent."Ovidiu" wrote:
It is interesting to consider also what happens when the failure occurs right in the middle of "sentTransaction()" invocation.
Again same reasoning applies.
There is a problem here of an exception being received and a retry occuring but the transaction/send actually went through on the previous node.
Please see discussion on duplicated message detection in a previous thread for more information on this."Ovidiu" wrote:
What is more interesting is what happens with the messages that are already in the MessageCallbackHandler's buffer.
For a seamless fail-over, they will need to be transferred in the new MessageCallbackHandler's buffer. Also important, immediately after the failover condition is detected, any in-progress read should be completed, and no further reads should be accepted until the client-side fail-over is complete ("client side failover lockdown"). The next post-failover read should be done from the new MessagingCallbackHandler's buffer.
There is no need to copy anything since Clebert is re-using the same connection, consumer, buffer objects before and after failover - he is just changing the ids."Ovidiu" wrote:
Contrary to what has been discussed so far on this thread, I think we can also salvage non-persistent messages, with minimum of effort. I'll address this issue again later. The acknowledgments for these messages (persistent and non-persistent) will be sent by the new Connection Delegate.
We also have the acknowledgments accumulated in a transaction on the client-side. The case should be dealt with similarly with the way we handle transacted messages (copy the TxState instance).
Again, no copying is necessary - just re-use the same object."Ovidiu" wrote:
Tim wrote:
Yes - we should send the ids of every persistent message as part of the failover protocol - the server then repopulates the delivery list in the server consumer endpoint
I think we can go a step further and also send the IDs of non-persistent messages that have been "failed-over" on the client side. This way, the client will continue to receive (and successfully acknowledge) non-persistent messages that otherwise would have been lost.
This makes no sense. When server A fails and server B takes over, only the persistent messages are resurrected into server B's queues.
The non persistent messages are lost.
Therefore it's not possible that the non persistent messages can be successfully acknowledged on server B, since server B won't know about them.
This is why I said the non persistent messages should be removed from the client state so they don't attempt to be acked."Ovidiu" wrote:
Tim wrote:
Clebert wrote:
- Should we ignore ACKs for non existent messages on the server?
Non existent messages on the server will be non persistent messages that didn't survive the failover.
They should be removed from the client side list on failover so the acks will never get sent.
Not necessarily. See my above comment. We could also include the ids of non-persistent messages with the list of message ID sent to the server as part of the failover protocol, and thus be able to "salvage" those messages as well. I don't see any problem if we do that. We get better fault tolerance.
How can you salvage a message that doesn't exist any more? The non persistent messages wil have been lost when server A failed."Ovidiu" wrote:
What about the "fail-over protocol"? Your statement above seem to assume that the new server node is called into without any "preparation", as would a completely new client that creates a new connection, session and consumer endpoints. This is not going to work, those server-side objects need to undergo a "post-failover" preparation phase, where deliveries for the client-side failed over messages are created and so forth.
Correct."Ovidiu" wrote:
Tim wrote:
So to summarise:
...
3) Let the server "stall" you until server failover has completed
...
What exactly does this mean?
When a server fails, the server side failover kicks in, and the server loads those queues which it is taking over responsibuility for.
This may take several seconds, during which time we do not want a failed over client connection to start sending/consuming from those queues since they might receive a partial state.
Hence we need to stall the connection at reconnection until the server completes its failover protocol. I.e. a "valve".
This is covered in the wiki page I believe (like most of this stuff)."Ovidiu" wrote:
Tim wrote:
So to summarise:
...
5) Delete any non persistent messages from the client list of unacked messages in any sessions in the failed connection.
...
Why? See my comment above. Why do you think "salvaging" non-persistent messages too isn't going to work?
Because the new server know nothing about the non persistent messages, since they won't have survived the server failure."Ovidiu" wrote:
Non-persistent message ids too.
No point doing that, for the reasons explained twice already in this thread. -
41. Re: Client failover redeliveries discussion
Tim wrote:
I thought we had gone over all this already, but here goes again....
ovidiu.feodorov@jboss.com wrote:
A client may happen to be sending a message when the failure occurs. If the message is sent individually (not in the context of a transaction), then the on-going synchronous invocation is going to fail, the client code will catch the exception, and most likely will re-try to send the message,
What do you mean the "client code" will catch the exception?
Do you mean the application code?
If so, this is incorrect - JBoss Messaging failover is supposed to be automatic - this is one of our selling points.
Applications shouldn't have to catch connection exceptions and retry like in JBossMQ.
Ovidiu wrote:
If the client happens to send message in the context of a transaction when the failure occurs, we could either throw an exception, and discard everything, or go for the more elegant solution of transparently copying the transactional state (the corresponding TxState instance) into the new ResourceManager and send the messages over the new connection when transaction commits. We're probably not doing this right now, but this is what should be doing.
Same reasoning applies as previous comment. It should be transparent.
That is exactly what I am saying: "We're probably not doing this right now, but this is what should be doing."
To spell this out for you, so you won't go over is again: The current code, that is in the SVN, does not behave this way. This is understandable, it's just a prototype. I was going over the use cases for which we need test cases, so we can make sure the correct behavior is preserved in the future. The correct behavior is "transparently copying the transactional state (the corresponding TxState instance) into the new ResourceManager and send the messages over the new connection when transaction commits." Please note the use of the word "transparent". -
42. Re: Client failover redeliveries discussion
Tim wrote:
This makes no sense. When server A fails and server B takes over, only the persistent messages are resurrected into server B's queues.
The non persistent messages are lost.
Therefore it's not possible that the non persistent messages can be successfully acknowledged on server B, since server B won't know about them.
This is why I said the non persistent messages should be removed from the client state so they don't attempt to be acked.
Please consider this simple example: the non-persistent messages N1 (id = 1) and N2 (id=2), sent by the server A, sit in the client-side buffer. The server A fails, the VM goes down. The connection is re-established (transparently) to server B. Server B knows that is a secondary server for the connection that just has established (as a result of the failover protocol). So, it could naturally assume that there are undelivered non-persistent messages in the client-side buffer. Which have not been lost. It also learns that the id of the messages are 1 and 2 (as a result of the fail-over protocol). So, what is stopping the server B (again, knowing that is a secondary server that has just been failed-over to), to accept acknowledgments for non-persistent messages that have been salvaged this way? -
43. Re: Client failover redeliveries discussion
"ovidiu.feodorov@jboss.com" wrote:
Please consider this simple example: the non-persistent messages N1 (id = 1) and N2 (id=2), sent by the server A, sit in the client-side buffer. The server A fails, the VM goes down. The connection is re-established (transparently) to server B. Server B knows that is a secondary server for the connection that just has established (as a result of the failover protocol). So, it could naturally assume that there are undelivered non-persistent messages in the client-side buffer. Which have not been lost. It also learns that the id of the messages are 1 and 2 (as a result of the fail-over protocol). So, what is stopping the server B (again, knowing that is a secondary server that has just been failed-over to), to accept acknowledgments for non-persistent messages that have been salvaged this way?
Still makes no sense.
What's the point of sending an ack that's just going to be ignored?
It's much easier to just not send it in the first place, you can still process the np messages in the client buffer, just completely pointless to send acks to the server for them. -
44. Re: Client failover redeliveries discussion
"ovidiu.feodorov@jboss.com" wrote:
To spell this out for you, so you won't go over is again: The current code, that is in the SVN, does not behave this way. This is understandable, it's just a prototype. I was going over the use cases for which we need test cases, so we can make sure the correct behavior is preserved in the future. The correct behavior is "transparently copying the transactional state (the corresponding TxState instance) into the new ResourceManager and send the messages over the new connection when transaction commits." Please note the use of the word "transparent".
I don't think you need to copy anything - just re-use the old one.