Order of Deliveries between Backup and Live Node

[clebert.suconic]

I have been investigating https://jira.jboss.org/jira/browse/JBMESSAGING-1558 and this is what I have found so far:


On a regular flow, this is what happens when messages are delivered to Consumers:




References are aways backed up before delivered to Consumers. This process is totally async. We will fire the replication and wait the callback, and when it's backed up, we will deliver messages to the consumers.


However if the delivering messages are canceled for any reason. (that includes close, or rollback):



QueueImpl::deliverAsync (or DeliverRunner->deliver) will remove references of the top of the list on independently of the threads being used on the repplication.

As a result it may remove the message on LiveNode and when the message is repplicated the backup will have a different element on the top list.

[clebert.suconic]

Another version of the second diagram with more details:

[clebert.suconic]

On the real example, when you close a consumer you are using a new consumer.

Just use that as implicit on the diagram. Or else you would have too many participants and it would be harder to draw it.


BTW: rollback would be the same case. The issue is QueueImpl::asyncDelivery working independenty of the threads used on the repplication.

[clebert.suconic]

I'm using www.websequencediagrams.com for these.


This URL has a few changes I made:

http://www.websequencediagrams.com/?lz=cGFydGljaXBhbnQgQ29uc3VtZXIoQSkKAAMVQikKCiNTZWNvbmQgZGlhZ3JhbQoAKgstPlNlcnZlcgA9CzogY2xvc2UKYWN0aXZhdGUgABERCgAjEQA3EEJhY2t1cABDCiAgIChSZWZlcmVuY2UgKEEpADQQACgJAHwVZG9uZQpkZQB3GwCBcA0AgUQPQgCCDQ4AgRUVQgA0D1F1ZXVlSW1wbACBfQoACQoAFAktPgAfCTogcHJvbXB0RGVsaXZlcnkgAIFmCShBKQpub3RlIG92ZXIATgtTb21ldGhpbmcgd2lsbCAAgmYJADoOCmVuZCBub3RlCgCDLQpCAIMlEkIAgmEJAIJcDEIpKQoAgSIMAIJHFACBKgYAgSENAIEaFUF0IHRoaXMgcG9pbnQgTGl2ZSdzIHRvcCBpcwCBYgpBAIEkCwCEFhAAgSQSAIJzCQCBKhIAgTUGAIQ8LHJlcGxpY2F0ZQCCeQkAQwtBKQCCeAwAhRUXCkV4Y2VwdGlvbiEgVG9wIG9mIHRoZSBsaXN0IGlzIHIAgRELIGEAgxAGAAoLQSkAgXAM&s=napkin

[clebert.suconic]

There is a race between QueueImpl::deliveryAsync and the repplication of the deliveries.


ServerConsumer will handle a message and it will replicate it on the ServerConsumer(Backup).

When the delivery call arrives on the backup node, the backup's list could be already different than what it was on live node, because of the races caused by QueueImpl.

[timfox]

I think I can understand this now.

I believe it can be explained in a more simple use case:

consumer A rolls back message M1 on Queue Q
rollback is replicated
delivery of message M2 on live to consumer B on Queue Q
delivery is replicated
rollback hits backup and M1 added to top of Q
replicated delivery hits backup looking to find M2 on top of Q but instead find M1