JBM 2.0 Strict ordering via ordering groups

[ataylor]

Firstly add a groupid and group sequence to message reference, this will be set on send similar to how scheduled delivery time is set. (theres a todo in ServerSessionImpl::send to do with duplicate code which i'll fix whilst I'm doing this)

Create a new interface and impl for controlling the delivery of grouped messages, we'll call this GroupedMessageHandler. This will maintain what messages are in transit for each group, it wll have a method canDeliver(MessageReference ref) that will return true if the message can be delivered, i.e., the consumer has acked the last reference sent and the group seq if this reference is the next in sequence. The deliver method of QueueImpl will use the GroupMessageHandler to decide if what messages can be delivered to the consumer and also notify it of each acknowledgment.

I don't think it makes sense for a queue to handle messages that are both grouped and non grouped so a flag can be added to specify tjhat a queue is handling grouped messages only. If a queue is 'enableGroupOrdering' the it should ignore any messages routed that don't have the groupid and seq set and also disallow message scheduling.

When creating a core producer it should be possible to set its groupid to an arbitrary string and allow it to set the sequence automatically or let the user set it via the message itself.

[timfox]

Isn't your message grouping code which already exists in TRUNK sufficient to enforce strict ordering semantics?

If not, how so?

[ataylor]

GroupingRoundRobinDistributionPolicy pins a consumer to a message group, so if a queue has multiple consumers, messages with a certain group id set will always be distributed to the same consumer.

An ordering group, as far as i understood, was to do with the order of messages within a group. So you could send 2 sets of messages for 2 different groups and the messages would be delivered in order within their own group.

[timfox]

"ataylor" wrote:
GroupingRoundRobinDistributionPolicy pins a consumer to a message group, so if a queue has multiple consumers, messages with a certain group id set will always be distributed to the same consumer.

Yes, doesn't that give you ordering?

[ataylor]

Yes, doesn't that give you ordering?

yes it does, but isn't this done differently. You could have multiple consumers for a start, which is why you don't release the next message in the group until the previous has been acked. Also lets say you have 2 groups of messages odd and even, say m1 to m10, you should be able to deliver m2 even if m1 hasn't been delivered yet. Another thing is that you should always deliver in sequence, so if you send 3 messages with sequence 1, 3 then 2, the queue shouldn't deliver the message with sequence 3 before it delivers sequence no 2.

[timfox]

"ataylor" wrote:

yes it does, but isn't this done differently. You could have multiple consumers for a start,

What's the advantage in allowing multiple consumers? If you're ordering, then by definition, you will need to serialise consumption

Also lets say you have 2 groups of messages odd and even, say m1 to m10, you should be able to deliver m2 even if m1 hasn't been delivered yet.

I'm not sure I understand that, can you elaborate?

Another thing is that you should always deliver in sequence, so if you send 3 messages with sequence 1, 3 then 2, the queue shouldn't deliver the message with sequence 3 before it delivers sequence no 2.

Why would message groups deliver messages out of order? I don't follow that either.

[ataylor]

What's the advantage in allowing multiple consumers? If you're ordering, then by definition, you will need to serialise consumption

I was just taking this from the comments on the Jira, 'In order to guarantee strict ordering, even on rollback or in the presence of multiple consumers or with xa transactions, the queue needs to ensure that no more than one message with the value of ordering group is being delivered at any one time.'

I'm not sure I understand that, can you elaborate?

So you send 10 messages, 5 for group A, with seq 1 to 5, and 5 for group B again with seq 1 to 5. A consumer consumes message group A seq 1 but does not acknowledge, the next message in group A with seq 2 can't be delivered but message group B seq 1 can, This gives you strict ordering within the defined group.

Why would message groups deliver messages out of order? I don't follow that either.

The producer can specify on send the sequence number of a message to specify its delivery order.

[timfox]

"ataylor" wrote:

What's the advantage in allowing multiple consumers? If you're ordering, then by definition, you will need to serialise consumption

I was just taking this from the comments on the Jira, 'In order to guarantee strict ordering, even on rollback or in the presence of multiple consumers or with xa transactions, the queue needs to ensure that no more than one message with the value of ordering group is being delivered at any one time.'

I still don't see the advantage of multiple consumers.

So you send 10 messages, 5 for group A, with seq 1 to 5, and 5 for group B again with seq 1 to 5. A consumer consumes message group A seq 1 but does not acknowledge, the next message in group A with seq 2 can't be delivered but message group B seq 1 can, This gives you strict ordering within the defined group.

Yes, but the current message grouping also provides that same ordering guarantee, does it not?

The producer can specify on send the sequence number of a message to specify its delivery order.

I'm not sure what that means. Again, does not message groups preserve delivery order.

I'm just trying to understand here how message grouping does not accomplish the goal.

The goal here is to provide strict ordering of delivery of messages, even on rollback and with multiple consumers.

[ataylor]

Ive written some tests to verify that the current GroupingRoundRobinDistributionPolicy will provide strict ordering. Apart from one slight problem it does.
The problem is that the GroupingRoundRobinDistributionPolicy tries available consumers until it finds one it can use. If one is busy then it tries the next available. If a queue holds a set of messages that belong to two different groups, and the queue has 2 consumers, as soon as consumer 1 is started it will try to deliver the messages and both groups can be pinned to the first consumer before the second consumer can be started. Of course The user could make sure that the session is started before any messages are delivered to the queue, but since the session:start method is non blocking the first messages could still arrive before all consumers are started. making the start method a blocking call would fix this tho.

[ataylor]

actually, calling start as a blocking call won't make a difference since the sender will probably be on different sessions.

[timfox]

Why not pin groups to consumers irrespective of whether they're started or not?

I think that would make more sense anyway, e.g. with a pool of MDBs

[ataylor]

good idea, that solves it and makes it simpler as well.

[gaohoward]

My understanding of message group in JBM 2.0 is that all messages that belong to one message group are guaranteed to be delivered to one consumer. This by definition has nothing to do with strict ordering (or even ordering). They are different concepts. Message groups doesn't necessarily need to observe ordering, and ordering doesn't necessarily enforce single consumer.

Although we can combine the two into one code module, we still can't mingle the two concepts, we still should consider the functionalities for each feature. As for ordering vs message group

1. consumers of strict ordered messages cannot receive second message without first being acknowledged. Consumers of message group can.

2. If we can guarantee strict ordering, there is no need to nail the ordered messages to one consumer. Consider if there are two consumers on one queue that deliver ordered messages. If one consumer gets a failure situation and is not able to consume anymore, the current message will not be acked (or will be rolled back), the other consumer may take over the job (as backup) and continue working (it may call session.recover() first). Will message group allow that happen?

3. If user only wants message group, he shouldn't pay the performance cost brought by strict ordering.

4. In failover and cluster cases, message group and strict ordering are handled differently. Consumer identity is critical for a message group to survive a failure, but sequence is vital to ordering.

[timfox]

"gaohoward" wrote:

1. consumers of strict ordered messages cannot receive second message without first being acknowledged.

I don't agree. For ordering there is no requirement that previous message is acked first - that's an implementation detail. The only requirement is that messages are delivered in order.

2. If we can guarantee strict ordering, there is no need to nail the ordered messages to one consumer. Consider if there are two consumers on one queue that deliver ordered messages. If one consumer gets a failure situation and is not able to consume anymore, the current message will not be acked (or will be rolled back), the other consumer may take over the job (as backup) and continue working (it may call session.recover() first). Will message group allow that happen?

Yes, with message grouping, if a consumer closes, another consumer will get pinned to that group.

3. If user only wants message group, he shouldn't pay the performance cost brought by strict ordering.

What performance penalty is that?

4. In failover and cluster cases, message group and strict ordering are handled differently. Consumer identity is critical for a message group to survive a failure, but sequence is vital to ordering.

I didn't understand this last point. Can you elaborate?

[timfox]

There are however two cases I can think of where an message groups approach by itself is not sufficient to provide strict ordering.

1) ConnectionConsumer - this is not an issue in JBM 2.0 since there is no connection consumer, but it is an issue in JBM 1.4

2) XA transactions. Messages could be consumed in multiple transactions using the same consumer, and then rolled back in a different order to which they were consumed.

To fix 2) we can use a message group but would also need to pin the group to the XA transaction preventing messages being consumed in any other XA transaction until it's committed/rolled back. This would also have to be persisted in storage with the prepared tx data, since if failure occurs after prepare and the server is restarted, delivery of messages for that message group cannot occur until the tx is resolved (rolled back/committed).