Message sequence guarantees (or lack thereof)

[almac]

Hi,

What kind of guarantees does errai-bus make about message delivery sequencing?

For example, suppose client code does something like this:

{code}
  ...
  bus.subscribe("ServiceSnacks", snackMessageCallback);
  MessageBuilder.createMessage().toSubject("MyService")
    .command("go")
    .defaultErrorHandling()
    .repliesTo(goMessageCallback);
  ...
{code}

and the corresponding server code looks something like this:

{code}
  @Service
  public class MyService {
    ...
    @Command("go")
    public void go(final Message message) {
    ...
    MessageBuilder.createConversation(message).subjectProvided()
      .signaling()
      .defaultErrorHandling()
      .reply();
    MessageBuilder.createMessage().toSubject("ServiceSnacks")
      .defaultErrorHandling()
      .sendNowWith(bus);
    ...
    }
  ...
  }
{code}

Which of the client callbacks is executed first?

I could imagine (and would prefer) that it would be goMessageCallback, followed by snackMessageCallback ---

since the service .reply()'s to the "go" message before initiating the broadcast to "ServiceSnacks". 

I ask because I'm seeing both orderings (mostly depending on where and when I run the code), and I'm

wondering if I have discovered a bug or a feature.

[cbrock]

In this case there's really no way to garauntee ordering. You can if you want, provide a salience number to the bus as part of the message payload. ie.

with("PriorityProcessing", 1);

Currently there is only two priority levels. Although the spec appears to implement unlimited saliency, we only distinguish between salient and non-salient messages right now.  In other words, if you get one message priority processing, and not the other... and they both arrive at the same time, the one with priority processing will always arrive first.

[almac]

"In this case there's really no way to garauntee ordering."  Ok... in what cases *can* ordering be guaranteed?

Please don't say "never", or I'll just go back to using UDP and strcpy(). 

(It wouldn't hurt to have a document which provides an overview of the bus architecture.

I vaguely recollect that I saw some document with pictures/diagrams a few months ago,

but I can't find anything like it anywhere now.  Could have been dreaming.)

[cbrock]

Well, I didn't really say "never". =)

What I said is that the actual saliency number is in the protocol but it only respects two classes of messages right now. Largely because the coordinating overhead in a fully asynchronous eventing pattern would be extremely complex. There's no way to say, garauntee that two messages send from two seperate callsites with two different saliencies would be candidates for ordering. They'd both have to be in transit at the exact same time.

[almac]

I think we're talking about two different (albeit related things).  I'm not asking about message priorities,

rather I'm concerned with queuing behavior.

For example, given the following:

{code}
...
MessageBuilder.createMessage().toSubject("Anyone").withValue("a")
      .defaultErrorHandling().sendNowWith(bus);
MessageBuilder.createMessage().toSubject("Anyone").withValue("b")
      .defaultErrorHandling().sendNowWith(bus);
MessageBuilder.createMessage().toSubject("Anyone").withValue("c")
      .defaultErrorHandling().sendNowWith(bus);
MessageBuilder.createMessage().toSubject("Anyone").withValue("d")
      .defaultErrorHandling().sendNowWith(bus);
...
{code}

Is a subscriber to "Anyone" guaranteed to receive the four messages in the order a-b-c-d, or not?

Does the answer depend on whether or not the subscriber is on the local bus or a remote bus?

I'd like to know enough about the bus/queue architecture to be able to know how delivery order is

related to submission order.

I had assumed that all messages on a bus were queued in order of submission, and that messages

travelling to a remote bus would go over a single channel, maintaining that same order --- hence my

surprise at the behavior I discovered in my initial message on this discussion.  Obviously I don't

understand some key features of the architecture of the bus.

[cbrock]

Yes the messages are sent in-order and generally should be received in in-order.

[almac]

Ok... so then what is different between this case and the case from the beginning of this discussion?

[cbrock]

The only thing I can think of is that the second message you're replying with is a broadcast message and as such, it is being globally queued instead of serially queued to the recipient as in conversational messages. So, there is an opportunity for the two messages to appear inconsistantly.

The truth is, ErraiBus does make a best-effort attempt to deliver messages in-order. However, for performance reasons it will also take advantage of various strategies for getting messages to their destination as fast as possible, which could, in theory result in inconsistent ordering.

You should always remember that messages are asyncrhonous, and performance of message delivery can be affected by network conditions and topologies. I would tend towards saying that if you have a component which relies on receiving two or messages in a particular order, you should consider creating one composite message which contains all that state..

[almac]

The only thing I can think of is that the second message you're replying with is a broadcast message and as such, it is being globally queued instead of serially queued to the recipient as in conversational messages. So, there is an opportunity for the two messages to appear inconsistantly.

Ah-ha... so there is more than one queue per bus?

The truth is, ErraiBus does make a best-effort attempt to deliver messages in-order. However, for performance reasons it will also take advantage of various strategies for getting messages to their destination as fast as possible, which could, in theory result in inconsistent ordering.

What does "as fast as possible" mean?  It's one thing to make delivery as efficient as possible (minimize clock cycles to move a given volume of messages.  It's another thing to reorder delivery (moving certain classes of communication ahead of others) --- if that behavior is not well understood (by the user of the bus), the system risks starving/stalling certain consumers/producers. (Explicit priority/saliency settings, in counterpoint, do this reordering, too, but are in the direct control of the bus/library user.)

You should always remember that messages are asyncrhonous, and performance of message delivery can be affected by network conditions and topologies.

Asynchronicity, guaranteed successful delivery, and delivery ordering are orthogonal concerns --- you can have one without the others (and vice-versa). 

I would tend towards saying that if you have a component which relies on receiving two or messages in a particular order, you should consider creating one composite message which contains all that state..

If you have all that state available at one moment, certainly, put it all in one message.  That's not the case where the order matters, though.

The "a-b-c-d" example above was an abbreviated version of:

• send 'a'
• [time passes, other things happen]
• send 'b'
• [time passes, other things happen]
• send 'c'
• [time passes, other things happen]
• send 'd'

For example... the ol' stock ticker example: every message is encodes a trade executed on a stock.  The sender will submit a message for each trade as each trade is executed.  Those trades may be milliseconds apart, or hours apart.  The subscriber is going to want to know that the messages it receives are in that same order they got submitted, otherwise it will have to buffer them and reorder them itself.  That's more complexity, and latency, for the subscriber, that wouldn't be necessary if it could rely on the order.