[ataylor]

I am assuming that by receiver you mean consumer,

yes i do, too many trains of thought, i can't multtask :)

The whole point of using queues in a load balanced environment is so that you can have a single work queue (or virtual single queue as in a clustered queue) and be able to scale up your workers to meet the load.

what i am saying is that the JMS specification does not specify anything to do with load balancing, or how messages should be redistributed if multiple consumers are on a queue, see sec5.8. On Jboss Messaging our load balancing is achieved by delivering to different nodes, then each consumer consumes from each node fifo.

[ataylor]

The idea of a queue having only one process that can pull messages off is absurd

For every message a consumer receives from a queue it needs to send an ack back to the server.
If a queue has a single consumer then the server can deliver many messages to it which the consumer can ack in its own time, this is how we do it in JBoss Messaging with a client side buffer..
If a queue has multiple consumers the server can not to this. It has to wait until the first message is acked from the first consumer before it sends a message to the second so ordering could be guaranteed, meaning throughput would be slow and it wouldn't matter how many workers you had.
If it doesn't wait until the first message is acked then you cant guarantee ordering on the client side.

[chipschoch]

From 4.4.10.1 JMS 1.1 Spec

JMS defines that messages sent by a session to a destination must be received
in the order in which they were sent (see Section 4.4.10.2 �Order of Message
Sends,� for a few qualifications). This defines a partial ordering constraint on a
session’s input message stream.

From 4.4.10.2 of JMS 1.1. Spec:

The only ordering that is visible to receiving clients is the order of messages a session sends to a particular destination.

Bar priority messages etc. this tells me that my consumers should be consuming my messages in the order they were produced.

From the sun jms faq:

Note that in order to prevent duplicate delivery of a message from a durable subscription or queue, a message that can still be acknowledged by a session cannot be redelivered to another message consumer. The message can only be redelivered to another message consumer when it can no longer be acknowledged by the session that initially received the message.

Nothing to preclude sending the next message in the queue to a different consumer before receiving the ack form the first one.

From 5.8 in the JMS 1.1 Spec

Only QueueReceivers without a
message selector will read messages in message producer order

My test program has one producer that produces messages that are consumed by 2 consumers. I use no priority fetch and no message selectors. Everything I read convinces me that the order that the messages are consumed should be the order that they were produced. Now it may be that you cannot guarantee order of delivery because if message 1 failed then that would preclude you from being able to deliver message 2 until message 1 was finally delivered and acknowledged. But one would think that bar any failures, the order of delivery would be the order the messages were received by the queue, not some absolutely random order.

The point is academic, however, because the JBM implementation dos not do this.

[ataylor]

As clebert mentioned earlier, setting DefaultPreserveOrdering on serverpeer will assure message ordering.

[chipschoch]

I certainly appreciate your taking the time to participate in this discussion. It seems we have a permanent miscommunication. I am aware that setting DefaultPreserveOrdering will preserve the ordering of the message consumption but after numerous tests, some of which the results are posted in this thread, this clear that when that is set all the messages from the producer go to only one consumer. My question was how can I make it preserve the order of consumption when there are is more than one consumer.

Refer back to my bank line anecdote. All the people in the single line represent the messages in the queue. Each teller window represents a consumer. The first person in line goes to the next available teller. That person may successfully conduct their business or they may not. Irregardless, the following person in the line goes to the next available teller. That is the proper function of a queue. People don't randomly leave the line to race for the next available teller. The head of the queue goes next.

I don't understand why this concept seems so difficult to communicate. Either JBM can be configured to function like this or it cannot. Thus far my testing has shown it cannot. If there is a way to configure it so that it can please enlighten me. When I set DefaultPreserveOrdering to true all my messages go to one consumer. To employ my metaphor, all persons in line go to the same teller while the other teller sits there and does nothing. This is not optimal behavior.

[ataylor]

I certainly appreciate your taking the time to participate in this discussion.

no problems.

In Jboss Messaging if a single queue has more than one Consumer, we will deliver to the first available. If multiple consumers are registered with the queue(with no message selectors) then we will round robin and send to the next available consumer. If there are no local consumers on the queue then we distribute to another node in the cluster to be consumed.

The problem is that both of your consumers are attached to different nodes, if they were both attached to the same node the load would be shared. Always routing to the local consumer is the most performant way to do this.

So in answer to your question, you would need to use a singleton queue that all your consumers used and use the other node as a backup for fail over.

[clebert.suconic]

Andy wrote:
So in answer to your question, you would need to use a singleton queue that all your consumers used and use the other node as a backup for fail over.

A Singleton Queue is the way JBossMQ decided to implement cluster, and that was as performant as having a single node.

If you need multiple clients sharing a single queue with JBossMessaging, and you can't have the localQueues behavior.. you should use a single node, and you could have other nodes as backup nodes. Another way is if you aways had your producer on the BackupNode and consumers on remote nodes, the load would be shared with your clients.

I believe we have the Singleton Queue option planned for JBM 2.

Notice that this is still correct according to the JMS specification as it doesn't state how cluster whould behave on clustered distributions. (this is something up to the implementation).

Just coming back to your Banking example, when you go to the bank the next cashier will get you.. you won't need to go to the other side of the city to get the next available teller. The local queue has this behavior... you get the next teller at the agency you are currently on which is the cheapest operation.

What we can do now is consider the Single option for a future release for users that don't need to scale up their queues. I'm sure you can find a way to work this out with JBM 1.4 if you want... but I'm not sure if this will satisfy your needs.

[chipschoch]

Thank you Andy and Clebert for taking the time to explain this in detail. To reiterate, what you are saying is that if I need to have multiple consumers processing messages from a clustered queue in FIFO order, than I should have all my client consumers connect to the same node in the cluster.

I assume this means the MessageSucker will be pulling message from the nodes that have no consumers, to the node that has consumers.

Got it.

[clebert.suconic]

"chip_schoch" wrote:

I assume this means the MessageSucker will be pulling message from the nodes that have no consumers, to the node that has consumers.

Yep... exactly!

[timfox]

"chip_schoch" wrote:
Thank you Andy and Clebert for taking the time to explain this in detail. To reiterate, what you are saying is that if I need to have multiple consumers processing messages from a clustered queue in FIFO order, than I should have all my client consumers connect to the same node in the cluster.

The key point here, is that if you want to have true FIFO queue semantics, then there is absolutely no point in having more than one consumer - it will never give you better throughput

Think about it this way: For true FIFO, ordered processing, you take a message from the queue, you process it, when you have finished processing it, you can take another, ad infinitum...

So you can clearly see there is no point in having more than one consumer on the queue since you can never process them in parallel if you want to guarantee order.

Now, that there is only one consumer, it always makes sense for it to live on the same node, since you don't want to incur an extra network hit to deliver it to another node before processing.

Furthermore it makes zero sense to have different consumers on different nodes for the same reasons outlined above. If you want FIFO then messages *must be processed sequentially*, so if message 1 is being processed on node 1, then message 2 can be processed on node 2 until message 1 processing is done.

Again, you can see that there's absolutely no point in having multiple consumers, since you'll never get better throughput, since the processing is serialized. In fact you'll get a negative hit due to the extra overhead of moving messages around unnecessarily.

For a more technical explanation see Amdahl's law: http://en.wikipedia.org/wiki/Amdahl's_law. In the case of a strict FIFO queue, zero proportion of the work is parallelizable, so there is zero gain, in clustering it.

Clustering will only help you to improve throughput if at least some of your work load is parallelizable.