-
1. Re: Sequential processing of messages with the same Key
adrian.brock Jul 24, 2007 12:01 PM (in response to mzeijen)FAQ: (Not actually in the JBoss FAQ, but is probably in Sun's JMS FAQ, it's in the spec).
FAQ == Asked and answered many times before. -
2. Re: Sequential processing of messages with the same Key
mzeijen Jul 25, 2007 2:24 AM (in response to mzeijen)I did search and I found something specific for Websphere but nothing in general or specific for JBoss.
In the Sun FAQ, the Java EE 5 Tutorial and on Google I didn't find the solution.
That is why I ask here, because I didn't find it... -
3. Re: Sequential processing of messages with the same Key
adrian.brock Jul 25, 2007 3:09 PM (in response to mzeijen)Ok, since I haven't answered this question this month, I'll answer it again,
but only in the short version (well actually with the full caveats in case
somebody else figures out where the search engine is - its in the top right by the way :-).
What you are asking for is the default behaviour, but you need to think about it.
Messages are delivered in the order they are sent.
But with the following caveats.
1) They all have the same priority
2) The receiver is not filtering messages (i.e. ignoring some messages)
3) There is only one sender (senders can race with each other when sending)
4) There is only one receiver on the queue (messages could go to different receivers
if there are multiple) - not relevant for topics
5) A message is not NACKed (negatively acknowledged) - if a message is Nacked
and redelivered then all bets are off
6) You don't use scheduled delivery
So the answer to your question is (partially) to create topic subscriptions with a selector
for each customer, but be prepared for redelivery not respecting the order.
MessageListener instances (client side) can be attached to multiple subscriptions.
And JBoss supports multiple subscriptions for an MDB (not in the spec)
by configurating multiple invoker-proxy-(bindings) in jboss.xml
How you avoid the senders racing with each other is something you'll have to
resolve yourself (if that is relevant), probably with some kind of shared lock
on who can send messages. -
4. Re: Sequential processing of messages with the same Key
adrian.brock Jul 25, 2007 3:14 PM (in response to mzeijen)"adrian@jboss.org" wrote:
And JBoss supports multiple subscriptions for an MDB (not in the spec)
by configurating multiple invoker-proxy-(bindings) in jboss.xml
And before you ask, you'll need to configure the MDB singleton otherwise it
behave like (4 - multiple receivers - even for topics :-) -
5. Re: Sequential processing of messages with the same Key
mzeijen Jul 26, 2007 3:09 AM (in response to mzeijen)Thanks for taking the time for the explanation. Maybe I was looking for the wrong words (like: sequential jms) because I did use the search engine... I searched for an hour without finding anything useful. Please don't think that I easily post such questions on this forum because I know that you JBoss guys have a lot to do.
I need to study you solution a bit more because I don't understand several things about it. Like:
- Why do I need a topic? Isn't a queue enough because a message only needs to be processed by one receiver?
- Do I need to configure a invoke-proxy-binding for every customer? I do have 700.000 customers, and those can be added dynamically...
I thought that the solution would be like a cluster wide locking mechanism. This is what I mean:
- A message comes in to the message queue with a certain customer id property
- For this id a lock is acquired via the cluster wide singleton locking mechanism by the MB
- The next message for this id is received while the other is still being processed
- The process for the second message tries to acquire a lock, but needs to wait for the other process to finish for this id.
- The moment the lock is released the next process in line receives the lock and can start processing. In this case the second process.
Is this a stupid idea? Am I missing something about the whole JMS technology? -
6. Re: Sequential processing of messages with the same Key
adrian.brock Jul 26, 2007 12:14 PM (in response to mzeijen)"mzeijen" wrote:
- Why do I need a topic? Isn't a queue enough because a message only needs to be processed by one receiver?
Because then you need multiple receivers and selectors to get concurrency.
Contested queues with selectors are an anti-pattern for any reasonable
volume of messages (again see other posts).
- Do I need to configure a invoke-proxy-binding for every customer? I do have 700.000 customers, and those can be added dynamically...
You don't create 700,000 subscriptions that would be a management nightmare. ;-)
It would also require every send of a message checking 700,000 selectors to see
which subscription matches the message.
You use a "like", e.g. all customers beginning with A, B, C, etc.
to divide them into groups according to how much concurreny you need.
Or better yet, you use a "hash" of the customer id if you want a better spread.
I thought that the solution would be like a cluster wide locking mechanism. This is what I mean:
- A message comes in to the message queue with a certain customer id property
- For this id a lock is acquired via the cluster wide singleton locking mechanism by the MB
- The next message for this id is received while the other is still being processed
- The process for the second message tries to acquire a lock, but needs to wait for the other process to finish for this id.
- The moment the lock is released the next process in line receives the lock and can start processing. In this case the second process.
Is this a stupid idea? Am I missing something about the whole JMS technology?
Sounds stupid to me. :-) If I only because I doubt it would scale.
JMS is about asynchronicity, so you need to consider an asynchronous design
(if JMS is even relevant for what you are trying to achieve).
Think of JMS as more like a set of pipes and valves that let you control
where messages flow from and to. You want to direct all messages of a particular
type to a particular destination so they flow "sequentially" down that pipe. -
7. Re: Sequential processing of messages with the same Key
mzeijen Jul 27, 2007 2:47 AM (in response to mzeijen)You don't create 700,000 subscriptions that would be a management nightmare. ;-)
It would also require every send of a message checking 700,000 selectors to see
which subscription matches the message.
You use a "like", e.g. all customers beginning with A, B, C, etc.
to divide them into groups according to how much concurreny you need.
Or better yet, you use a "hash" of the customer id if you want a better spread.
I am missing something totally. A queue can process multiple messages at once, depending on the size of the thread pool, right?. Doesn't this create concurrency? If I understood the whole JMS clustering thing then the cluster should make sure that those processes get distributed one the cluster. If I want to make it more scalable then I only make sure that the queue can be processed by more threads and that more machines are available on the cluster. Lets forget about the DB for now, because that will become the bottleneck eventually (if you don't let it scale ;)).
But I see where you are going. Doesn't your solution also mean that a subscription can only have one thread to make sure that it processes all the messages sequentially? Because that would indeed be the solution. -
8. Re: Sequential processing of messages with the same Key
adrian.brock Jul 27, 2007 11:08 AM (in response to mzeijen)"mzeijen" wrote:
Doesn't your solution also mean that a subscription can only have one thread to make sure that it processes all the messages sequentially? Because that would indeed be the solution.
That's what I said above in the "before you ask" post about MDB singletons.
Let's say you have 26 durable topic subscriptions one for A, B, C, D, etc.
Then you create 26 MDBs to process those messages (single threadedly for each).
In a cluster you put the MDB in deploy-hasingleton or use some other
mechanism to distribute them across the cluster but making sure the MDB
for clients beginning with "A" only starts on one node. -
9. Re: Sequential processing of messages with the same Key
mzeijen Jul 27, 2007 11:15 AM (in response to mzeijen)Yeah, sorry you are right. I should have remembered that.
Is this solution only based on JBoss MQ or is this the general solution? Does the JBoss Messaging give me more possibilities? -
10. Re: Sequential processing of messages with the same Key
adrian.brock Jul 30, 2007 10:37 AM (in response to mzeijen)"mzeijen" wrote:
Is this solution only based on JBoss MQ or is this the general solution? Does the JBoss Messaging give me more possibilities?
It's just the JMS spec, at no point are any features specific to JBossMQ used.
I know JBoss Messaging tries harder than JBossMQ to solve the ordering problem
for NACKed messages. -
11. Re: Sequential processing of messages with the same Key
mzeijen Jul 31, 2007 2:28 AM (in response to mzeijen)Thank you for your help and your patience.