-
1. Collision avoidance during redelivery
ataylor Apr 5, 2011 10:44 AM (in response to nbhatia)no, this sounds like a problem you need to solve in your app, if you explain more what your trying to acheive maybe i can suggest a solution
-
2. Collision avoidance during redelivery
nbhatia Apr 5, 2011 11:07 AM (in response to ataylor)The event in my app is the placement of an order with a quantity. The quantity can be filled from multiple different sources causing a flurry of responses along with the quantities they can offer. The application has to be able to process these responses in parallel (I do not want to create a bottleneck by limiting to a singleton MDB). This involves keeping track of a cumulative quantity that is filled in the Order object. In addition, I need to attach information about the sources that have offered to fullfill, close the order when it is filled and also create an associated (financial) transaction. I am using optimistic locking (with JPA/Hibernate), so at times I am seeing StaleObjectStateException's and LockAcquisitionException's which are causing the JMS messages to be rolled back. So my thinking is that if these messages are redivered at different times then I can process them successfully without any further exceptions. Now this can happen accidentally, but I think being able to specify some variability in redelivery will enable very clean recovery.
-
3. Collision avoidance during redelivery
ataylor Apr 5, 2011 12:55 PM (in response to nbhatia)cant you just use pessimistic locking?
-
4. Collision avoidance during redelivery
clebert.suconic Apr 5, 2011 12:59 PM (in response to nbhatia)I would do it through a single consumer (non-mdb), and batch transactions. Since you have a single row to update you cant really do it in parallel.
Maybe I would do just the update part in a single consumer.
This is something you have to do in your app though.
-
5. Collision avoidance during redelivery
nbhatia Apr 5, 2011 3:00 PM (in response to ataylor)Andy Taylor wrote:
cant you just use pessimistic locking?
Thanks for the suggestion Andy. I will give it a try. What that would mean is that bunch of MDBs will get blocked waiting for the lock, while they could be processing other unrelated orders. Hard to say what the performance tradeoff would be.
-
6. Re: Collision avoidance during redelivery
clebert.suconic Apr 5, 2011 3:07 PM (in response to nbhatia)Naresh Bhatia wrote:
Andy Taylor wrote:
cant you just use pessimistic locking?
Thanks for the suggestion Andy. I will give it a try. What that would mean is that bunch of MDBs will get blocked waiting for the lock, while they could be processing other unrelated orders. Hard to say what the performance tradeoff would be.
That's why I was suggesting you batching Transactions, and doing your processing outside of MDBs.
You could maybe create a POJO application, that would start as a MBean, and you could just batch in a single Transaction.
Your DB won't be able to sustain the load of many parallel updates on a single row. HornetQ will certainly be able to deliver more records than you can process in parallel.
Try parallelizing the components:
Say: if you need to update a single row with the totals: Make that one of your Consumers
If you need to generate records to another system: Make that another consumer.
I have worked with OLAP/Star schema Systems about 12 years ago.. that used to be my main expertise before I got into message systems, so I know what I'm talking about.
-
7. Collision avoidance during redelivery
nbhatia Apr 5, 2011 3:08 PM (in response to clebert.suconic)Clebert Suconic wrote:
I would do it through a single consumer (non-mdb), and batch transactions. Since you have a single row to update you cant really do it in parallel.
Thanks Clebert. It may come down to that. However if I have multiple mdbs they could be working on different orders without any contention. Hard to say which approach is going to give me better performance. I will just have to try under heavy load.
That also gives me another idea. May be I should try message groups. Every order has its own message group. What do you think?
-
8. Re: Collision avoidance during redelivery
clebert.suconic Apr 5, 2011 3:09 PM (in response to clebert.suconic)basically, what I'm suggesting you is this:
ClientConsumer
onMessage
.. you will need to deal with XA of course
if (count% 1000)
{
commit();
}
You will probably need to consider timeouts.
And there's a feature request I've made on having MDBs supporting batched transactions. You should give it a vote! (I can't find it now)
-
9. Re: Collision avoidance during redelivery
clebert.suconic Apr 5, 2011 3:10 PM (in response to nbhatia)Every order has its own message group. What do you think?
That would do it actually.
I had the impression you were doing a star-schema like application where you would be updating a single row.
-
10. Re: Collision avoidance during redelivery
clebert.suconic Apr 5, 2011 3:12 PM (in response to clebert.suconic)don't use the order-id as the message group though. There's a hashMap that's populated on somewhere on the Server objects.
I would maybe get the order ID you have and divide it by a factor, so you have a max number of Groups.
-
11. Collision avoidance during redelivery
nbhatia Apr 5, 2011 3:14 PM (in response to clebert.suconic)Clebert Suconic wrote:
You could maybe create a POJO application, that would start as a MBean, and you could just batch in a single Transaction.
Your DB won't be able to sustain the load of many parallel updates on a single row. HornetQ will certainly be able to deliver more records than you can process in parallel.
Try parallelizing the components:
Say: if you need to update a single row with the totals: Make that one of your Consumers
If you need to generate records to another system: Make that another consumer.
Looks like our responses crossed! But yes, good suggestions. I can tell that you have been doing this for a long time!
-
12. Collision avoidance during redelivery
nbhatia Apr 5, 2011 3:18 PM (in response to clebert.suconic)Clebert Suconic wrote:
don't use the order-id as the message group though. There's a hashMap that's populated on somewhere on the Server objects.
I would maybe get the order ID you have and divide it by a factor, so you have a max number of Groups.
This looks very promising. First on my list to try!