12 Replies Latest reply on Apr 5, 2011 3:18 PM by nbhatia

    Collision avoidance during redelivery

    nbhatia

      I have a situation where a single event in my application causes a flurry of JMS messages. These messages all affect the same set of objects and as a result I am getting deadlocks in my database. I am afraid that using a fixed redelivery delay is simply going to postpone the problem to a later time. Is there a way to have a random redelivery delay (say from 1 to 3 seconds) in order to avoid future collisions? Perhaps add a parameter that will vary the delay in a certain range (redilivery-delay +- redelivery-delta). I have carefully coded my application so it does not rely on sequencing - redivered messages can arrive in any sequence.

        • 1. Collision avoidance during redelivery
          ataylor

          no, this sounds like a problem you need to solve in your app, if you explain more what your trying to acheive maybe i can suggest a solution

          • 2. Collision avoidance during redelivery
            nbhatia

            The event in my app is the placement of an order with a quantity. The quantity can be filled from multiple different sources causing a flurry of responses along with the quantities they can offer. The application has to be able to process these responses in parallel (I do not want to create a bottleneck by limiting to a singleton MDB). This involves keeping track of a cumulative quantity that is filled in the Order object. In addition, I need to attach information about the sources that have offered to fullfill, close the order when it is filled and also create an associated (financial) transaction. I am using optimistic locking (with JPA/Hibernate), so at times I am seeing StaleObjectStateException's and LockAcquisitionException's which are causing the JMS messages to be rolled back. So my thinking is that if these messages are redivered at different times then I can process them successfully without any further exceptions. Now this can happen accidentally, but I think being able to specify some variability in redelivery will enable very clean recovery.

            • 3. Collision avoidance during redelivery
              ataylor

              cant you just use pessimistic locking?

              • 4. Collision avoidance during redelivery
                clebert.suconic

                I would do it through a single consumer (non-mdb), and batch transactions. Since you have a single row to update you cant really do it in parallel.

                 

                 

                Maybe I would do just the update part in a single consumer.

                 

                 

                This is something you have to do in your app though.

                • 5. Collision avoidance during redelivery
                  nbhatia

                  Andy Taylor wrote:

                   

                  cant you just use pessimistic locking?

                   

                  Thanks for the suggestion Andy. I will give it a try. What that would mean is that bunch of MDBs will get blocked waiting for the lock, while they could be processing other unrelated orders. Hard to say what the performance tradeoff would be.

                  • 6. Re: Collision avoidance during redelivery
                    clebert.suconic

                    Naresh Bhatia wrote:

                     

                    Andy Taylor wrote:

                     

                    cant you just use pessimistic locking?

                     

                    Thanks for the suggestion Andy. I will give it a try. What that would mean is that bunch of MDBs will get blocked waiting for the lock, while they could be processing other unrelated orders. Hard to say what the performance tradeoff would be.

                     

                    That's why I was suggesting you batching Transactions, and doing your processing outside of MDBs.

                     

                     

                    You could maybe create a POJO application, that would start as a MBean, and you could just batch in a single Transaction.

                     

                    Your DB won't be able to sustain the load of many parallel updates on a single row. HornetQ will certainly be able to deliver more records than you can process in parallel.

                     

                     

                    Try parallelizing the components:

                     

                    Say: if you need to update a single row with the totals: Make that one of your Consumers

                    If you need to generate records to another system: Make that another consumer.

                     

                     

                     

                     

                    I have worked with OLAP/Star schema Systems about 12 years ago.. that used to be my main expertise before I got into message systems, so I know what I'm talking about.

                    • 7. Collision avoidance during redelivery
                      nbhatia

                      Clebert Suconic wrote:

                       

                      I would do it through a single consumer (non-mdb), and batch transactions. Since you have a single row to update you cant really do it in parallel.

                       

                      Thanks Clebert. It may come down to that. However if I have multiple mdbs they could be working on different orders without any contention. Hard to say which approach is going to give me better performance. I will just have to try under heavy load.

                       

                      That also gives me another idea. May be I should try message groups. Every order has its own message group. What do you think?

                      • 8. Re: Collision avoidance during redelivery
                        clebert.suconic

                        basically, what I'm suggesting you is this:

                         

                         

                         

                        ClientConsumer

                          onMessage

                            .. you will need to deal with XA of course

                         

                           if (count% 1000)

                            {

                               commit();

                            }

                         

                         

                         

                        You will probably need to consider timeouts.

                         

                         

                         

                         

                         

                        And there's a feature request I've made on having MDBs supporting batched transactions. You should give it a vote! (I can't find it now)

                        • 9. Re: Collision avoidance during redelivery
                          clebert.suconic
                          Every order has its own message group. What do you think?

                           

                           

                          That would do it actually.

                           

                           

                          I had the impression you were doing a star-schema like application where you would be updating a single row.

                          • 10. Re: Collision avoidance during redelivery
                            clebert.suconic

                            don't use the order-id as the message group though. There's a hashMap that's populated on somewhere on the Server objects.

                             

                             

                            I would maybe get the order ID you have and divide it by a factor, so you have a max number of Groups.

                            • 11. Collision avoidance during redelivery
                              nbhatia

                              Clebert Suconic wrote:

                               

                              You could maybe create a POJO application, that would start as a MBean, and you could just batch in a single Transaction.

                               

                              Your DB won't be able to sustain the load of many parallel updates on a single row. HornetQ will certainly be able to deliver more records than you can process in parallel.

                               

                              Try parallelizing the components:

                               

                              Say: if you need to update a single row with the totals: Make that one of your Consumers

                              If you need to generate records to another system: Make that another consumer.

                               

                              Looks like our responses crossed! But yes, good suggestions. I can tell that you have been doing this for a long time!

                              • 12. Collision avoidance during redelivery
                                nbhatia

                                Clebert Suconic wrote:

                                 

                                don't use the order-id as the message group though. There's a hashMap that's populated on somewhere on the Server objects.

                                 

                                 

                                I would maybe get the order ID you have and divide it by a factor, so you have a max number of Groups.

                                 

                                This looks very promising. First on my list to try!