5 Replies Latest reply on Mar 12, 2010 9:54 AM by davsclaus

    Use of Aggregator EIP, JMS Consumer, and CLIENT_ACK

    scranton_scranton
    scranton_scranton Mar 11, 2010 10:37 AM

      I have a Camel 1.6.1.2-fuse deployment with a route that is basically aggregating JMS messages into batches of 50 to optimize updating a back-end data system.

       

      The route (InOnly)

      1) pulls messages from an ActiveMQ queue

      2) uses an Aggregator to aggregate into 50 message batches (with a 1 min timeout for slow processing times of day)

      3) sends aggregate message to back end system

       

      The challenge is if the camel node fails (hosted in ServiceMix 3.4.0.4-fuse) due to system failure or, more likely, maintenance, then all messages in the aggregator are lost (not currently using Transactions).

       

      We've tried setting the ActiveMQComponent to CLIENT_ACK with the belief that Camel will only acknowledge the JMS message once the route has completed (the system can handle duplicate messages correctly). Basically, poor man's transactions. However, what we're seeing is that we're still losing messages, so it feels like Camel is acknowledging messages when they get into the Aggregator versus when they exit the route.

       

      Any insight into how to the Camel Aggregator and ActiveMQ CLIENT_ACK are interacting? Is there a better Camel approach to this problem?

       

      Thanks,

      Scott

      • 499 Views
      • Tags:
        • 1. Re: Use of Aggregator EIP, JMS Consumer, and CLIENT_ACK
          njiang
          njiang Mar 11, 2010 10:45 PM (in response to scranton_scranton)

          Hi,

           

          Can you have a look at your route definition ?

          As the camel-jms which is the base of camel-activemq component's CLIENT_ACK is setting on the Spring jms MessageListenerContainer.Quoting from the javadoc of AbstractMessageListenerContainer, here is how the semantics are defined:

           

          "sessionAcknowledgeMode" set to "AUTO_ACKNOWLEDGE" (default): Automatic message acknowledgment before listener execution; no redelivery in case of exception thrown.

           

          "sessionAcknowledgeMode" set to "CLIENT_ACKNOWLEDGE": Automatic message acknowledgment after successful listener execution; no redelivery in case of exception thrown.

           

          As the Aggregation will fork a new thread to keep on running the processors which follow the aggregation for the aggregated message, if the exception is thrown from these processors, the listener which is running from the camel-jms consumer will not aware it.

           

          You may need to use Transaction to do this kind of job, or set the ErrorHandler to put the message back to queue.

           

          Willem

            • Actions
          • 2. Re: Use of Aggregator EIP, JMS Consumer, and CLIENT_ACK
            davsclaus
            davsclaus Mar 12, 2010 12:44 AM (in response to scranton_scranton)

            AFAIK you cannot use Spring DMLC for batch like functionality.

             

            You need to do this with more manual JMS code. I remember Rob / Charles moulliard posted some code like this in either this forum or the camel mailinglist.

              • Actions
            • 3. Re: Use of Aggregator EIP, JMS Consumer, and CLIENT_ACK
              scranton_scranton
              scranton_scranton Mar 12, 2010 9:34 AM (in response to njiang)

              Willem,

               

              So I get that a thrown Exception could be a problem in this scenario, and probably the Aggregator timing out is done via an exception. My assumption is that under the "normal" scenario where the Aggregator is just waiting for 50 messages, that no exception is thrown as part of that.

               

              Does the fact that the Aggregator use a different thread trigger client_ack for each message being aggregated?

               

              Thanks for the help,

              Scott

                • Actions
              • 4. Re: Use of Aggregator EIP, JMS Consumer, and CLIENT_ACK
                scranton_scranton
                scranton_scranton Mar 12, 2010 9:39 AM (in response to davsclaus)

                Claus,

                 

                Are you saying that the underlying Spring DMLC cannot handle having multiple interactions going at the same time? That is, Spring DMLC doesn't allow Camel to have multiple not-ACKed messages in flight as part of a single route?

                 

                If so, given your and Willem's comments it sounds like if we want to prevent message loss, our only option would be XA transactions?

                 

                Thanks,

                Scott

                  • Actions
                • 5. Re: Use of Aggregator EIP, JMS Consumer, and CLIENT_ACK
                  davsclaus
                  davsclaus Mar 12, 2010 9:54 AM (in response to scranton_scranton)

                  I am not 100% sure. But my impressions is that DMLC is designed as a event driven consumer, eg like EJB MessageDrivenBean.

                   

                  And each message consumed is like an isolated event.

                   

                  Having batch like / multiple consumed message in one TX is not something I think DMLC can do.

                   

                  For that you must write code where you do something

                   

                  1) all the usual JMS bla bla about ConnectionFactory

                  2) loop and get the X number of messages you want

                  3) ACK the messages

                  4) cleanup the JMS stuff eg connection factory and whatnot

                   

                  I may be wrong.

                   

                  Go ask Rob on irc as he knows more about this

                    • Actions