1 2 Previous Next 24 Replies Latest reply on Aug 26, 2010 9:23 AM by timfox

    HornetQ (in AS trunk) ends up in infinite redeliveries

    jaikiran

      Slightly related to this discussion http://community.jboss.org/message/537904#537904

       

      We have got a MDB, in EJB3 testsuite, which does this (intentionally):

       

      public void onMessage(Message message)
         { 
              System.out.println("*** DlqTestMDB onMessage, delivery count=" + (++count) + " is redelivered? " + message.getJMSRedelivered());
              throw new RuntimeException("force resend");
         }
      
      

       

      The HornetQ docs says that the default max delivery count is 10. However in AS trunk, I see that this message gets redelivered infinitely. Also, the message.getJMSRedelivered returns false always. Looking at HornetQ code (QueueImpl#checkDLQ), I can see that the maxDeliveryCount is picked up as 10 (the default), but the MessageReference#getDeliveryCount always returns 0 and as a result the message is re-delivered infinitely and never gets sent to DLQ.

       

      This looks somehow related to the persist-delivery-count-before-delivery config mentioned in the hornetq doc:

       

      21.3. Delivery Count Persistence

       

      In normal use, HornetQ does not update delivery count persistently until a message is rolled back (i.e. the delivery count is not updated before the message is delivered to the consumer). In most messaging use cases, the messages are consumed, acknowledged and forgotten as soon as they are consumed. In these cases, updating the delivery count persistently before delivering the message would add an extra persistent step for each message delivered, implying a significant performance penalty.

       

      However, if the delivery count is not updated persistently before the message delivery happens, in the event of a server crash, messages might have been delivered but that will not have been reflected in the delivery count. During the recovery phase, the server will not have knowledge of that and will deliver the message with redelivered set to false while it should be true.

       

      As this behavior breaks strict JMS semantics, HornetQ allows to persist delivery count before message delivery but disabled it by default for performance implications.

       

      To enable it, set persist-delivery-count-before-delivery to true in hornetq-configuration.xml:

       

      <persist-delivery-count-before-delivery>true</persist-delivery-count-before-delivery>
          

      Looking at the AS hornetq-configuration.xml, I don't see this being set. I think this should be set in the AS by default, or else it breaks the JMS semantics. I locally, added that property, but haven't seen any difference to the behaviour.

       

      Am I missing something?

        • 1. Re: HornetQ (in AS trunk) ends up in infinite redeliveries
          timfox

          Can you provide a failing test case that someone can take a look at?

           

          BTW, this won't be related to persist-delivery-count-before-delivery. That just copes with an edge case if the server crashes at a certain point. It should only be enabled if the user wants to be really, anal. It's disabled by default because it will reduce performance drastically.

          • 2. Re: HornetQ (in AS trunk) ends up in infinite redeliveries
            timfox

            I suspect HornetQ isn't actually redelivering this message - it's getting redelivered somehow in the JCA / MDB layer

            • 3. Re: HornetQ (in AS trunk) ends up in infinite redeliveries
              jaikiran

              Sorry, should have posted the testcase and the logs in my earlier post.

               

              The testcase resides in the EJB3 testsuite org.jboss.ejb3.test.mdb.unit.MDBUnitTestCase#testDlqMaxResent http://anonsvn.jboss.org/repos/jbossas/projects/ejb3/trunk/testsuite/src/test/java/org/jboss/ejb3/test/mdb/unit/MDBUnitTestCase.java

              To run the testcase:

               

              cd [EJB3_Trunk]/testsuite
              
              // set JAVA_HOME and JBOSS_HOME appropriately
              // start JBoss AS separately
              
              ant -f build-test.xml one-test -Dtest=org.jboss.ejb3.test.mdb.unit.MDBUnitTestCase
              
              

               

              The MDB in question is http://anonsvn.jboss.org/repos/jbossas/projects/ejb3/trunk/testsuite/src/test/java/org/jboss/ejb3/test/mdb/DlqTestMDB.java (you can ignore the activation config properties on that MDB, for now). Locally, I just updated the System.out.println in that MDB's onMessage() method include the redelivery count as well as a call to message.getJMSRedelivered(). Here's the output and the stacktrace (it keeps redelivery until the bean is undeployed):

               

              20:45:20,524 INFO  [STDOUT] *** DlqTestMDB onMessage, (re)delivery count=359 is redelivered? false
              20:45:20,526 ERROR [org.jboss.aspects.tx.TxPolicy] javax.ejb.EJBTransactionRolledbackException: force resend
              20:45:20,526 ERROR [org.hornetq.ra.inflow.HornetQMessageHandler] Failed to deliver message: javax.ejb.EJBTransactionRolledbackException: force resend
                  at org.jboss.ejb3.tx.Ejb3TxPolicy.handleInCallerTx(Ejb3TxPolicy.java:115) [:1.0.2]
                  at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:130) [:1.0.0.GA]
                  at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:194) [:1.0.0.GA]
                  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.0.Final]
                  at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42) [:1.0.2]
                  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.0.Final]
                  at org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:80) [:1.3.1]
                  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.0.Final]
                  at org.jboss.ejb3.BlockContainerShutdownInterceptor.invoke(BlockContainerShutdownInterceptor.java:67) [:1.3.1]
                  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.0.Final]
                  at org.jboss.ejb3.core.context.CurrentInvocationContextInterceptor.invoke(CurrentInvocationContextInterceptor.java:47) [:1.3.1]
                  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.0.Final]
                  at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67) [:1.0.1]
                  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.0.Final]
                  at org.jboss.ejb3.interceptor.EJB3TCCLInterceptor.invoke(EJB3TCCLInterceptor.java:86) [:1.3.1]
                  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.0.Final]
                  at org.jboss.ejb3.mdb.MessagingContainer.localInvoke(MessagingContainer.java:277) [:1.3.1]
                  at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.delivery(MessageInflowLocalProxy.java:299) [:1.3.1]
                  at org.jboss.ejb3.mdb.inflow.MessageInflowLocalProxy.invoke(MessageInflowLocalProxy.java:152) [:1.3.1]
                  at $Proxy158.onMessage(Unknown Source)    at org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(HornetQMessageHandler.java:249) [:]
                  at org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:804) [:]
                  at org.hornetq.core.client.impl.ClientConsumerImpl.access$100(ClientConsumerImpl.java:46) [:]
                  at org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:922) [:]
                  at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:96) [:]
                  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_16]
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_16]
                  at java.lang.Thread.run(Thread.java:619) [:1.6.0_16]
              Caused by: java.lang.RuntimeException: force resend
                  at org.jboss.ejb3.test.mdb.DlqTestMDB.onMessage(DlqTestMDB.java:59) [:]
                  at sun.reflect.GeneratedMethodAccessor281.invoke(Unknown Source) [:1.6.0_16]
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [:1.6.0_16]
                  at java.lang.reflect.Method.invoke(Method.java:597) [:1.6.0_16]
                  at org.jboss.aop.joinpoint.MethodInvocation.invokeTarget(MethodInvocation.java:122) [jboss-aop.jar:2.2.0.Final]
                  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111) [jboss-aop.jar:2.2.0.Final]
                  at org.jboss.ejb3.interceptors.container.ContainerMethodInvocationWrapper.invokeNext(ContainerMethodInvocationWrapper.java:72) [:1.0.6]
                  at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.invoke(InterceptorSequencer.java:76) [:1.0.6]
                  at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.aroundInvoke(InterceptorSequencer.java:62) [:1.0.6]
                  at sun.reflect.GeneratedMethodAccessor280.invoke(Unknown Source) [:1.6.0_16]
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [:1.6.0_16]
                  at java.lang.reflect.Method.invoke(Method.java:597) [:1.6.0_16]
                  at org.jboss.aop.advice.PerJoinpointAdvice.invoke(PerJoinpointAdvice.java:174) [jboss-aop.jar:2.2.0.Final]
                  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.0.Final]
                  at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.fillMethod(InvocationContextInterceptor.java:72) [:1.0.6]
                  at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_fillMethod_13246582.invoke(InvocationContextInterceptor_z_fillMethod_13246582.java)
                  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.0.Final]
                  at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.setup(InvocationContextInterceptor.java:88) [:1.0.6]
                  at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_setup_13246582.invoke(InvocationContextInterceptor_z_setup_13246582.java)
                  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.0.Final]
                  at org.jboss.ejb3.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:62) [:1.3.1]
                  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.0.Final]
                  at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:56) [:1.3.1]
                  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.0.Final]
                  at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47) [:1.3.1]
                  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.0.Final]
                  at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42) [:1.0.2]
                  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.0.Final]
                  at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:68) [:1.3.1]
                  at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102) [jboss-aop.jar:2.2.0.Final]
                  at org.jboss.aspects.tx.TxPolicy.invokeInCallerTx(TxPolicy.java:126) [:1.0.0.GA]
                  ... 26 more
              
              • 4. Re: HornetQ (in AS trunk) ends up in infinite redeliveries
                ataylor

                I will investigate this issue today to get to the bottom of whats happening!

                • 5. Re: HornetQ (in AS trunk) ends up in infinite redeliveries
                  jimmy.hui

                  Hi,

                   

                  Was a solution found for this problem? I'm currently forcing a transaction rollback inside my MDB (throwing a runtime exception) and I've set max redelivery to 2 however it ends up in an infinate redelivery scenario. I've tried setting the DLQMaxResent on the activation config of the MDB as well but to no affect. Any ideas on how to get max redelivery working?

                   

                  Cheers,

                  Jimmy

                  • 6. Re: HornetQ (in AS trunk) ends up in infinite redeliveries
                    jaikiran

                    I believe this was fixed in JBoss AS 6.0.0.M3. Which version of JBoss AS are you using?

                    • 7. Re: HornetQ (in AS trunk) ends up in infinite redeliveries
                      jimmy.hui

                      We are using JBoss 5.0.1GA (Enterprise version). Is there a Jira issue for this that I could possibly refer to as a support task? Or any known workarounds for JBoss 5?

                      • 8. Re: HornetQ (in AS trunk) ends up in infinite redeliveries
                        clebert.suconic

                        Maybe your config is not being passed to HornetQ. Look at HornetQ's doc regarding maxRedeliveries. Maybe you have to define the address-settings.. etc.

                         

                        http://hornetq.sourceforge.net/docs/hornetq-2.1.1.Final/user-manual/en/html/undelivered-messages.html#d0e4093

                         

                        I'm not sure we are integrating that setting from the JCA Adapter properties. I would have to look at it tomorrow (I don't have the access to the code now. If I was 20 I would know the code in top of my head.. but 10 years later.. I can't..  ( aham... a few more than 10) ). Maybe Andy Taylor will know?

                        • 9. Re: HornetQ (in AS trunk) ends up in infinite redeliveries
                          jimmy.hui

                          I played around with general testing and still no luck. I removed the max-delivery-attempts attribute to test the default (10) to see if that helps but it will continue to re-deliver the message beyond 10 attempts. I know the address-setting works for that queue as i set the redelivery-delay to 2000ms but I don't know why the max redelivery isn't working. The queue is connected to a core bridge and sending a message via the bridge and directly to the queue produces the same redelivery issue.

                          • 10. Re: HornetQ (in AS trunk) ends up in infinite redeliveries
                            andreas_back

                            Hello Jimmy,

                             

                            in our application max-delivery-attempts is set to 1 and as you can see from

                             

                                 https://community.jboss.org/message/555796#555796

                             

                            it works for us in the case of a client shut down.

                             

                            But in the special case

                             

                            *     of a server shut down and restart with a pause larger than the ClientFailureCheckPeriod

                            *     while the message is consumed in the client

                             

                            we see

                             

                            *     that the JMSXDeliveryCount is increased to values larger than 1

                            *     and that the message is redelivered to clients with this value larger than 1.

                             

                            In this case the max-delivery-attempts does seem not to work (a simple test case for this is pending).

                             

                            Best regards,

                             

                            Andreas

                            • 11. Re: HornetQ (in AS trunk) ends up in infinite redeliveries
                              jimmy.hui

                              I attempted  to get the default max-delivery working. So I copied the default-with-hornetq JBoss server profile and left all the default HornetQ 2.1.0Final configurations as is. I then deployed a simple Message Bean that would throw a RuntimeException in the onMessage function. I then sent a message to the ExpiryQueue (the test message bean was configured to listen to this queue for testing) and the bean would recognise a message was there, processes it, and then throw the runtime exception. Now I expected it to stop redelivering this message after 10 attempts but it would continue forever.

                               

                              Is there something else I'm missing here? An EJB annotation that I should be setting?

                               

                              Cheers

                              • 12. Re: HornetQ (in AS trunk) ends up in infinite redeliveries
                                andreas_back

                                Hello Jimmy,

                                 

                                you could try to connect your test MDB to a normal queue that has a dead-letter-address configured.

                                 

                                Best regards,

                                 

                                Andreas

                                • 13. Re: HornetQ (in AS trunk) ends up in infinite redeliveries
                                  andreas_back

                                  Hello Jimmy,

                                   

                                  based on the optimism that the redelivery and sending to the DLQ queue repeatedly works for our POJO clients we wanted

                                  to send a special message consumed by an MDB into the DLQ by raising a RuntimeException in the onMessage-Method

                                   

                                                  int i = 0;
                                                  if (i == 0) {
                                                      throw new java.lang.IllegalStateException("just a dummy exception to provoke DLQ-sendings");
                                                  }

                                   

                                  This has sent the JBoss 5.1.0.GA and HornetQ 2.1.1.Final combination into an endless loop, see the attachted part of the logfile - with trace enabled for HonetQ.

                                   

                                  What scares us a little bit is that the LogInterceptor reports an error and that we have an stmp-appender configured for jboss.

                                  If such a runtime-exception happened in a productive system it could consume some mail server rescources ...

                                   

                                  With best regards

                                   

                                  Andreas

                                  • 14. Re: HornetQ (in AS trunk) ends up in infinite redeliveries
                                    jimmy.hui

                                    Hi Andreas,

                                     

                                    So throwing an exception within the MDB causes an infinite redelivery pattern? Is there any way around this? I'm still finding the same behaviour on normal queues with default configurations and a fresh jboss/hornetq server with only my test application deployed in it

                                    1 2 Previous Next