1 2 3 Previous Next 33 Replies Latest reply on Jul 5, 2013 5:05 AM by ataylor

    Could double-acking be due to HornetQ resending a message?

    fwilson_adaptavist

      I keep getting messages such as the following (using hornetq with the queue, producer and consumer all in the same JVM).

       

      HornetQException[errorType=ILLEGAL_STATE message=HQ119057: Could not find reference on consumer ID=0, messageId = 331 queue = queue.test-message]

         at org.hornetq.core.server.impl.ServerConsumerImpl.acknowledge(ServerConsumerImpl.java:645)

         at org.hornetq.core.server.impl.ServerSessionImpl.acknowledge(ServerSessionImpl.java:624)

         at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.

         at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:616)

         at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.

         at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.

         at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(

         at org.hornetq.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:160)

         at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106)

         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)

         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)

         at java.lang.Thread.run(Thread.java:680)

       

       

      I think this is due to my program double-acking a message. It seems that I get twice as many messages out of HornetQ as I sent in.

       

      Under what circumstances will HornetQ resend a message? I have seen the documentation section [1] on detecting duplicates but there is not much of delay and I do

      not know why duplicates would be sent. I'm afraid I couldn't find anything on when messages are resent, maybe I missed something?

       

      [1] http://docs.jboss.org/hornetq/2.3.0.CR1/docs/user-manual/html/duplicate-detection.html

        • 1. Re: Could double-acking be due to HornetQ resending a message?
          fwilson_adaptavist

          Hmm, I tried settting "max-delivery-attempts" to 0 on the queue address and that didn't seem to stop the duplicates.

          • 2. Re: Could double-acking be due to HornetQ resending a message?
            fwilson_adaptavist

            I'm still investigating what is causing this problem. It seems that the 'duplicate' message is in fact a redelivery (the delivery count attribute on the ClientMessage object is > 1). Perhaps my code is not taking account of redeliveries (i.e. not acknowledging the redelivery when the first copy was acknowledged)? In fact, this sounds a bit more complicated than it ought to be and I am worried I could be going 'off track' by trying to identify redeliveries like this.

            • 3. Re: Could double-acking be due to HornetQ resending a message?
              fwilson_adaptavist

              I've been meadering through this thread on my own hopefully converging on a question that perhaps someone can answer.

               

              I think the client code I am looking at right now is pulling another message from the queue before the last message has been acknowledged.

               

              If this is the case can someone confirm that HornetQ will schedule a redelivery of the unacknowledged message in this scenario (even if only a short period of time has passed e.g. < 1s )?

               

              What happens if the original message is acknowledged before the redlivery is actually sent. It the redelivery still sent regardless?

               

              Thanks

              • 4. Re: Could double-acking be due to HornetQ resending a message?
                ataylor

                a message will only be redelivered if the client receives but does not ack a message, i.e. when you close it.

                 

                If you have some code to show maybe I could be of more help

                • 5. Re: Could double-acking be due to HornetQ resending a message?
                  fwilson_adaptavist

                  I appear to be getting message redeliveries to the same consumer, i.e. without closing the consumer.

                   

                  I'll see if I can provide some code to share.

                  • 6. Re: Could double-acking be due to HornetQ resending a message?
                    mlange

                    Frank did you figure out any solution for this, I am facing exactly the same problem...

                    • 7. Re: Could double-acking be due to HornetQ resending a message?
                      ataylor

                      again, without some code, its impossible to say what the issue may be, if there is one.

                      • 8. Re: Could double-acking be due to HornetQ resending a message?
                        mlange

                        Hi Andy,

                         

                        my consumer is pretty easy:

                         

                        ConnectionFactory cf = .... lookup JMS Connection Factory from JNDI

                        QueueConnectionFactory queueCf = (QueueConnectionFactory) cf;

                         

                        // JMS msg listener

                        DefaultMessageListener msgListener = new DefaultMessageListener("Listener " + queueName);

                         

                        // starts several consumer threads

                        for (int count = 0; count < consumerConfig.getListenerThreadCount(); count++) {

                          QueueConnection connection = queueCf.createQueueConnection(user, password);

                         

                          // auto acknowledge

                          QueueSession session = connection.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);

                          Queue queue = (Queue) HornetQDestinationResolver.resolveDestinationName(queueName);

                          MessageConsumer messageConsumer = session.createConsumer(queue, consumerConfig.getFilterString());

                         

                          // start connection

                          connection.start();

                         

                          Thread queueReceiverThread = new Thread(new MessageConsumerRunnable(messageConsumer, msgListener, eventConsumer, session));

                          queueReceiverThread.setDaemon(true);

                          queueReceiverThread.setName("Receiver" + count + "-" + queueName + " (" + session + ")");

                          queueReceiverThread.start();

                        }

                         

                        // Consumer Runnable Thread

                        public void run() {

                          while (listenerRegistry.isConsumerRegistered(eventConsumer)) {

                             Message message = null;

                             message = queueReceiver.receive(2000L);

                         

                             if (message != null) {

                               messageListener.onMessage(message);

                            }

                        }

                         

                        How I test:

                        *3 node cluster with a backup node for each, client-side load balancing in place

                        *putting 200.000 messages in 10 different queues

                        *starting consumers on all queues (20 listener threads for each queue)

                        *stopping one node from the cluster --> failover to a backup node is done (during the threads consuming from all nodes)

                        *checking number of received messages --> there are more than 200.000 messages

                        *~100 messages seem to be duplicates (JMSXDeliveryCount >1), the duplicateID already exists in one of the received messages

                         

                        I am completely stuck how to move on. Thanks!

                        • 9. Re: Could double-acking be due to HornetQ resending a message?
                          fwilson_adaptavist

                          This was a while ago, but looking at my notes ensuring that messages are acknowleged before requesting the next message seemed to help cut down on occurrences of this error. I'm not saying this is the general solution (and I do not wish to contradict Andy at all). It so happened that this was the behaviour we wanted in the first place.

                          • 10. Re: Could double-acking be due to HornetQ resending a message?
                            ataylor

                            How I test:

                            *3 node cluster with a backup node for each, client-side load balancing in place

                            *putting 200.000 messages in 10 different queues

                            *starting consumers on all queues (20 listener threads for each queue)

                            *stopping one node from the cluster --> failover to a backup node is done (during the threads consuming from all nodes)

                            *checking number of received messages --> there are more than 200.000 messages

                            *~100 messages seem to be duplicates (JMSXDeliveryCount >1), the duplicateID already exists in one of the received messages

                            This is the behaviour I would expect, the delivery count is updated when the message is sent out for delivery (this is also persisted so survives failover), this means any messages not acknowledged when the server crashes will have a delivery counf of > 1.

                            • 11. Re: Could double-acking be due to HornetQ resending a message?
                              mlange

                              This is the behaviour I would expect, the delivery count is updated when the message is sent out for delivery (this is also persisted so survives failover), this means any messages not acknowledged when the server crashes will have a delivery counf of > 1.

                               

                              Andy thanks for your answer. What about auto_acknowledge of messages? And why does the duplicate detection not work? In the <cluster-connection> <duplicate-detection> ist set to true. Shouldn't this avoid to receive duplicates?

                               

                              Marek

                              • 12. Re: Could double-acking be due to HornetQ resending a message?
                                ataylor

                                Andy thanks for your answer. What about auto_acknowledge of messages? And why does the duplicate detection not work? In the <cluster-connection> <duplicate-detection> ist set to true. Shouldn't this avoid to receive duplicates?

                                duplicate detection on the cluster connection is so that duplicate messages aren't sent twice over the bridge, its not really anything to do with a clients consumption.

                                • 13. Re: Could double-acking be due to HornetQ resending a message?
                                  mlange

                                  But setting org.hornetq.api.core.Message.HDR_DUPLICATE_DETECTION_ID on the message does not help either? Doesn't this work reliably in a clustered setup maybe?

                                   

                                  I also wonder about "Caught exception: HornetQException[errorCode=104 message=Could not find reference on consumer 1" which comes up on the backup node.

                                   

                                  I am trying to reproduce this by modyfying one of the examples delivered.

                                  • 14. Re: Could double-acking be due to HornetQ resending a message?
                                    ataylor

                                    But setting org.hornetq.api.core.Message.HDR_DUPLICATE_DETECTION_ID on the message does not help either? Doesn't this work reliably in a clustered setup maybe?

                                    again, the duplicate detection on the message is only used on sent messages, we use these on cluster bridges, to avoud the same message being sent (not delivered)

                                    I also wonder about "Caught exception: HornetQException[errorCode=104 message=Could not find reference on consumer 1" which comes up on the backup node.

                                    I think you can ignore that its just a message being re acknowledged after failover.

                                     

                                    I am trying to reproduce this by modyfying one of the examples delivered.

                                    again, this is the behaviour i expect. messages are aut acked after the receive method has been called, this means that the same message can be redelivered to the consumer (and why there is a delivery count on the message).

                                     

                                    If you need higher gaurantees then you need to use a jms transaction or even an xa transaction, it all depends what you need

                                    1 2 3 Previous Next