-
1. Re: Could double-acking be due to HornetQ resending a message?
fwilson_adaptavist Mar 15, 2013 1:06 PM (in response to fwilson_adaptavist)Hmm, I tried settting "max-delivery-attempts" to 0 on the queue address and that didn't seem to stop the duplicates.
-
2. Re: Could double-acking be due to HornetQ resending a message?
fwilson_adaptavist Mar 18, 2013 7:36 AM (in response to fwilson_adaptavist)I'm still investigating what is causing this problem. It seems that the 'duplicate' message is in fact a redelivery (the delivery count attribute on the ClientMessage object is > 1). Perhaps my code is not taking account of redeliveries (i.e. not acknowledging the redelivery when the first copy was acknowledged)? In fact, this sounds a bit more complicated than it ought to be and I am worried I could be going 'off track' by trying to identify redeliveries like this.
-
3. Re: Could double-acking be due to HornetQ resending a message?
fwilson_adaptavist Mar 18, 2013 8:35 AM (in response to fwilson_adaptavist)I've been meadering through this thread on my own hopefully converging on a question that perhaps someone can answer.
I think the client code I am looking at right now is pulling another message from the queue before the last message has been acknowledged.
If this is the case can someone confirm that HornetQ will schedule a redelivery of the unacknowledged message in this scenario (even if only a short period of time has passed e.g. < 1s )?
What happens if the original message is acknowledged before the redlivery is actually sent. It the redelivery still sent regardless?
Thanks
-
4. Re: Could double-acking be due to HornetQ resending a message?
ataylor Mar 19, 2013 6:35 AM (in response to fwilson_adaptavist)a message will only be redelivered if the client receives but does not ack a message, i.e. when you close it.
If you have some code to show maybe I could be of more help
-
5. Re: Could double-acking be due to HornetQ resending a message?
fwilson_adaptavist Mar 20, 2013 7:39 AM (in response to ataylor)I appear to be getting message redeliveries to the same consumer, i.e. without closing the consumer.
I'll see if I can provide some code to share.
-
6. Re: Could double-acking be due to HornetQ resending a message?
mlange Jun 27, 2013 3:15 AM (in response to fwilson_adaptavist)Frank did you figure out any solution for this, I am facing exactly the same problem...
-
7. Re: Could double-acking be due to HornetQ resending a message?
ataylor Jun 27, 2013 3:57 AM (in response to mlange)again, without some code, its impossible to say what the issue may be, if there is one.
-
8. Re: Could double-acking be due to HornetQ resending a message?
mlange Jun 27, 2013 4:41 AM (in response to ataylor)Hi Andy,
my consumer is pretty easy:
ConnectionFactory cf = .... lookup JMS Connection Factory from JNDI
QueueConnectionFactory queueCf = (QueueConnectionFactory) cf;
// JMS msg listener
DefaultMessageListener msgListener = new DefaultMessageListener("Listener " + queueName);
// starts several consumer threads
for (int count = 0; count < consumerConfig.getListenerThreadCount(); count++) {
QueueConnection connection = queueCf.createQueueConnection(user, password);
// auto acknowledge
QueueSession session = connection.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);
Queue queue = (Queue) HornetQDestinationResolver.resolveDestinationName(queueName);
MessageConsumer messageConsumer = session.createConsumer(queue, consumerConfig.getFilterString());
// start connection
connection.start();
Thread queueReceiverThread = new Thread(new MessageConsumerRunnable(messageConsumer, msgListener, eventConsumer, session));
queueReceiverThread.setDaemon(true);
queueReceiverThread.setName("Receiver" + count + "-" + queueName + " (" + session + ")");
queueReceiverThread.start();
}
// Consumer Runnable Thread
public void run() {
while (listenerRegistry.isConsumerRegistered(eventConsumer)) {
Message message = null;
message = queueReceiver.receive(2000L);
if (message != null) {
messageListener.onMessage(message);
}
}
How I test:
*3 node cluster with a backup node for each, client-side load balancing in place
*putting 200.000 messages in 10 different queues
*starting consumers on all queues (20 listener threads for each queue)
*stopping one node from the cluster --> failover to a backup node is done (during the threads consuming from all nodes)
*checking number of received messages --> there are more than 200.000 messages
*~100 messages seem to be duplicates (JMSXDeliveryCount >1), the duplicateID already exists in one of the received messages
I am completely stuck how to move on. Thanks!
-
9. Re: Could double-acking be due to HornetQ resending a message?
fwilson_adaptavist Jun 27, 2013 4:45 AM (in response to mlange)This was a while ago, but looking at my notes ensuring that messages are acknowleged before requesting the next message seemed to help cut down on occurrences of this error. I'm not saying this is the general solution (and I do not wish to contradict Andy at all). It so happened that this was the behaviour we wanted in the first place.
-
10. Re: Could double-acking be due to HornetQ resending a message?
ataylor Jun 27, 2013 9:19 AM (in response to fwilson_adaptavist)How I test:
*3 node cluster with a backup node for each, client-side load balancing in place
*putting 200.000 messages in 10 different queues
*starting consumers on all queues (20 listener threads for each queue)
*stopping one node from the cluster --> failover to a backup node is done (during the threads consuming from all nodes)
*checking number of received messages --> there are more than 200.000 messages
*~100 messages seem to be duplicates (JMSXDeliveryCount >1), the duplicateID already exists in one of the received messages
This is the behaviour I would expect, the delivery count is updated when the message is sent out for delivery (this is also persisted so survives failover), this means any messages not acknowledged when the server crashes will have a delivery counf of > 1.
-
11. Re: Could double-acking be due to HornetQ resending a message?
mlange Jun 27, 2013 9:30 AM (in response to ataylor)This is the behaviour I would expect, the delivery count is updated when the message is sent out for delivery (this is also persisted so survives failover), this means any messages not acknowledged when the server crashes will have a delivery counf of > 1.
Andy thanks for your answer. What about auto_acknowledge of messages? And why does the duplicate detection not work? In the <cluster-connection> <duplicate-detection> ist set to true. Shouldn't this avoid to receive duplicates?
Marek
-
12. Re: Could double-acking be due to HornetQ resending a message?
ataylor Jun 27, 2013 9:37 AM (in response to mlange)Andy thanks for your answer. What about auto_acknowledge of messages? And why does the duplicate detection not work? In the <cluster-connection> <duplicate-detection> ist set to true. Shouldn't this avoid to receive duplicates?
duplicate detection on the cluster connection is so that duplicate messages aren't sent twice over the bridge, its not really anything to do with a clients consumption.
-
13. Re: Could double-acking be due to HornetQ resending a message?
mlange Jun 27, 2013 9:46 AM (in response to ataylor)But setting org.hornetq.api.core.Message.HDR_DUPLICATE_DETECTION_ID on the message does not help either? Doesn't this work reliably in a clustered setup maybe?
I also wonder about "Caught exception: HornetQException[errorCode=104 message=Could not find reference on consumer 1" which comes up on the backup node.
I am trying to reproduce this by modyfying one of the examples delivered.
-
14. Re: Could double-acking be due to HornetQ resending a message?
ataylor Jun 27, 2013 9:54 AM (in response to mlange)But setting org.hornetq.api.core.Message.HDR_DUPLICATE_DETECTION_ID on the message does not help either? Doesn't this work reliably in a clustered setup maybe?
again, the duplicate detection on the message is only used on sent messages, we use these on cluster bridges, to avoud the same message being sent (not delivered)
I also wonder about "Caught exception: HornetQException[errorCode=104 message=Could not find reference on consumer 1" which comes up on the backup node.
I think you can ignore that its just a message being re acknowledged after failover.
I am trying to reproduce this by modyfying one of the examples delivered.
again, this is the behaviour i expect. messages are aut acked after the receive method has been called, this means that the same message can be redelivered to the consumer (and why there is a delivery count on the message).
If you need higher gaurantees then you need to use a jms transaction or even an xa transaction, it all depends what you need