Duplicate messages

[mkprim]

Hi I'm using JBoss 3.2.3, to run an application that uses JBossMQ as it's JMS provider.
The performance is not bad, however we've been capturing the network packets using a sniffer (Ethereal).
We've seen that when the server sends an object message to the client (using a TemporaryQueue established at the beggining of the app), the client resends back the same message to the server.
The JMS options are:
- TTL = 0
- NON_PERSISTENT (using messageCache)
- MaxRedelivery = 10
- priority = 4.

We know for sure it's not a network problem, but an application one. And our app is not sending back the messages.
If anyone knows what this could be, or how to fix it, I would greatly appreciate your help.
Thanks a lot,
Marcelo

[genman]

I wouldn't concern yourself with messages appearing twice or going back the other way in the network trace, unless they show up in your application as duplicates. It could be a performance issue, however.

[mkprim]

Already know that it does not have impact in my application.
What I NEED is to lower bandwidth consumption, so I'm trying to discover why this is happening, and how to fix it.
Thanks,
Marcelo

[mkprim]

I did a simple test to discard the fact that my app was adding some noise.
I made a QueueSender (sends a BytesMessage to the JBoss queue/testQueue) and a QueueReceiver (consumes messages from the same queue).
I captured the traffic between those machines, and the result was the same.

I here add the frames capture made with ethereal (the sender's ip is 192.168.7.121, the receiver's ip is 192.168.7.192, jboss jms server running on 192.168.7.121:8093)

source 192.168.7.121 dest 192.168.7.192 8093 > 40662

0000 00 08 a1 55 cb e3 00 10 4b 8a 87 9e 08 00 45 00 ...U.... K.....E.
0010 00 9f 2e 55 40 00 80 06 3b 7a c0 a8 07 79 c0 a8 ...U@... ;z...y..
0020 07 c0 1f 9d 9e d6 87 16 42 a4 05 f7 66 5b 80 18 ........ B...f[..
0030 fa 29 5a e5 00 00 01 01 08 0a 00 35 db 44 03 66 .)Z..... ...5.D.f
0040 96 c9 77 68 19 80 00 00 21 00 00 00 00 01 02 02 ..wh.... !.......
0050 00 09 74 65 73 74 51 75 65 75 65 00 00 00 02 00 ..testQu eue.....
0060 00 00 00 00 00 00 00 00 00 00 04 07 00 13 49 44 ........ ......ID
0070 3a 33 2d 31 31 30 39 39 35 35 32 35 37 31 38 36 :3-11099 55257186
0080 33 00 00 01 02 6e 7d 23 62 01 09 ff ff ff ff 00 3....n}# b.......
0090 09 00 00 01 07 00 04 49 44 3a 33 00 00 00 00 00 .......I D:3.....
00a0 00 00 04 48 6f 6c 61 01 80 00 00 00 79 ...Hola. ....y

source 192.168.7.192 dest 192.168.7.121 40662 > 8093

0000 00 10 4b 8a 87 9e 00 08 a1 55 cb e3 08 00 45 00 ..K..... .U....E.
0010 00 9f b4 0c 40 00 40 06 f5 c2 c0 a8 07 c0 c0 a8 ....@.@. ........
0020 07 79 9e d6 1f 9d 05 f7 66 5b 87 16 43 0f 80 18 .y...... f[..C...
0030 16 d0 3c de 00 00 01 01 08 0a 03 66 97 bf 00 35 ..<..... ...f...5
0040 db 44 77 68 19 80 00 00 21 00 00 00 00 01 02 02 .Dwh.... !.......
0050 00 09 74 65 73 74 51 75 65 75 65 00 00 00 02 00 ..testQu eue.....
0060 00 00 00 00 00 00 00 00 00 00 04 07 00 13 49 44 ........ ......ID
0070 3a 33 2d 31 31 30 39 39 35 35 32 35 37 31 38 36 :3-11099 55257186
0080 33 00 00 01 02 6e 7d 23 62 01 09 ff ff ff ff 00 3....n}# b.......
0090 09 00 00 01 07 00 04 49 44 3a 33 00 00 00 00 00 .......I D:3.....
00a0 00 00 04 48 6f 6c 61 01 80 00 00 00 79 ...Hola. ....y
source 192.168.7.192 dest 192.168.7.121 40662 > 8093

0000 00 10 4b 8a 87 9e 00 08 a1 55 cb e3 08 00 45 00 ..K..... .U....E.
0010 00 64 b4 0d 40 00 40 06 f5 fc c0 a8 07 c0 c0 a8 .d..@.@. ........
0020 07 79 9e d6 1f 9d 05 f7 66 c6 87 16 43 0f 80 18 .y...... f...C...
0030 16 d0 b9 12 00 00 01 01 08 0a 03 66 97 bf 00 35 ........ ...f...5
0040 db 44 77 2d 01 00 00 00 3b 00 01 01 02 00 09 74 .Dw-.... ;......t
0050 65 73 74 51 75 65 75 65 00 13 49 44 3a 33 2d 31 estQueue ..ID:3-1
0060 31 30 39 39 35 35 32 35 37 31 38 36 33 80 00 00 10995525 71863...
0070 00 79 .y

source 192.168.7.192 dest 192.168.7.121 40662 > 8093

0000 00 10 4b 8a 87 9e 00 08 a1 55 cb e3 08 00 45 00 ..K..... .U....E.
0010 00 4a b4 0e 40 00 40 06 f6 15 c0 a8 07 c0 c0 a8 .J..@.@. ........
0020 07 79 9e d6 1f 9d 05 f7 66 f6 87 16 43 19 80 18 .y...... f...C...
0030 16 d0 22 88 00 00 01 01 08 0a 03 66 97 bf 00 35 .."..... ...f...5
0040 db 44 77 13 0d 00 00 00 3c 00 80 00 00 00 00 00 .Dw..... <.......
0050 00 00 00 00 00 00 00 79 .......y

Thanks a lot,
Marcelo.

PS: Maybe I could add this to JBossMQ JIRA? I will look in the sources to see if i can fix this problem....

[genman]

Could you please? Or even better, fix it :-) and send a patch.

[adrian.brock]

Moderated: Test it with an up-to-date version before reporting bugs.

My guess is this is OIL, which we don't even use anymore.

[mkprim]

Tested also on JBoss 4.01, using UIL2 connection.
For discarding a network problem, also tested on different networks.
The same problem goes on.
It's a real bug, and actually i'm debugging JBossMQ code.
If anyone wants to help, it will be welcomed.
Thanks,
Marcelo

[starksm64]

Create a bug report in jira with the testcase your using to see this behavior.
http://jira.jboss.com

[mkprim]

OK, I've done some debugging, and inspecting the trace generated by the client (the receiver/consumer) gone through the code.
Now, the code in the class org.jboss.mq.il.uil2.UILClientILService, in the method handleMsg we saw this:

 switch( msgType )
 {
 case m_receiveRequest:
 ReceiveRequestMsg rmsg = (ReceiveRequestMsg) msg;
 ReceiveRequest[] messages = rmsg.getMessages();
 connection.asynchDeliver(messages);
 socketMgr.sendReply(msg);
 break;
 .......
 }

The line in red, is sending the full message back when a receiveRequest message is received. That's the part that seems to be resending the message back.
So, I've commented this line and recompiled the jboss sources. I took the jbossmq.jar for the clients, rerun the test, and everithing worked just fine, and the duplicate messages disappeared.
I wanted to know if I can just do this, or it will break something.
What's the purpose of returnin the message, even if the message is one way?
Thanks,
Marcelo

[adrian.brock]

Good catch, although your solution is wrong. :-)

If you don't reply to the message, it will gradually build up
on the server as an unreplied message and you will run out of memory.
You will also be blocking delivery threads waiting for the reply.

The sendReply should be trimming the message (it is missing from
org.jboss.mq.il.uil2.msgs.ReceiveRequestMsg)

 public void trimReply()
 {
 messages = null;
 }

However, your solution brings an interesting point.
The server does not need to know at this point whether the messages were actually
received (that is handled by acknowledgements).

It only needs to know whether there is an error communicating with the
client so you are not continually writing messages into a broken connection.

So a better fix is to send the ReceiveRequest as a "one-way" message:
UILClientIL

 public void receive(ReceiveRequest messages[])
 throws Exception
 {
 ReceiveRequestMsg msg = new ReceiveRequestMsg(messages);
- socketMgr.sendMessage(msg);
+ socketMgr.sendReply(msg);
 }

UILClientILService

 case m_receiveRequest:
 ReceiveRequestMsg rmsg = (ReceiveRequestMsg) msg;
 ReceiveRequest[] messages = rmsg.getMessages();
 connection.asynchDeliver(messages);
- socketMgr.sendReply(msg);
 break;

But then we need to add code to the WriteTask so it can signal
on the server side that the message didn't make it to the client.
Currently, this is given to the thread waiting for the response:

 if (msg != null)
 {
 msg.setError(e);
 synchronized (msg)
 {
 msg.notify();
 }
 }

[adrian.brock]

Analyzing the path, we are ok to send it as one-way
since if the WriteTask spots an error, it will log it and stop the ReadTask.
The ReadTask will then do connectionClosing() to tell the main broker
that the client has vanished.
There is no need to report the error back to the delivery thread.
This will allow the delivery thread to be recycled more quickly.