MDB container prefetching messages?

[georgewinters]

Howdy!

I have two JBoss 3.2.3 servers each running one instance of a MDB. Both are servicing the same remote queue which resides on a third machine running JBoss 3.2.3. It looks like the client machines are prefetching messages. Here's what I'm seeing:

1. Message #1 is sent to the queue.
2. Server A begins processing mesage #1.
3. Message #2 is sent to the queue.
4. Server A is busy processing message #1, server B is idle, and the JMX console shows the queue is empty. Where is message #2?
5. Server A finishes processing message #1.
6. Server A begins processing message #2. - Oh there it is! Why didn't Server B get it?

If I configure the servers to each have two instances of the MDB then the first server eats three messages before the second server receives any.

I can understand that if messages were processed quite quickly that this behavior may be necessary to sustain maximum throughput. However, for my application this is very bad behavior because the processing time for an individual message can vary greatly and are always from different users. I don't want one user's request to be held up for an extended period of time while another server sits idle.

I have looked around for a setting that might allow me to configure this but to no avail. Is this a feature or a bug? Is there a way to work around it?

Thanks for reading this far. If you have some bit of relevant information *please* reply with it.

George

[adrian.brock]

Somebody else reported this problem.
It is apparantly due to the use of HashSet for receivers in BasicQueue,
he suggested changing it to an ArrayList but I haven't had time to properly test it.
There are obvious differences in the behaviour of a Set and a List.

[georgewinters]

I changed the receivers collection in BasicQueue from a HashSet to an ArrayList and it doesn't fix the problem it just changes it around some. Using the same format as I used previously:

1. Message #1 is sent to the queue.
2. Server A begins processing message #1.
3. Message #2 is sent to the queue.
4. Server B begins processing message #2.
5. Message #3 is sent to the queue.
6. Server B finishes processing message #2 and sits idle. (Where is message #3?)
7. Server A finishes processing message #1.
8. Server A begins processing message #3. (Oh there it is! Why didn't Server B get it?)

I have poked through the code some and it appears that immediately upon giving Message #1 to Server A, Server A gets resubscribed and added to the receivers. Thus, it doesn't really matter how receivers is organized (HashSet or ArrayList) it can still receive another message before it has acknowledged the message it was already given.

What I have not been able to figure out is *why* it gets immediately resubscribed to the queue. Is Server A initiating the resubscription or is it within the JMS server? Any thoughts would be appreciated. I can spend a little more time on this before I revert to a fairly ugly work-around. The work-around I'm thinking of is to send a no-op message after each real message. This should get around the problem.

[adrian.brock]

When using MDBs you have a pool of sessions that need loading.

The default is 15.

So the MDB will try to read 16 messages, 15 to load the sessions and one for readahead
for when one of the sessions has finished processing its current message.
Only when all the sessions are loaded and the one readahead has been done will it not
re-ask the server for a message.

The problem the other guy was seeing with the HashSet was is that it is read
in hashCode order so it favours receivers with lower hashCodes.
With the ArrayList, it should be "round robining". What are you seeing?
Add a
log.debug("Receivers: " + receivers);
in addReceiver, receive and internalAddMessage

I'm interested in fixing this, but it is not a high priority to me.

Regards,
Adrian

[georgewinters]

Changing receivers to an ArrayList does make it round robin the messages. What is causing me headaches isn't the order the messages are received but rather the side affects of the read ahead.

I have configured the pool of sessions to one and am seeing cases where a message that will process very quickly is stuck behind a message that takes a long time to process. With receivers as a HashSet, one client reads both messages (one to process and one read ahead) while other clients sit idle. Changing to a round robin scheme just delays the problem until the case where more messages are received than there are servers. Then the same problem occurs - messages that can be processed quickly and are unlucky get stuck behind messages that take longer to process.

How do I go about disabling the read ahead?

[adrian.brock]

It is not currently configurable.

The code is in org.jboss.mq.SpyConnectionConsumer.run()

It basically reads (pseudo code):

getNextMessage()
getServerSession()
runTheSession()

The code isn't quite as straightforward as above.

To avoid the readahead, you would need to reverse the order of getNextMessage()
and getServerSession() so it waits for sessions to become available before
receiving messages.

Removing the readahead would increase the latency,
because you only ask the jms server for a message (in general the most expensive
operation) after the previous request finishes.

If I understand you correctly, I think your mistake is in your design.
If one of your requirements is the response time,
you should not mix long and short running processing on the same
queue. In fact you should not be using asynchronous designs in the first place.

[klavergne]

I am having a similar problem. I have one JBoss instance that is only acting as a JMS server. I then have two other instances that each have an MDB that is listening to a queue on the JMS server. The last one to connect is the only one that receives messages. Each message should also be processed by only one MDB.

Should I modify the BasicQueue as mentioned before, or is there some other way to make sure that all the MDBs that are listening on the queue receive messages (not a topic).

I am using JBoss 3.2.3. Is it possible that this has been fixed in a later version?

Thanks.

Kevin

[genman]

Yes. Pour over the release notes for 3.2.4 or 3.2.5.