JMS send() slow JBoss 3.2.3

[tysmeister]

Hi,

I am running an app on JBoss 3.2.3 (SPARC 1.4.1_05-b01 VM). The application uses a number of MDBs that are a sink of P2P JMS messages.
I have noticed that the call to sender.send() can take a long time to return, and the length of time appears to be linked to the processing duration of the MDB onMessage() method (i.e. the call to send() appears to be almost synchronous wrt to the MDB that processes the message).

I am using the JVM invocation layer because everything occurs within the EJB container. The MDB components are set to Bean managed trx management, and AFAIK this should result in the container auto acknowledging receipt of messages by the MDB.

The flow of control is like this;

1. A Stateless Session EJB component sends a message to a queue Q1
2. An MDB instance MDB1 is the recipient of Q1.
3. MDB1 does its processing and then sends an arbitrarily configurable number of messages to a queue Q2. This thread then does a wait().
4. One or more MDB instances MDB2 are the recipient(s) of Q2.
5. Each MDB2 instance does its processing and then sends a message to queue Q3.
6. An MDB instance MDB3 receives the message Q3.
7. MDB3 does its processing, and then checks if it is the last thread of control that were set off as a result of the messages in step 3. If so it uses notify() to wake up the thread that is waiting in 3.
8. MDB1 checks whether it needs to send any more messages, and if so return to step 4.

A pseudo sequence diagram is as follows;

SSB -- M1 --> Q1 --> MDB1 -- M2 --> Q2 --> MDB2 -- M3 --> Q3 --> MDB3
 -- M2 --> Q2 --> MDB2 -- M3 --> Q3 --> MDB3
 -- M2 --> Q2 --> MDB2 -- M3 --> Q3 --> MDB3

The reason that it is designed this way is in the interests of flow control, and also to be able to take advantage of an SMP host with respect to the processing of M2 that can be done in parallel. Because it takes so long for the sender.send() call to return in MDB1 this benefit is reduced though.

Thanks and regards,
Andrew

[adrian.brock]

Moderated: READ THIS FIRST on how to debug this.

This is generally an anti-pattern. Everytime a message is moved from one
queue to another, it is treated as a NEW message.
Everytime you send a message it will be persisted to the persistent store.
Everytime you acknowledge a message it will deleted from the store.

So you generally end up with a system that is spending most of its time
reading and writing the store, rather than doing the business logic.

If you are still using hypersonic (don't) it does not execute queries concurrently.

MDBs are already multi-threaded (15 threads is the default).

[tysmeister]

Hi Adrian,

Thanks for your response. I am still using the default Hypersonic persistent store, as evidenced by the very large files in $JBOSS_HOME/server/default/data/hypersonic. This is a surprise to me though because I explicitly set the delivery mode in the header of the JMS message to NON_PERSISTENT, which apparently is being ignored by the sender.

The thing that surprises me most is the apparent link between the time taken for the sender.send() to return, and the time it takes for the MDB to process the message consumed as a result of the send. When the individual MDBs processing is quick, then the whole thing works very efficiently, and I can see that by configuring a arbitary number of parallel instances on an SMP host that I get better throughput.

I will try setting up our RDBMS as the persistent store if get better mileage.

Thanks and regards,
Andrew

[tysmeister]

Hi,

Ok - I have read the Wiki, and understand the problem a lot better now. Using the org.jboss.mq.pm.none.PersistenceManager in place of the default JDBC/Hypersonic combination has improved the response of the send() orders of magnitude. We can endure lost messages so this approach might be the best way to go for us.

Apparently this PM does not come bundled with 3.2.3 however, although I managed to obtain the appropriate class files from the 3.2.5 distribution.

I would be interested to know why the Oracle based PM setup does not work (as pe my other thread). I might try the latest 3.2.6 release to see if that resolves the SQLException on a write.

Thanks and regards,
Andrew