JMS Distributed Destinations

[malmit]

Currently, I am building a system which must have a messaging system which can support 100 - 100KB messages/second. From what I've implemented in my prototypes the max number of messages that I can send via clustered HA JMS is about 60 persistent messages. In the current implementation of JBossMQ, a single node (destination) in the cluster handles all messages which seems to be causing the congestion in the entire system.

I believe that in the future release of JBoss Messaging the distributed destination feature will be implemented. Will distributed destinations (destinations will be distributed so that not just 1 destination is doing all of the work) help solve the slowness in my application? If so, is there an expected time frame when JBoss Messaging will be released? I would really like to use JBoss because I don't want to be stuck using something like Oracle App Server.

[genman]

Probably the bottleneck you're running into is at the database (or persistence) layer.

If you want to go with JBoss, you can use JBoss plus another messaging provider pretty easily.

[malmit]

I don't think it is a bottleneck at the database. Just for testing purposes I initialized my connection pool with 300 connections split up so that there are 3 nodes each with 100 connections. This I would imagine would be plenty of connections. This set-up results in the 60 - 100KB messages/second. I decreased the size of the message to about 30 KB and was able to get 100 30KB messages/second. And if I decreased it down to 1KB then I got around 140 - 1KB messages/second. I will try looking at other messaging products and I might even be able to wait for JBoss Messaging to be released. Hopefully that will be pretty soon :) Thanks for the help!

[genman]

The JBossMQ is backed using a messaging database. If this is slow (e.g. on a slow disk), you're going to have slow JMS. Try testing with persistence turned off, you should be see those numbers improve 10-100 times I'd imagine.

[adrian.brock]

"Probably the bottleneck you're running into is at the database (or persistence) layer."

Why not the network? In a word "bandwidth".
100 x 100KB = 10MB a second.

100Mbps Ethernet ~ 12MB a second

Even if all the packets were optimally filled, that doesn't include
* Other parts of the JMS protocol (like acknowledgements)
* Resends or delays
* Other traffic on the network (including the database traffic if it is on the same network)

[malmit]

Thanks for all of the great input! I did try sending non-persistent messages as genman suggested and it increased the number of messages sent from 60 to about 100, but that doesn't help me any since the application requires persistent message handling. And adrian you made a good point about the bandwidth.

One final question to close out this topic is: Would JMS distributed destinations help out my situation at all or should I not even look into it when JBoss releases this feature?

[genman]

Well, Gigabit ethernet would give you enough bandwidth, then you'll have to look at faster disks, etc.

Out of curiosity, what is your application doing that requires so much bandwidth? You might want to go with a hybrid approach where files are stored in a content management system, and use pointers to the files inside the message.

[adrian.brock]

"malmit" wrote:
Would JMS distributed destinations help

Hmm....
I swamped my network so lets add more traffic to the network to implement "clustering".
Of course, this really depends upon your network topology and the quality of the router. :-)

This is also another one of my favourite "bug bears".
TCP/IP is too clever for its own good.
It makes what is essentially a single threaded device look multi-threaded because
of the packet switching.
The network card can still only deal with one packet at once and it has a maximum
throughput (bandwidth).

Anyway, your questions really belong in some network administration forum
NOT a JMS forum.

[adrian.brock]

"genman" wrote:
hybrid approach where files are stored in a content management system, and use pointers to the files inside the message.

Exactly. It is even worse when you see people trying to propogate these
large messages through the jms "chain of responsibility" anti-pattern. :-)

[malmit]

Currently the application is sending finger print images via JMS. Never thought about the hybird approach. I will definitely go this route! Thanks again for all the helpful input!