JBossDeveloper

You want to know the optimal number of consumers you need in order to process messages as fast as possible?

Let's assume you have a server machine A, and a client machine B. The consumers are created on B to consume from machine A. Let's also assume the network has infinitte bandwidth, so you're not limited by that.

We'll also assume the work done in processing each message does not block on I/O, i.e. it's CPU intensive, e.g. some String manipulation.

Let's first consider the case where the client machine only has one processor (cores). This will help us understand the problem better. We'll then generalise it to the real world case of multiple cores.

Let's assume that processing of one message one core, where that core is *only* processing that message (i.e. it's not doing anything else) takes time t.

In this case, the optimal number of consumers to have is exactly one. If you had two consumers then the core would have to take turns in processing each consumer and processing would take twice as long i.e. 2 * t.

Each core has a limited number of instructions per second it can process. Adding more consumers simply splits that fixed number of instructions across multiple consumers so does not speed anything up.

In fact, adding more consumers in this case, would actually slow things down since the processor would take extra time in context switching between the consumer threads.

Let's move to the multiple core case. Most machins these days have multiple cores, let's say number of cores is N. In this case, and assuming that work done in processing message A is not dependent on work done in processing message B, then the optimal number of consumers is N. I.e. one per core. This allows each core to be fully utilised while minimising context switching.

Caveat: If the work done in process each message ends up blocking on I/O, e.g doing some database operation. Then that processing is not using CPU cycles while it is blocking, therefore the optimal number of consumers can be higher than the number of cores. The exact amount higher depends on the ratio of blocking to non blocking time in the processing of the message.

I should say, that none of the above is specific to JMS. It's relevant to optimising the number of threads for any sort of processing on a server.

The figure of around 700k messages per second, as quoted in the peformance report pdf on the wiki, as explained in the report, was done with three machines:

1. Producer machine

2. Server machine

3. Consumer machine

There was one JMS producer on the producer machine, and there were 15 JMS consumers on the consumer machine. IIRC we got about the same throughput with 10 consumers.

Each machine had 8 cores (actual 4 cores with hyper threading which is equivalent to 8 cores), so we're pretty close to what the theory predicts. During the test all cores on server and clients were pretty much fully utilised.

The moral of the story, don't assume throwing more consumers at it will make things process more quickly.

If you tell us

a) What sort of processing you do in your listeners - is it CPU cycle crunching or is it blocking on I/O, e.g. database operations, writing/reading to files or writing/reading to sockets. If it's I/O bound what proportion of the processing time is spent in blocking on I/O?

b) How many cores/processors does your client machine have (or server if you are using the server for the client too)?

c) Is the machine doing other stuff as well as processing your messages? If so, what proportion of it's available CPU time is spent doing this other stuff?

Based on the above I can give you an estimate of the optimal number of consumers.

andjarnic wrote:

 

@Leos, ok..makes sense.. but clearly Andy/Tim are indicating that what I am saying does not make sense.

Maybe that it does not make sense to them, fortunatelly it does make sense to me.

They've both said just create more consumers.

... but did not tell you how exactly you should do this. Frankly, this is not anything HornetQ-specific - you should be able to find out the information about using JMS in many freely available documents, that's probably what they thought you would do.

Is this the "right path" to creating more consumers on a single server? Ideally I'd have a List of Session objects, loop through them, createing receivers and setting a new instance of MyServerQueueListener on each? If that is the right way to do it, what about threading?

If you would do it this way, you let the threading to be controlled by the JMS provider (its client library). Each JMS provider implements the threading in a different way - you would have to study its documentation and hope that it is described properly there.

Is that what you mean by each session can then be run in it's own thread? I am basically trying to figure out how to do some sort of consumer pool, so that my single server can handle multiple messages at once to maximize the capacity of the computer.

If you would use a J2EE server, you should study its documentation and the documentation of the JCA Resource Adapter (from HornetQ) to find out the "proper" way of doing it in the J2EE world via MDB. If you would be fine with ignoring the J2EE world, there are basically two other ways how to do it - either use the MessageListener interface and let the JMS provider handle the threading for you, or handle the threading yourself and receive the messages in each of your threads (which would have its own Session) by calling MessageConsumer#receive.

MDB's will give you pooling, transactions, security and centralised management etc. For your application i guess you need only the pooling so there will be an extra overhead for the other items. best way forward is to do some load testing and see what results you get.

Leos is correct you either create message listeners and let HQ handle the threading or use consumer.receive() and use your own threading model. If i did either of these ways i would just make sure that number sessions/consumers etc are configurable and that you can tune it at deploy time to get the best results.

basically, do some prototyping and see which way best fits your application.