Hornetq consume a lot of CPU resource

[changgeng]

Hi,

Our system recently start using hornetq as messaging middleware. The profiling result shows more than 30% cpu time are consumed by hornetq. Also our system throughput dropped from 80 per second to about 45 per second. For every request we are sending about 8 different message to different queues, and the consumer is running in the same system to process these message.

I attached a screenshot by jprofiler in sampling mode, and I'm seeking any suggestion to reduce the CPU usage as our system is CPU bounded. We are currently using AIO, and switching to NIO doesn't help on CPU usage.

Thank you,

changgeng

[ataylor]

Our system recently start using hornetq as messaging middleware. The profiling result shows more than 30% cpu time are consumed by hornetq.

How much CPU is used is dependant on lots of things, its hard to comment on this, how fast your producers are sinding, consumers receiving, etc etc

Also our system throughput dropped from 80 per second to about 45 per second.

In comparison to what?

For every request we are sending about 8 different message to different queues, and the consumer is running in the same system to process these message.

request? how often does this request occurr, how many consumers, etc etc.

I attached a screenshot by jprofiler in sampling mode, and I'm seeking any suggestion to reduce the CPU usage as our system is CPU bounded. We are currently using AIO, and switching to NIO doesn't help on CPU usage.

well it shows that a lot of aio is happening but maybe you are persisting a lot of messages.

Also, what version are you running, if its an old version have you tried a newer version to see if theres any change, also how are you running horneyq, standalone. in jboss, in spring,embedded etc etc

[changgeng]

Thanks for the quick reply.

Our system recently start using hornetq as messaging middleware. The profiling result shows more than 30% cpu time are consumed by hornetq.

How much CPU is used is dependant on lots of things, its hard to comment on this, how fast your producers are sinding, consumers receiving, etc etc

The producers are sending 350 message every second, and the consumers are processing at nearly same speed: we use hornetq for asynchronous, grouping message, and ensure no message is lost or is processed twice. There are totally 4 queue, and each is with one consumer.

In comparison to what?

This is comparing with if we disable the producer and consumer.  The consumer doesn't perform a heavy task. It just composes a http message and send it out.

We are running 2.1.2Final, inside jboss application server with spring.  The profiling result doesn't show spring is a big problem, although I'm not quite sure now whether we can save some time in creating and closing session. As you said, most of the time is spent on AIO, does this mean we have to pay the price for such amount of messages?

[ataylor]

I assumed you were comparing to another messaging provider but actually i guess this is just the processor power needed. AIO will be doing quite a lot of work, especially if your using tx's and persistant messages, remember all messages, message references, acks and tx's will be persisted.

Also, this obviously depends on how fast your processor is too.

[clebert.suconic]

Your profiler actually just shows the AIO thread returning events. Everything will happen through Asynchronous IO when you send persistent, so I would expect that.

your second thread is the thread that will receive your messages....     I see it quite fit in your profiler.

HornetQ doesn't have any syncing threads at the server's side... This workflow will happen:

- You send a persistent message

- At the server we will parse the receiver (bufferReceived)

- We send a request to the Journal

- We install a callback to deliver the message as soon as the message is persistent, and the thread is free to receive another message

- As soon as the disk is done, the callback (pollEvents) will deliver the message

Also: the poller is a native method. I have written profilers before, and I'm not sure JVMTI would inform you the CPU usage correctly. There are two variables a profiler will use.. ellapsedTime and cpuTime. I'm not sure if (don't think so) cpuTime will be properly calculated on a native method.

[changgeng]

I managed to profile the system again with the same test load using nio mode. The total cpu cost on hornetq package seems to be about the same.

Also is there any document for explanation of TimedBuffer?

[clebert.suconic]

I don't see anything bad at the report you sent. You probably don't understand system optimization very well.

What would you expect us to do? How could we treat 100 messages per second (buffer receive), persist them (NIO or AIO) without consuming any CPU?

The chart is showing the percentage of the CPU used. It doesn't mean it's using a lot of CPU.

Besides, we need some CPU to process high loads.. I don't see any other way around.

(100 messages / second is a moderate Load BTW). I see space to increase loading there.

[clebert.suconic]

Moderate to low actually

[changgeng]

Clebert Suconic wrote:

I don't see anything bad at the report you sent. You probably don't understand system optimization very well.

What would you expect us to do? How could we treat 100 messages per second (buffer receive), persist them (NIO or AIO) without consuming any CPU?

I just need to confirm whether I have misconfigured anything, and try to understand more about the internal of hornetq. This is why I have the question about TimedBuffer.

Hornetq is deployed in jboss AS together with our other applications, and our system is cpu bound. If we have to pay that price, we could consider add more CPU resource or have separated server for it. 'A lot' doesn't mean it's more than other products and we didn't compare with others.

And we actually have about 300~400 messages to be produced and consumed every second rath than 100.

There's another post in this forum about the cpu consumed on Thread.yield  during bufferReceived. What do you think the proposed improvement?

http://community.jboss.org/thread/162971

[clebert.suconic]

There are two methods where everthing will go through. Thread.run() which is the Thread pool, and the TimedBuffer / AIO

The fact that you don't see anything else on your report means HornetQ is actually quite optimized and there's nothing else to be tuned. I would say this is the price of your 400 messages / second.

You should probably add more hardware.

On your last post you said you had 30% of CPU. I would say this is quite low considered high loads.