HornetQ performance in Solaris

[pnranjith]

Hi

When I tested hornetq performance in Solaris, it is giving a throughput of 50 messages per second.When I tested it with Ubuntu, it was giving more than 500 messages per second.

What could be the issue. I have used same settings in both. Actually physically I copied the hornetq folder to the solaris machine.

Even the performance example was giving only 110 messages/ sec which gave 2000 messages in linux machine.

Please find the solaris system spec where queue is running given bellow:

HP Pro 3090 Desktop

Sun Solaris 10 SP 7

Intel Core2Duo E7500(2.93 GHz)

Intel G43 Chipset

4 GB DDR3 RAM

320 GB Smart IV HDD

The Linux machine conf is bellow:

HP Pro Desktop

Fedora core 9

Intel Core2Duo E6550(2.33 GHz)

Intel Chipset

3 GB DDR3 RAM

320 GB Smart IV HDD

Hornetq version : 2.1.0

Note : Solaris is Virtualised with 4 zones running.

Please find the hotrnetq configuration files attached here.

[timfox]

Your question is almost impossible to answer since you've haven't described the use case at all.

Are you talking about sending or consuming messages? Persistent or non persistent? How many producers? How many consumers? Message size? Transacted or not transacted? All these things hugely affect performance.

Anyway... I can make a guess to read your mind. Probably you're talking about a single producer sending persistent messages one after another?

If so, (and this has been discussed many times before), you are limited by two factors:

1) The latency (round trip time) of your network. Sending a persistent message, by default, requires a messages to be sent one way and a response sent back to the user when it is persisted. When the response is received the call to send() unblocks. This is a hardware limit of your network.

2) The latency of your hard drive. (I.e. sync speed or completion latency). Most disks can do about 100-200 syncs per second. Again that is a hardware limit. If you invest on expensive disks with NV cache you can go faster, you can also go faster with expensive solid state memory.

Many desktop operating systems by default have disk write cache enabled, this means syncs don't really sync to disk immediately! Probably you have disk write cache enabled on the Ubuntu box, or you just have better hardware.

Anyway, all this is explained in the user manual: http://hornetq.sourceforge.net/docs/hornetq-2.1.1.Final/user-manual/en/html/persistence.html

Note that you even though the performance with one producer may be only 200 msgs/sec (hardware limit), you can scale up with multiple producers, since they can share syncs. In our messaging lab we can get around 9000 msgs/sec with full syncing, this is about twice as fast as the next best performing system on the market, Tibco EMS.

Also bear in mind, that on Solaris you won't have Linux AIO. Best performance will be on Linux.

[pnranjith]

Thanks Tim.

My worries are not related to the 200 messages/sec performance in ubuntu. That is OK for us considering the hardware limitations in QA.

But the issue is actually the drastic performance degradation when we run the same hornetq configuration in a Solaris machine(physically copying the entire foder to Solaris). Where as Linux gave a throughput of 2000 messages/ second in betchmarking(runnig your jms performance script), the hornetq on solaris was giving only 110messages per second(persistant, message size: 1024Bytes).

As we changed our hardware in producton to high performace Solaris with 16 cores etc.. we have to make it run in Solaris soon with the same/nearly similar throughput.

[timfox]

Ranjith PN wrote:

Thanks Tim.

My worries are not related to the 200 messages/sec performance in ubuntu. That is OK for us considering the hardware limitations in QA.

But the issue is actually the drastic performance degradation when we run the same hornetq configuration in a Solaris machine(physically copying the entire foder to Solaris). Where as Linux gave a throughput of 2000 messages/ second in betchmarking(runnig your jms performance script), the hornetq on solaris was giving only 110messages per second(persistant, message size: 1024Bytes).

As we changed our hardware in producton to high performace Solaris with 16 cores etc.. we have to make it run in Solaris soon with the same/nearly similar throughput.

If you're getting 2000 msgs/sec with a single producer you are almost certainly running with disk write cache on. Please read my previous email.

110 msgs/sec sounds about right for real hardware.

[pnranjith]

Tim,

Yes. Disk write cache is on and libaio is enabled.

But why dont we get the same with Solaris ?

Is there any settings tweek specifically for Solaris to improve performance?

[timfox]

Ranjith PN wrote:

Tim,

Yes. Disk write cache is on and libaio is enabled.

But why dont we get the same with Solaris ?

Is there any settings tweek specifically for Solaris to improve performance?    

Please read the section in the docs about this.

You are getting 2000 msgs/sec because disk write cache is *on*. That means data is NOT being synced reliable to disk synchronously, so in event of machine failure *you might lose data*.

This is a *performance optimisation* that most desktop OSes use in order to boost performance at the expense of data reliability. Try turning off disk write cache in Linux and you will see a big change in performance, this is because the system is now syncing data to disk *properly* which it needs to do in order to provide a once and only once reliability guarantee.

Performance is worse on Solaris probably because disk write cache is disabled. In other words Solaris is not cheating! This will be the disks *hardware limit*. There is nothing you can do to boost that in software.

To boost performance, you can either:

1) Getting a RAID array and stripe - this allows parallel writes and can boost performance.

2) Getting an expensive solid state disk

3) Get an expensive SAN

4) Enabled disk write cache on Solaris (consult your manual on how to do this) - this will mean *you might lose data if your machine crashes*.

[pnranjith]

Thanks Tim,

I would like go with a raid array if it is feasible. Or else I have to enable write cache as the other two methods are out of scope now.

Will let you know the increase in throuput.

Thanks a lot

[timfox]

As I mentioned before, bear in mind, 110 msgs/sec is the perf for *one producer*. In the real world you rarely have just one producer, the system will scale up to much more than 110 msgs/sec with multiple producers, since syncs can be batched.

E.g. we can do around 9000 msgs/sec with no disk write cache on standard disks and 40 producers and consumers. Would go even higher without consumers. Don't get fooled by poor benchmarks with one producer.