HornetQ - low performance in real world scenarion?| JBoss.org Content Archive (Read Only)

15. Re: HornetQ - low performance in real world scenarion?

mreasy Oct 16, 2009 9:18 AM (in response to mreasy)

/hornetq/examples/core/perf/perf.properties

num-messages=10000
num-warmup-messages=100
message-size=1024
durable=true
transacted=false
batch-size=1000
drain-queue=false
throttle-rate=-1
address=perfAddress
queue-name=perfQueue
host=localhost
port=5445
tcp-buffer=1048576
tcp-no-delay=false
send-window=1048576
pre-ack=true
block-ack=false
block-persistent=true

/hornetq/examples/core/perf/server0/hornetq-configuration.xml

<configuration xmlns="urn:hornetq"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">


 <!-- Acceptors -->
 <acceptors>
 <acceptor name="netty-acceptor">
 <factory-class>org.hornetq.integration.transports.netty.NettyAcceptorFactory</factory-class>
 <param key="hornetq.remoting.netty.tcpnodelay" value="false" type="Boolean"/>
 <param key="hornetq.remoting.netty.tcpsendbuffersize" value="1048576" type="Integer"/>
 <param key="hornetq.remoting.netty.tcpreceivebuffersize" value="1048576" type="Integer"/>
 </acceptor>
 </acceptors>

 <security-enabled>false</security-enabled>

 <persistence-enabled>true</persistence-enabled>

 <journal-sync-non-transactional>true</journal-sync-non-transactional>
 <journal-type>ASYNCIO</journal-type>
 <journal-min-files>20</journal-min-files>
 <journal-aio-buffer-timeout>20000</journal-aio-buffer-timeout>
 <log-journal-write-rate>true</log-journal-write-rate>
 <!-- <perf-blast-pages>5000</perf-blast-pages> -->


 <queues>
 <queue name="perfQueue">
 <address>perfAddress</address>
 </queue>
 </queues>

</configuration>

/hornetq/examples/core/perf/build.xml: replaced "org.hornetq.jms.example" with "org.hornetq.core.example" to make the test startable.
Fixed build.sh (see https://jira.jboss.org/jira/browse/HORNETQ-181)
/export/home/neubauer/test/hornetq/hornetq/examples/core/perf
-bash-3.00$ ./build.sh runServer
*wait*
-bash-3.00$ ./build.sh runSender

16. Re: HornetQ - low performance in real world scenarion?

timfox Oct 16, 2009 9:41 AM (in response to mreasy)

I don't get it, I just used your *exact* configuration that you posted, running on my laptop, Fedora 11, and I get about 10 times your figures.

Here is the output from my console, in case you don't believe me:

[tim@localhost perf]$ ./build.sh runSender
ANT_HOME is ../../../tools/ant
Found javac
Using the following ant version from ../../../tools/ant:
Apache Ant version 1.7.1 compiled on June 27 2008
Buildfile: build.xml

init:

compile:
 [echo] src.example.dir=/home/tim/workspace/trunk/examples/core/perf/src

runSender:
 [java] 16-Oct-2009 14:33:27 org.hornetq.core.example.PerfBase getPerfFileName
 [java] INFO: Using file name perf.properties
 [java] 16-Oct-2009 14:33:27 org.hornetq.core.example.PerfBase getParams
 [java] INFO: num-messages: 10000
 [java] 16-Oct-2009 14:33:27 org.hornetq.core.example.PerfBase getParams
 [java] INFO: num-warmup-messages: 100
 [java] 16-Oct-2009 14:33:27 org.hornetq.core.example.PerfBase getParams
 [java] INFO: message-size: 1024
 [java] 16-Oct-2009 14:33:27 org.hornetq.core.example.PerfBase getParams
 [java] INFO: durable: true
 [java] 16-Oct-2009 14:33:27 org.hornetq.core.example.PerfBase getParams
 [java] INFO: transacted: false
 [java] 16-Oct-2009 14:33:27 org.hornetq.core.example.PerfBase getParams
 [java] INFO: batch-size: 1000
 [java] 16-Oct-2009 14:33:27 org.hornetq.core.example.PerfBase getParams
 [java] INFO: drain-queue: true
 [java] 16-Oct-2009 14:33:27 org.hornetq.core.example.PerfBase getParams
 [java] INFO: address: perfAddress
 [java] 16-Oct-2009 14:33:27 org.hornetq.core.example.PerfBase getParams
 [java] INFO: queue name: perfQueue
 [java] 16-Oct-2009 14:33:27 org.hornetq.core.example.PerfBase getParams
 [java] INFO: throttle-rate: -1
 [java] 16-Oct-2009 14:33:27 org.hornetq.core.example.PerfBase getParams
 [java] INFO: host:localhost
 [java] 16-Oct-2009 14:33:27 org.hornetq.core.example.PerfBase getParams
 [java] INFO: port: 5445
 [java] 16-Oct-2009 14:33:27 org.hornetq.core.example.PerfBase getParams
 [java] INFO: tcp buffer: 1048576
 [java] 16-Oct-2009 14:33:27 org.hornetq.core.example.PerfBase getParams
 [java] INFO: tcp no delay: false
 [java] 16-Oct-2009 14:33:27 org.hornetq.core.example.PerfBase getParams
 [java] INFO: pre-ack: true
 [java] 16-Oct-2009 14:33:27 org.hornetq.core.example.PerfBase getParams
 [java] INFO: send-window: 1048576
 [java] 16-Oct-2009 14:33:27 org.hornetq.core.example.PerfBase getParams
 [java] INFO: block-ack:false
 [java] 16-Oct-2009 14:33:27 org.hornetq.core.example.PerfBase getParams
 [java] INFO: block-persistent:true
 [java] 16-Oct-2009 14:33:27 org.hornetq.core.example.PerfBase runSender
 [java] INFO: params = message to send = 10000, Durable = true, session transacted = false, drain queue = true, queue name = perfQueue, Throttle rate = -1, blockOnPersistent = true. blockOnACK = false
 [java] 16-Oct-2009 14:33:28 org.hornetq.core.example.PerfBase drainQueue
 [java] INFO: Draining queue
 [java] 16-Oct-2009 14:33:43 org.hornetq.core.example.PerfBase drainQueue
 [java] INFO: Drained 10100 messages
 [java] 16-Oct-2009 14:33:43 org.hornetq.core.example.PerfBase runSender
 [java] INFO: warming up by sending 100 messages
 [java] 16-Oct-2009 14:33:43 org.hornetq.core.example.PerfBase runSender
 [java] INFO: warmed up
 [java] 16-Oct-2009 14:33:45 org.hornetq.core.example.PerfBase sendMessages
 [java] INFO: sent 2000 messages in 2.27s
 [java] 16-Oct-2009 14:33:47 org.hornetq.core.example.PerfBase sendMessages
 [java] INFO: sent 4000 messages in 4.44s
 [java] 16-Oct-2009 14:33:50 org.hornetq.core.example.PerfBase sendMessages
 [java] INFO: sent 6000 messages in 6.73s
 [java] 16-Oct-2009 14:33:52 org.hornetq.core.example.PerfBase sendMessages
 [java] INFO: sent 8000 messages in 9.19s
 [java] 16-Oct-2009 14:33:54 org.hornetq.core.example.PerfBase sendMessages
 [java] INFO: sent 10000 messages in 11.55s
 [java] 16-Oct-2009 14:33:54 org.hornetq.core.example.PerfBase displayAverage
 [java] INFO: average: 865.58 msg/s (10000 messages in 11.55s)

BUILD SUCCESSFUL
Total time: 28 seconds

The above is using AIO. When I use NIO I get the following:

[tim@localhost perf]$ ./build.sh runSender
ANT_HOME is ../../../tools/ant
Found javac
Using the following ant version from ../../../tools/ant:
Apache Ant version 1.7.1 compiled on June 27 2008
Buildfile: build.xml

init:

compile:
 [echo] src.example.dir=/home/tim/workspace/trunk/examples/core/perf/src

runSender:
 [java] 16-Oct-2009 14:35:45 org.hornetq.core.example.PerfBase getPerfFileName
 [java] INFO: Using file name perf.properties
 [java] 16-Oct-2009 14:35:45 org.hornetq.core.example.PerfBase getParams
 [java] INFO: num-messages: 10000
 [java] 16-Oct-2009 14:35:45 org.hornetq.core.example.PerfBase getParams
 [java] INFO: num-warmup-messages: 100
 [java] 16-Oct-2009 14:35:45 org.hornetq.core.example.PerfBase getParams
 [java] INFO: message-size: 1024
 [java] 16-Oct-2009 14:35:45 org.hornetq.core.example.PerfBase getParams
 [java] INFO: durable: true
 [java] 16-Oct-2009 14:35:45 org.hornetq.core.example.PerfBase getParams
 [java] INFO: transacted: false
 [java] 16-Oct-2009 14:35:45 org.hornetq.core.example.PerfBase getParams
 [java] INFO: batch-size: 1000
 [java] 16-Oct-2009 14:35:45 org.hornetq.core.example.PerfBase getParams
 [java] INFO: drain-queue: true
 [java] 16-Oct-2009 14:35:45 org.hornetq.core.example.PerfBase getParams
 [java] INFO: address: perfAddress
 [java] 16-Oct-2009 14:35:45 org.hornetq.core.example.PerfBase getParams
 [java] INFO: queue name: perfQueue
 [java] 16-Oct-2009 14:35:45 org.hornetq.core.example.PerfBase getParams
 [java] INFO: throttle-rate: -1
 [java] 16-Oct-2009 14:35:45 org.hornetq.core.example.PerfBase getParams
 [java] INFO: host:localhost
 [java] 16-Oct-2009 14:35:45 org.hornetq.core.example.PerfBase getParams
 [java] INFO: port: 5445
 [java] 16-Oct-2009 14:35:45 org.hornetq.core.example.PerfBase getParams
 [java] INFO: tcp buffer: 1048576
 [java] 16-Oct-2009 14:35:45 org.hornetq.core.example.PerfBase getParams
 [java] INFO: tcp no delay: false
 [java] 16-Oct-2009 14:35:45 org.hornetq.core.example.PerfBase getParams
 [java] INFO: pre-ack: true
 [java] 16-Oct-2009 14:35:45 org.hornetq.core.example.PerfBase getParams
 [java] INFO: send-window: 1048576
 [java] 16-Oct-2009 14:35:45 org.hornetq.core.example.PerfBase getParams
 [java] INFO: block-ack:false
 [java] 16-Oct-2009 14:35:45 org.hornetq.core.example.PerfBase getParams
 [java] INFO: block-persistent:true
 [java] 16-Oct-2009 14:35:45 org.hornetq.core.example.PerfBase runSender
 [java] INFO: params = message to send = 10000, Durable = true, session transacted = false, drain queue = true, queue name = perfQueue, Throttle rate = -1, blockOnPersistent = true. blockOnACK = false
 [java] 16-Oct-2009 14:35:46 org.hornetq.core.example.PerfBase drainQueue
 [java] INFO: Draining queue
 [java] 16-Oct-2009 14:35:57 org.hornetq.core.example.PerfBase drainQueue
 [java] INFO: Drained 10100 messages
 [java] 16-Oct-2009 14:35:57 org.hornetq.core.example.PerfBase runSender
 [java] INFO: warming up by sending 100 messages
 [java] 16-Oct-2009 14:35:58 org.hornetq.core.example.PerfBase runSender
 [java] INFO: warmed up
 [java] 16-Oct-2009 14:36:01 org.hornetq.core.example.PerfBase sendMessages
 [java] INFO: sent 2000 messages in 2.87s
 [java] 16-Oct-2009 14:36:03 org.hornetq.core.example.PerfBase sendMessages
 [java] INFO: sent 4000 messages in 5.69s
 [java] 16-Oct-2009 14:36:06 org.hornetq.core.example.PerfBase sendMessages
 [java] INFO: sent 6000 messages in 8.53s
 [java] 16-Oct-2009 14:36:09 org.hornetq.core.example.PerfBase sendMessages
 [java] INFO: sent 8000 messages in 10.92s
 [java] 16-Oct-2009 14:36:11 org.hornetq.core.example.PerfBase sendMessages
 [java] INFO: sent 10000 messages in 13.36s
 [java] 16-Oct-2009 14:36:11 org.hornetq.core.example.PerfBase displayAverage
 [java] INFO: average: 748.62 msg/s (10000 messages in 13.36s)

BUILD SUCCESSFUL
Total time: 26 seconds

Do you have a linux PC anywhere? - I would like to see what results you get on Linux.

17. Re: HornetQ - low performance in real world scenarion?

mreasy Oct 16, 2009 9:47 AM (in response to mreasy)

Yes, will run an additional test on Linux.

All my core-perf-tests were using NIO, just fyi.

Just curious, what VM do you use?
Mine: Sun JDK 1.6 Update 16, 64bit

18. Re: HornetQ - low performance in real world scenarion?

timfox Oct 16, 2009 9:51 AM (in response to mreasy)

One thing I just spotted - you said you edited hornetq-configuration.xml from:

/hornetq/examples/core/perf/server0/hornetq-configuration.xml

but that's the wrong one

when you use ./build.sh runServer it uses config from:

/src/config/trunk/standalone

19. Re: HornetQ - low performance in real world scenarion?

mreasy Oct 16, 2009 10:00 AM (in response to mreasy)

Results from Linux CentOS 5.3 64bit machine on Quad-Core, 8GB RAM, 15000 rpm-hard-disk:
AIO: 790 msgs/sec (8MB/sec)
NIO: 207 msgs/sec

still way slower than your results on the laptop.

"timfox" wrote:
(...)
when you use ./build.sh runServer it uses config from:

/src/config/trunk/standalone

There is no such file in /src/config/trunk/standalone.

Files in
/export/home/neubauer/test/hornetq/hornetq/src/config/stand-alone/non-clustered
/export/home/neubauer/test/hornetq/hornetq/src/config/trunk/non-clustered
I did not touch, which means default, which means sync=true, so should make no difference

20. Re: HornetQ - low performance in real world scenarion?

timfox Oct 16, 2009 10:10 AM (in response to mreasy)

This one:

"MrEasy" wrote:

/export/home/neubauer/test/hornetq/hornetq/src/config/stand-alone/non-clustered

21. Re: HornetQ - low performance in real world scenarion?

timfox Oct 16, 2009 10:12 AM (in response to mreasy)

You should also run with a *much* bigger warmup and more messages.

100 messages warmup is way too low

22. Re: HornetQ - low performance in real world scenarion?

timfox Oct 16, 2009 10:15 AM (in response to mreasy)

OK, so far we have tried 1 crappy laptop with linux, and 1 macbook pro running OS/X and got pretty much the same results on both.

I will look for a Windows machine next.

23. Re: HornetQ - low performance in real world scenarion?

mreasy Oct 16, 2009 10:17 AM (in response to mreasy)

"timfox" wrote:
You should also run with a *much* bigger warmup and more messages.

100 messages warmup is way too low

What should the warm-up do - trigger the hotspot-compiler? c'mon...
Also this does not explain the huge differences in direct comparison.

What filesystem has your Linux? jmesnil's Mac is HFS I suppose.

24. Re: HornetQ - low performance in real world scenarion?

timfox Oct 16, 2009 10:22 AM (in response to mreasy)

"MrEasy" wrote:
[
What should the warm-up do - trigger the hotspot-compiler? c'mon...

Hotspot make a *huge* difference and can take 30 seconds or more to kick in.

No real benchmark will run from cold (e.g. specJMS), it can skew the results enormously.

25. Re: HornetQ - low performance in real world scenarion?

mreasy Oct 16, 2009 10:37 AM (in response to mreasy)

Test with warm code on that Solaris box:
average: 82.26 msg/s (10000 messages in 121.57s)

26. Re: HornetQ - low performance in real world scenarion?

timfox Oct 16, 2009 5:49 PM (in response to mreasy)

Ok, so I think I have got the bottom of this.

As I suspected on IRC, it seems that many cheap disk controllers (like in crappy Dell laptops ;) ) have write caching enabled in the disk controller by default.

This means when you do a fsync() from the operating system (which is what happens when we call flush()) it doesn't mean the data has definitely reached the disk.

This appears to be a cheap trick played by computer manufacturers to make their cheap hardware seem faster.

On better/enterprise disk hardware, typically write caching is turned *off* by default, so we know your data really reaches the disk. Or, on expensive disks/SANs they often have non volatile disk RAM cache so, although they still just write into the cache, that will survive a crash.

This explains why I can see such good sync rates on my cheap laptop, compared to your more expensive SCSI drives.

So.. the good news is HornetQ is doing the best it can - it's not in control of how fast the operating system / disk actually records the data.

The bad news is you will never get more than maybe a couple of hundred syncs per sec on a reasonable disk.

But like I say, very expensive disks can have NV RAM and give you the reliability guarantees but let you do a couple of thousands syncs per sec - but it will cost you.

On linux, to see whether disk write cache is enabled (note this is a device setting, not an OS setting) do:

hdparm -W <device_name>

where <device_name_ e.g. is /dev/sda

to turn write cache off do:

hdparm -W0 <device_name>

on windows you can apparently just write click on your disk, click properties and enable/disable there

Note this issue is not specific to HornetQ. It applies to any application that needs to sync disk reliable to disk.

Here's an interesting article on this:

http://www.jasonbrome.com/blog/archives/2004/04/03/writecache_enabled.html

27. Re: HornetQ - low performance in real world scenarion?

timfox Oct 16, 2009 5:58 PM (in response to mreasy)

When I re-run the perfSender test with write caching turned off, I get a send rate of 112 msgs/sec which is more in keeping with your results.

28. Re: HornetQ - low performance in real world scenarion?

timfox Oct 16, 2009 5:58 PM (in response to mreasy)

WARNING! hdparm can do nasty things to your disk if you make a mistake - please use with caution!

29. Re: HornetQ - low performance in real world scenarion?

timfox Oct 16, 2009 6:26 PM (in response to mreasy)

Here's another way to look at it.

If you disk rotates at 7200 RPM- that means 120 revolutions per sec.

That means it can never really write to the same place on the disk more than 120 times per sec, since it has to wait for the disk rotate around and go past the head again.

This figures with the sync rates of around 100 syncs / sec.

If you want faster than that you need good disks with battery backed cache or solid state disks.