Paging and HornetQ performance

[mpr_prabhu]

I am trying to test performance of HornetQ when a persistent queue has built a large backlog (million+ messages)

Currently I am seeing degraded performance  when server starts paging messages.

-- The message senders seem severly throttled and publish rates drop from 50 messages/sec to 1 per second or less

-- Also, message receivers seem to slow down significantly too and de-queuing rate drops significantly

Here is the test setup I am using

-- JBoss 4.3 with HornetQ as messaging provider + testQueue + Simple MDB that receives and simulates random processing times (1-100 ms)

   Running on linux, Using AIO,

   Min journal files are configured to 400

   and Queue memory size is set to 400 MB (have tried 100,400 and 800 MB of total heap 1.5 GB)

   Paging file size (100 MB)

-- Message pump test program which spawns multiple threads with each thread sending messages with sleep times of about 5 ms

   Sample message is about 4K in size when saved to file

When everything is running fine, that is both message pump and receivers are running fine

Message pump threads are sending at 50 messages per sec (Interestingly I was able to get higher publish rate after I increased the min journal file size to 400. With default configuration I got about 35 TPS which increased to 50 with 400 journal files). I have about 40 threads configured.

Message receive/processing is also about 1800-2000 TPS with about 15 MDB instances and almost do not get any backlog on the queue

Now I wanted to create a backlog and see the performance (as this is our current problem with JBM persistent queues that performance comes to a crawl when we have event burst and queue backlog of 500,000 messages or more)

I paused the queue which stops the delivery of incoming messages to MDB

Publish rate of the message pump threads remains fine close to their peak at 50 TPS until HornetQ starts paging messages.

At max message size of 400 MB this happened at about 53233 messages and at 800 MB happened at 106,466 (exactly double the previous number).

When this number  is reached publish rates fall to less than 1 per second and message counters (for the queue MBean) is not updated at all.

This backlog and queue size is not good enough for us as we are likely to encounter much larger message volumes sitting in the queue in real life scenarios.

Post this when I resume delivery of the queue,

-- MDB cannot still run at the same rate as before. In fact runs at 1/100 previous consumption rate

-- Message publishers cannot send at original publish rates

However interesting behavior I have seen I shutdown the message pump and the MDB is immediately able to run at previous rate again and burns through messages in the queue in no time.

From all this, my suspicion is that paging is what is causing the performance degradation.

So far I have played with only these 2 parameters in tuning

-- Max memory size for the queue

-- Size of paging file

What other things I should be looking at tuning to get better performance when messages are to be paged?

Thanks

[timfox]

Firstly, you didn't say which version of HornetQ you're using, so I'll assume you're using the latest version CR2 (see http://community.jboss.org/wiki/Howtoreportabugissue)

Next thing would be to read the sections in the user manual on performance tuning, persistence and performance, which have some tips and recommendations including:

1) Don't use MDBs unless you have to. You mentioned "TPS" so I assume you're using transacted consumption on MDBs. Depending how you've configured them, transactions (especially XA) can introduce a massive overhead due to the extra sync'ing required on transaction boundaries. This is not an overhead due to the messaging provider, but due to the MDB layer of the AS

2) Put the journal and paging directory on different physical volumes. Otherwise the disk head will be skipping between writing to the journal and the paging store. This will make a big difference in performance. If you're using XA transactions in the AS the tx mgr log needs to go on a different physical volume.

Bear in mind, that of course paging will have a performance impact on the system, since it's doing twice as much IO!

I.e. instead of storing the message, then acknowledging the message, it's paging the message, depaging the message, storing the message and acking the message, so expect performance to be roughly halved.

Also you didn't mention what hardware you're using. If you're doing a lot of paging and persistence you'll be limited by the quality and setup of the disks you are using. If this experiment is on a cheap desktop and not on a real server with well configured modern disks then it's likely to be completely unrepresentative.

Another point is, have you tried BLOCKing instead of paging (see user manual). With blocking, producers will be throttled when address max size is reached. This will give you a better overall throughput than paging.