I am trying to test performance of HornetQ when a persistent queue has built a large backlog (million+ messages)
Currently I am seeing degraded performance when server starts paging messages.
-- The message senders seem severly throttled and publish rates drop from 50 messages/sec to 1 per second or less
-- Also, message receivers seem to slow down significantly too and de-queuing rate drops significantly
Here is the test setup I am using
-- JBoss 4.3 with HornetQ as messaging provider + testQueue + Simple MDB that receives and simulates random processing times (1-100 ms)
Running on linux, Using AIO,
Min journal files are configured to 400
and Queue memory size is set to 400 MB (have tried 100,400 and 800 MB of total heap 1.5 GB)
Paging file size (100 MB)
-- Message pump test program which spawns multiple threads with each thread sending messages with sleep times of about 5 ms
Sample message is about 4K in size when saved to file
When everything is running fine, that is both message pump and receivers are running fine
Message pump threads are sending at 50 messages per sec (Interestingly I was able to get higher publish rate after I increased the min journal file size to 400. With default configuration I got about 35 TPS which increased to 50 with 400 journal files). I have about 40 threads configured.
Message receive/processing is also about 1800-2000 TPS with about 15 MDB instances and almost do not get any backlog on the queue
Now I wanted to create a backlog and see the performance (as this is our current problem with JBM persistent queues that performance comes to a crawl when we have event burst and queue backlog of 500,000 messages or more)
I paused the queue which stops the delivery of incoming messages to MDB
Publish rate of the message pump threads remains fine close to their peak at 50 TPS until HornetQ starts paging messages.
At max message size of 400 MB this happened at about 53233 messages and at 800 MB happened at 106,466 (exactly double the previous number).
When this number is reached publish rates fall to less than 1 per second and message counters (for the queue MBean) is not updated at all.
This backlog and queue size is not good enough for us as we are likely to encounter much larger message volumes sitting in the queue in real life scenarios.
Post this when I resume delivery of the queue,
-- MDB cannot still run at the same rate as before. In fact runs at 1/100 previous consumption rate
-- Message publishers cannot send at original publish rates
However interesting behavior I have seen I shutdown the message pump and the MDB is immediately able to run at previous rate again and burns through messages in the queue in no time.
From all this, my suspicion is that paging is what is causing the performance degradation.
So far I have played with only these 2 parameters in tuning
-- Max memory size for the queue
-- Size of paging file
What other things I should be looking at tuning to get better performance when messages are to be paged?