Hornetq poor throughput

[pnranjith]

Hi,

We have decided to migrate our application to hornetq from activemq. We are getting a throughput of 300 messages per second with activemq. But after the migration we get only 20 messages per second. The application is configured in such a way that the queue can be switched easily. So when we again switch it back to activemq, it is giving 300 messages per second. But hornetq doesn't give anyway nearer. Any clue?

[gaohoward]

Can I ask how did you measure the throughput? Was it at the sending end or receiving end?

[pnranjith]

Both at sending and recieving end it is giving almost same throughput. We measure it using jconsole and also we have a database in which the messege status are written. So we use it to query the details.

[gaohoward]

Did you set any flow control parameters such as consumer-window-size or producer-window-size?

[pnranjith]

No. I did not set any window size. not at producer side nor at consumer.

[ataylor]

This is probably a config issue somewhere but without any context its hard for us to advise you. are you using default config, persistent messages, what size messages etc etc. post some config and some code and maybe we can help you.

[timfox]

Wow, that question is almost impossible to answer since you provide almost zero information.

No config, no test program etc.

[timfox]

What you need to do is create a simple test program we can run against both us and ActiveMQ, so we can see this difference.

[timfox]

Please read this as well: http://community.jboss.org/wiki/Howtoreportabugissue

When posting it's vital you provide as much information as possible, and this includes a test case.

Otherwise it's a bit like walking into a doctor, saying your friend is sick, giving no information and expecting the doctor to diagnose the problem.

[timfox]

"also we have a database in which the messege status"

Well, the db is going to be the weakest link for a start

[pnranjith]

So, getting deeper with the debugging.

Here I measured the time taken and its around 30ms sometime:

This is the writer part:

objMessage is an instance of javax.jms.ObjectMessage

msg is an instance of message

        long startTime = 0L ;
        ObjectMessage objMessage;
        try {
                 objMessage = session.createObjectMessage();
                 objMessage.setObject(msg);
                 startTime = System.currentTimeMillis() ;
                 producer.send(objMessage);
                 AppLogger.logger.info("Time in Writer.write "+(System.currentTimeMillis() - startTime));

        }

And I am creating the producer with the following code:

        connection = connectionFactory.createConnection();
        connection.start();
        session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);      

        Queue queue = createQueue(subject) ;      
        producer = session.createProducer(queue);
        producer.setDeliveryMode(DeliveryMode.PERSISTENT);

[timfox]

This is not a test program, it's just a code snippet.

We would need to see a working test program, that we can actually run and actually produces some result we can look at.

Also, you didn't say what versions of HornetQ or ActiveMQ you are using, nor did you provide your config.

Also, sending just one message means absolutely nothing!

You need to

a) send many thousands of messages, and average over a long time in order to minimise indeterminacy due to gc and thread scheduling

b) allow enough time for JIT to kick in

This has been discussed on several threads before. I'm getting a bit tired having to continually repeat myself on this (please search).

Also bear in mind that microbenchmarks using a single sender/consumers are rarely very interesting as they say nothing about how the system scales with many senders/consumers.

Also, did you make sure you have turned ActiveMQ syncing on?

http://activemq.apache.org/amq-message-store.html

Actually you didn't say either what ActiveMQ store you're using, so I'll assume it's AMQ message store.

Remember, that ActiveMQ "cheats" and does not sync to disk in it's out of the box config.

Set syncOnWrite to true

[pnranjith]

But with a test program I wrote, it is working fine. That is why I didn't send the test program. The issue comes when I connect my already live application to the hornetq. One more thing is when I run the queue seperately in another system with 2GB RAM, it is giving me 200 messages per second. Same program connected with the hornetq running on the same machine with 1GB RAM is giving only 5-10 message per second. Both systems with activemq is working with the same throughput which is much better than with the hornet.

version I use is:  hornetq-2.0.0.GA

amq version is : fuse amq 5.0.0.29

I am sending with 5 producer and 5 consumer with 250 million messages .

The results are taken after 6 minutes.

I have 5 queues which are created dynamically, viz. response queue, request queue, alertq, db queue and reprocess queue whose name implies its functionalities.

One more doubt is, Can I creae a cluster  to improve the performance

Thanks a lot

[timfox]

So you need to create a test program that does exhibit a problem. This will help you track down the issue, which is almost certainly a problem in your config, or in ActiveMQ config.

Like I said before, activemq doesn't sync to disk by default.

[timfox]

A few points:

You are sending the messages as persistent, this means, by default each send will do a network round trip and block until the message has been stored in persistent storage. Also when the message is acked, the ack also needs to be persisted.

By default HornetQ will force a sync to the disk to ensure the data has really got there. (Other systems e.g. ActiveMQ ***DO NOT** sync by defaut - on other words, they cheat and you could lose data with them).

An average disk on a laptop/desktop can do maybe 100 to 200 syncs per sec. THIS IS A HARDWARE LIMIT OF THE DISK, it is not a limit of HornetQ! This would explain a 100 to 200 message rate.

If you really want persistent messaging you can either:

a) Buy an expensive SCSI disk with battery backed cache - you might be able to get a couple of thousand syncs per sec

b) Buy solid state disk (e.g. FusionIO)

c) Send many messages in a single transaction

d) Use async send acknowledgements (see user manual) and don't block on sends.

e) Don't use persistent messages

f) Probably some other stuff I can't think of right now

Like I say, pretty much all this is discussed in the user manual. Please take some time to read it.