1 2 Previous Next 15 Replies Latest reply on Apr 30, 2010 1:43 PM by Roklee LLC

    Hornetq poor throughput

    Ranjith PN Novice

      Hi,

      We have decided to migrate our application to hornetq from activemq. We are getting a throughput of 300 messages per second with activemq. But after the migration we get only 20 messages per second. The application is configured in such a way that the queue can be switched easily. So when we again switch it back to activemq, it is giving 300 messages per second. But hornetq doesn't give anyway nearer. Any clue?

        • 1. Re: Hornetq poor throughput
          Yong Hao Gao Master

          Can I ask how did you measure the throughput? Was it at the sending end or receiving end?

          • 2. Re: Hornetq poor throughput
            Ranjith PN Novice

            Both at sending and recieving end it is giving almost same throughput. We measure it using jconsole and also we have a database in which the messege status are written. So we use it to query the details.

            • 3. Re: Hornetq poor throughput
              Yong Hao Gao Master

              Did you set any flow control parameters such as consumer-window-size or producer-window-size?

              • 4. Re: Hornetq poor throughput
                Ranjith PN Novice

                No. I did not set any window size. not at producer side nor at consumer.

                • 5. Re: Hornetq poor throughput
                  Andy Taylor Master

                  This is probably a config issue somewhere but without any context its hard for us to advise you. are you using default config, persistent messages, what size messages etc etc. post some config and some code and maybe we can help you.

                  • 6. Re: Hornetq poor throughput
                    Tim Fox Master

                    Wow, that question is almost impossible to answer since you provide almost zero information.

                     

                    No config, no test program etc.

                    • 7. Re: Hornetq poor throughput
                      Tim Fox Master

                      What you need to do is create a simple test program we can run against both us and ActiveMQ, so we can see this difference.

                      • 8. Re: Hornetq poor throughput
                        Tim Fox Master

                        Please read this as well: http://community.jboss.org/wiki/Howtoreportabugissue

                         

                        When posting it's vital you provide as much information as possible, and this includes a test case.

                         

                        Otherwise it's a bit like walking into a doctor, saying your friend is sick, giving no information and expecting the doctor to diagnose the problem.

                        • 9. Re: Hornetq poor throughput
                          Tim Fox Master

                          "also we have a database in which the messege status"

                           

                          Well, the db is going to be the weakest link for a start

                          • 10. Re: Hornetq poor throughput
                            Ranjith PN Novice

                            So, getting deeper with the debugging.

                             

                            Here I measured the time taken and its around 30ms sometime:

                             

                            This is the writer part:

                            objMessage is an instance of javax.jms.ObjectMessage

                            msg is an instance of message

                             

                                    long startTime = 0L ;
                                    ObjectMessage objMessage;
                                    try {
                                             objMessage = session.createObjectMessage();
                                             objMessage.setObject(msg);
                                             startTime = System.currentTimeMillis() ;
                                             producer.send(objMessage);
                                             AppLogger.logger.info("Time in Writer.write "+(System.currentTimeMillis() - startTime));

                                    }

                             

                             

                            And I am creating the producer with the following code:

                             

                                    connection = connectionFactory.createConnection();
                                    connection.start();
                                    session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);      

                                    Queue queue = createQueue(subject) ;      
                                    producer = session.createProducer(queue);
                                    producer.setDeliveryMode(DeliveryMode.PERSISTENT);

                            • 11. Re: Hornetq poor throughput
                              Tim Fox Master

                              This is not a test program, it's just a code snippet.

                               

                              We would need to see a working test program, that we can actually run and actually produces some result we can look at.

                               

                              Also, you didn't say what versions of HornetQ or ActiveMQ you are using, nor did you provide your config.

                               

                              Also, sending just one message means absolutely nothing!

                               

                              You need to

                               

                              a) send many thousands of messages, and average over a long time in order to minimise indeterminacy due to gc and thread scheduling

                               

                              b) allow enough time for JIT to kick in

                               

                              This has been discussed on several threads before. I'm getting a bit tired having to continually repeat myself on this (please search).

                               

                              Also bear in mind that microbenchmarks using a single sender/consumers are rarely very interesting as they say nothing about how the system scales with many senders/consumers.

                               

                              Also, did you make sure you have turned ActiveMQ syncing on?

                               

                              http://activemq.apache.org/amq-message-store.html

                               

                              Actually you didn't say either what ActiveMQ store you're using, so I'll assume it's AMQ message store.

                               

                              Remember, that ActiveMQ "cheats" and does not sync to disk in it's out of the box config.

                               

                              Set syncOnWrite to true

                              • 12. Re: Hornetq poor throughput
                                Ranjith PN Novice

                                But with a test program I wrote, it is working fine. That is why I didn't send the test program. The issue comes when I connect my already live application to the hornetq. One more thing is when I run the queue seperately in another system with 2GB RAM, it is giving me 200 messages per second. Same program connected with the hornetq running on the same machine with 1GB RAM is giving only 5-10 message per second. Both systems with activemq is working with the same throughput which is much better than with the hornet.

                                 

                                 

                                 

                                version I use is:  hornetq-2.0.0.GA

                                amq version is : fuse amq 5.0.0.29

                                 

                                I am sending with 5 producer and 5 consumer with 250 million messages .

                                The results are taken after 6 minutes.

                                I have 5 queues which are created dynamically, viz. response queue, request queue, alertq, db queue and reprocess queue whose name implies its functionalities.

                                 

                                One more doubt is, Can I creae a cluster  to improve the performance

                                 

                                Thanks a lot

                                • 13. Re: Hornetq poor throughput
                                  Tim Fox Master

                                  So you need to create a test program that does exhibit a problem. This will help you track down the issue, which is almost certainly a problem in your config, or in ActiveMQ config.

                                   

                                  Like I said before, activemq doesn't sync to disk by default.

                                  • 14. Re: Hornetq poor throughput
                                    Tim Fox Master

                                    A few points:

                                     

                                    You are sending the messages as persistent, this means, by default each send will do a network round trip and block until the message has been stored in persistent storage. Also when the message is acked, the ack also needs to be persisted.

                                     

                                    By default HornetQ will force a sync to the disk to ensure the data has really got there. (Other systems e.g. ActiveMQ ***DO NOT** sync by defaut - on other words, they cheat and you could lose data with them).

                                     

                                    An average disk on a laptop/desktop can do maybe 100 to 200 syncs per sec. THIS IS A HARDWARE LIMIT OF THE DISK, it is not a limit of HornetQ! This would explain a 100 to 200 message rate.

                                     

                                    If you really want persistent messaging you can either:

                                     

                                    a) Buy an expensive SCSI disk with battery backed cache - you might be able to get a couple of thousand syncs per sec

                                    b) Buy solid state disk (e.g. FusionIO)

                                    c) Send many messages in a single transaction

                                    d) Use async send acknowledgements (see user manual) and don't block on sends.

                                    e) Don't use persistent messages

                                    f) Probably some other stuff I can't think of right now

                                     

                                    Like I say, pretty much all this is discussed in the user manual. Please take some time to read it.

                                    1 2 Previous Next