1 2 3 4 Previous Next 47 Replies Latest reply on Jun 8, 2010 9:10 AM by timfox

    many topics, paused queues, memory growing

    hantunca

      I've been working with hornetq for a while - I'm building a system that deploys 3000 - 4000 JMS topics, and connect to it via a couple of consumers.  I noticed that when I did this, some queues were not receiving any messages, and that the JMS server was consuming more and more memory.  On a hunch, I wrote a small program that contacts the server via the JMX interface, gets a QueueControl object for each queue associated with a topic, and calls the resume method - this freed up the messages and memory stopped growing on the server.  I have never invoked the pause method on the QueueControl object, so I'm not sure how the queues get paused in the first place.  In addition, I decided to use the QueueControl object and call the isPaused method - none of the queues reported they were paused.  However, when I called the resume method, the queues started working.  Has anybody run into this problem before?

       

      thanks in advance.

        • 1. Re: many topics, paused queues, memory growing
          clebert.suconic

          You probably have messages building up (not being consumed) and waiting ACK before resume delivery.

           

          When you called resume on the queue, that forced delivery as other queues were probably delivering before that.

           

           

          What version are you currently using? can you tweak some of your configs on the max-size... paging.. etc?

           

           

          It would be nice if you could check how it behaves on trunk.

          • 2. Re: many topics, paused queues, memory growing
            hantunca

            Clebert,

             

            I have pre-ack set on all of my queues.  I'm not using any journaling - queues are setup to drop messages if they reach a certain size. No persistence. Currently using 2.0.0.GA.  Wanted to see if anybody else has run into this problem first (or I was doing something wrong) before digging into the code itself.  I'm assuming what I'm seeing is not the way the code should be behaving - otherwise, when I called isPaused() I would have expected the queue to return true - it's not, it's returning false.

             

            thanks again.

            • 3. Re: many topics, paused queues, memory growing
              clebert.suconic

              Do you think you could reproduce this on a self contained test?

              • 4. Re: many topics, paused queues, memory growing
                clebert.suconic

                Or even better.. can you test this with trunk? (2.1.0 is almost out now)

                • 5. Re: many topics, paused queues, memory growing
                  clebert.suconic

                  resume is just forcing a delivery.

                   

                   

                  If there is a bug in your case (with drops associated), a resume would force the delivery on the queues.

                  • 6. Re: many topics, paused queues, memory growing
                    hantunca

                    Clebert,

                     

                    I will try to replicate against trunk.  If I see it there, I will dig down into the code to see what's going on exactly.  I'm not seeing any messages dropped, no exceptions, etc, so I don't think that's causing the queues to be paused.  However, if a queue goes into a paused state, wouldn't isPaused() return true?

                    • 7. Re: many topics, paused queues, memory growing
                      timfox

                      Quite possibly you are sending messages faster than you can consume them. This will enevitably result in memory exhaustion eventually, unless you enable producer flow control or page messages.

                      • 8. Re: many topics, paused queues, memory growing
                        hantunca

                        Tim,

                         

                        that's what I thought at first and where I started looking.  Eventually, instead of starting up thousands of topics, I started 100 topics and connected to them and still memory was being eaten up (a queue had been paused) - I have a program now that iterates through all of the queues and calls resume - when I run this the messages go through just fine.  This same utility program also tells me how many messages I have in all of the queues - after resuming all, no messages back up in the queue.

                         

                        thanks.

                        • 9. Re: many topics, paused queues, memory growing
                          hantunca

                          Also, in order to guard against running out of memory, the queues are setup to drop messages when they reach a certain size via address-full-policy setting in the config file.

                          • 10. Re: many topics, paused queues, memory growing
                            timfox

                            Remember the max-size-bytes in address settings is *per queue*.

                             

                            So if you set max-size-bytes to 100 MiB and you have 100 queues, you will need 100 * 100 MiB = 10 GiB of RAM!

                             

                            Actually more like 11 GiB due to server overhead

                            • 11. Re: many topics, paused queues, memory growing
                              hantunca

                              Thanks Tim - I realize the setting is per queue.

                               

                              I tested my code against 2.1.0 (build from trunk in the svn repository) - same problem.  Just to give you a little more of an idea on my use case - I start an embedded JMS server - the server creates 2000-3000 topics dynamically via JMX and then starts publishing to the queues without any consumers on the other end.  I then start up a consumer that subscribes to all of the topics - some of the queues created by the consumer are "paused" - when I use JMX and resume them, messages start flowing.

                               

                              By the way, in my testing I found that 2.1.0 seems slower than 2.0.0 - I've been doing latency testing by setting timestamps on the message at server side and comparing it to the time it was received on the consumer side.  Just wondering if optimizations are still to come in 2.1.0?

                               

                              thanks again for your help,

                              Han

                              • 12. Re: many topics, paused queues, memory growing
                                timfox

                                han tunca wrote:


                                 

                                By the way, in my testing I found that 2.1.0 seems slower than 2.0.0 - I've been doing latency testing by setting timestamps on the message at server side and comparing it to the time it was received on the consumer side.  Just wondering if optimizations are still to come in 2.1.0?

                                 


                                I guess it depends what you mean by "speed".

                                 

                                2.1.0 can sustain a significantly higher overall throughput than 2.0. but the latency might be higher.

                                 

                                Often there's a trade-off between latency and throughput.

                                • 13. Re: many topics, paused queues, memory growing
                                  hantunca

                                  Understand about the balance - it would be nice to control that balance from the configuration files as I suspect each customer has their own definition of speed.

                                  • 14. Re: many topics, paused queues, memory growing
                                    timfox

                                    Have you looked at the new ThroughputConnectionFactory in the default hornet-jms.xml in TRUNK/2.1 ?

                                     

                                    If you are sending lots of small messages in quick succession, this gives a big gain in throughput (msgs/sec), although the average latency might be higher.

                                    1 2 3 4 Previous Next