-
1. Re: many topics, paused queues, memory growing
clebert.suconic May 11, 2010 4:34 PM (in response to hantunca)You probably have messages building up (not being consumed) and waiting ACK before resume delivery.
When you called resume on the queue, that forced delivery as other queues were probably delivering before that.
What version are you currently using? can you tweak some of your configs on the max-size... paging.. etc?
It would be nice if you could check how it behaves on trunk.
-
2. Re: many topics, paused queues, memory growing
hantunca May 11, 2010 4:45 PM (in response to clebert.suconic)Clebert,
I have pre-ack set on all of my queues. I'm not using any journaling - queues are setup to drop messages if they reach a certain size. No persistence. Currently using 2.0.0.GA. Wanted to see if anybody else has run into this problem first (or I was doing something wrong) before digging into the code itself. I'm assuming what I'm seeing is not the way the code should be behaving - otherwise, when I called isPaused() I would have expected the queue to return true - it's not, it's returning false.
thanks again.
-
3. Re: many topics, paused queues, memory growing
clebert.suconic May 11, 2010 4:47 PM (in response to hantunca)Do you think you could reproduce this on a self contained test?
-
4. Re: many topics, paused queues, memory growing
clebert.suconic May 11, 2010 4:48 PM (in response to clebert.suconic)Or even better.. can you test this with trunk? (2.1.0 is almost out now)
-
5. Re: many topics, paused queues, memory growing
clebert.suconic May 11, 2010 4:49 PM (in response to hantunca)resume is just forcing a delivery.
If there is a bug in your case (with drops associated), a resume would force the delivery on the queues.
-
6. Re: many topics, paused queues, memory growing
hantunca May 11, 2010 4:53 PM (in response to hantunca)Clebert,
I will try to replicate against trunk. If I see it there, I will dig down into the code to see what's going on exactly. I'm not seeing any messages dropped, no exceptions, etc, so I don't think that's causing the queues to be paused. However, if a queue goes into a paused state, wouldn't isPaused() return true?
-
7. Re: many topics, paused queues, memory growing
timfox May 11, 2010 5:02 PM (in response to hantunca)Quite possibly you are sending messages faster than you can consume them. This will enevitably result in memory exhaustion eventually, unless you enable producer flow control or page messages.
-
8. Re: many topics, paused queues, memory growing
hantunca May 11, 2010 5:18 PM (in response to timfox)Tim,
that's what I thought at first and where I started looking. Eventually, instead of starting up thousands of topics, I started 100 topics and connected to them and still memory was being eaten up (a queue had been paused) - I have a program now that iterates through all of the queues and calls resume - when I run this the messages go through just fine. This same utility program also tells me how many messages I have in all of the queues - after resuming all, no messages back up in the queue.
thanks.
-
9. Re: many topics, paused queues, memory growing
hantunca May 11, 2010 5:20 PM (in response to timfox)Also, in order to guard against running out of memory, the queues are setup to drop messages when they reach a certain size via address-full-policy setting in the config file.
-
10. Re: many topics, paused queues, memory growing
timfox May 12, 2010 1:33 PM (in response to hantunca)Remember the max-size-bytes in address settings is *per queue*.
So if you set max-size-bytes to 100 MiB and you have 100 queues, you will need 100 * 100 MiB = 10 GiB of RAM!
Actually more like 11 GiB due to server overhead
-
11. Re: many topics, paused queues, memory growing
hantunca May 12, 2010 2:08 PM (in response to timfox)Thanks Tim - I realize the setting is per queue.
I tested my code against 2.1.0 (build from trunk in the svn repository) - same problem. Just to give you a little more of an idea on my use case - I start an embedded JMS server - the server creates 2000-3000 topics dynamically via JMX and then starts publishing to the queues without any consumers on the other end. I then start up a consumer that subscribes to all of the topics - some of the queues created by the consumer are "paused" - when I use JMX and resume them, messages start flowing.
By the way, in my testing I found that 2.1.0 seems slower than 2.0.0 - I've been doing latency testing by setting timestamps on the message at server side and comparing it to the time it was received on the consumer side. Just wondering if optimizations are still to come in 2.1.0?
thanks again for your help,
Han
-
12. Re: many topics, paused queues, memory growing
timfox May 12, 2010 3:27 PM (in response to hantunca)han tunca wrote:
By the way, in my testing I found that 2.1.0 seems slower than 2.0.0 - I've been doing latency testing by setting timestamps on the message at server side and comparing it to the time it was received on the consumer side. Just wondering if optimizations are still to come in 2.1.0?
I guess it depends what you mean by "speed".
2.1.0 can sustain a significantly higher overall throughput than 2.0. but the latency might be higher.
Often there's a trade-off between latency and throughput.
-
13. Re: many topics, paused queues, memory growing
hantunca May 12, 2010 3:31 PM (in response to timfox)Understand about the balance - it would be nice to control that balance from the configuration files as I suspect each customer has their own definition of speed.
-
14. Re: many topics, paused queues, memory growing
timfox May 12, 2010 3:36 PM (in response to hantunca)Have you looked at the new ThroughputConnectionFactory in the default hornet-jms.xml in TRUNK/2.1 ?
If you are sending lots of small messages in quick succession, this gives a big gain in throughput (msgs/sec), although the average latency might be higher.