-
1. Re: JMX Console locked; JConsole shows no information
timfox Sep 21, 2010 5:58 AM (in response to robertjlee)in older versions, getMessageCount() used to block all delivery on a queue when it was in progress, so it was important that monitoring tools don't call it too often, it could really slow down the system.
In current TRUNK there is no longer a lock. However it's still an expensive operation so don't call it in a tight loop.
-
2. Re: JMX Console locked; JConsole shows no information
robertjlee Sep 21, 2010 6:31 AM (in response to timfox)That's great to hear.
Out of interest, is there a more efficient way to find out which queue at an address contains the largest number of messages?
-
3. Re: JMX Console locked; JConsole shows no information
robertjlee Sep 24, 2010 11:42 AM (in response to robertjlee)Hi Tim,
We've checked out and built the code from TRUNK, and it would appear that, while getMessageCount() now does not block when a topic is blocking, it throws an IllegalStateException instead.
I don't understand why this is an illegal state; the address-full policy of BLOCKing is clearly defined in the manual - so it seems wrong to throw an IllegalStateException here. It's not like the client code is calling methods on an object out of a required order, for example.
What I'm trying to do is identify, when the topic is blocking, which core queue/consumer is not consuming messages, i.e. which queue is causing the block. This is an exceptional case, but currently, the only way I have to do this is to go over all the queues in the topic (using JMX) and get the message count for each one. It seems that it would be an advantage to have a simpler mechanism to test if a queue was blocking without actually counting anything (assuming that the queue causing the error is the one which throws this error), but it would also seem wrong to catch an IllegalStateException as this exception is far too general.
It seems very odd that blocking addresses cause so much to stop working; I've just tried stopping the core bridge that feeds the topic, and the JMX method returns successfully but the bridge's state remains "started=true", and the stacktrace still shows the consumer parked in acquireCredits(). If you attempt to look at the attributes of the core queue MBean on the source of the bridge, then the JMX thread blocks on getConsumerCount() and doesn't recover until HornetQ is restarted - unless you can somehow remedy the problem of a consumer not picking up without using the JMX console!
I am also worried that we may not be able to move messages out of a core queue if its causing its address to block; this really would be a showstopper for us. (Although we've not been able to test this yet).
-
4. Re: JMX Console locked; JConsole shows no information
timfox Sep 24, 2010 12:35 PM (in response to robertjlee)If you can provide test case, instructions, stack trace etc someone can investigate.
-
5. Re: JMX Console locked; JConsole shows no information
timfox Sep 24, 2010 12:39 PM (in response to timfox)AIUI the only reason you do this is so you can avoid a slow subscription blocking delivery on other subscriptions on the same address when paging is in operation.
If so, Clebert is currently implementing new functionality in paging which should allow paging to be configured at the queue level, thus preventing blocking. See dev forum thread for more info.
-
6. Re: JMX Console locked; JConsole shows no information
robertjlee Oct 1, 2010 4:15 AM (in response to timfox)Hi Tim,
We are aware of Clebert's work, but we currently have a JBoss MQ system that isn't coping with the load that we're trying to send at it, and so we want to start moving to HornetQ as soon as possible. The HORNETQ-498 currently says 6 weeks remaining; that's obviously an estimate but I doubt we can keep going for anything like that long on JBoss MQ, hence trying to find a workaround.
We haven't managed to reproduce the problem reliably outwith the network where our server is running, but we have noticed a coincidence between IllegalStateExceptions when the outgoing bandwidth (from the server to the clients) is being fully used/contested; possibly there is something inside the static block in QueueImpl that is retrying or looping when trying to send messages to clients, slowing down JMX calls to the point of timing out?