[clebert.suconic]

One other thing that makes deadlock far more likely is iterating over the JMX counters while embedded (using java.lang.management.ManagementFactory.getPlatformMBeanServer() and recursively iterating over everything in queryMBeans(null,null)). I narrowed it down to a possibly involving AtomicIntegers and synchronized methods in QueueImpl.java, but when I put a Thread.sleep(1) between each JMX value fetch, my problems went away, presumedly because other threads were given a chance to release their locks.

I believe this will be another issue. You may introduce other issues with competing with the Queue if you constantly (too oftenly) look at the counters. That would be an anti-pattern.

[postmaxin]

But I want to graph counters like MessagesAdded in near-realtime Like I said, the problem pretty much goes away if a Thread.sleep(1) is introduced between each counter. But on the whole, things just shouldn't deadlock, ever........ at worst they should slow down.

[carl.heymann]

I agree, deadlocks should never happen.

Requesting message counters often is not that strange. If one double-clocks a queue attribute in jconsole, it generates a graph over time, with one point every few seconds. I guess this is not the as often as Tyler wants to do it though.

[carl.heymann]

I created the JIRA: https://issues.jboss.org/browse/HORNETQ-746  Sorry I took so long

[postmaxin]

Carl, I was actually doing my polls every 10 seconds. Somehow, the deadlock appeared to be with consumers, because on occasion while JMX counters were being polled, consumers would start timing out (usually on packet 32). Since I've added Thread.sleep(1) in my JMX loop server-side, and added setConfirmationWindowSize client-size, things seem to have cheered up a lot, but if it appears again I'll hit it with a larger hammer and post some more details.

[clebert.suconic]

There are two issues here:

One is this "possible deadlock" with large messages (the test provided) that I didn't have time yet to investigate. (what will happen this week).

The test provided is not using JMX at all (as far as I can see).

The other case is if you use JMX on every message, you will be synchronizing the queues to get your message. You should't use JMX that often. (it's not how a message system is supposed to be used).

[postmaxin]

Clebert,

   I agree that's quite an issue. Why should the queues be synchronized (an active operation) in order to obtain statistical infromation from JMX (a passive operation)? It really seems like accessing JMX counters should always be absolutely passive to the application, even if it means serving data that is a few milliseconds out of date.

[clebert.suconic]

Why would anyone need to access counters on an queue?

JMX is optimize for managing it.. not for the messaging operation.. That's quite a bad pattern.. (for any Message system you chose).

If you need something synchronous like that you probably need a database, a cache. It seems a wrong use case to me.

[postmaxin]

Why would anyone need to access counters on a queue?

Because I want to know how many messages are  in the queue, of course!

I'd also like to keep track of the DeliveredCount... but that's a feature request, not a bug. :-)

[zohar_melamed]

We have the same requirement / design.

Our application pulls a set of counters for all destinations from jmx every 20 secs or so. ( consumer counts, message counts etc )

We push the data into a RRD ( JRobin )  so we can then chart queue behaviour over time at various period sizes.

Is there any doubt that a product like HornetQ broker should never deadlock ? thats just the base-line expected from any production quality middleware.

As for the JMX calls , they should be ultra cheap and passive , and require no synchronization to read - at the cost of some inaccuracies.

[carl.heymann]

When querying message counts via jmx, I assume the request goes through a QueueControlImpl first (server-side), then eventually the call goes through QueueImpl.getInstantMessageCount(), which synchronizes on the queue itself. I assume this synchronized block is what Tyler was asking about.

[clebert.suconic]

Tyler MacDonald wrote:

Why would anyone need to access counters on a queue?

Because I want to know how many messages are  in the queue, of course!

I'd also like to keep track of the DeliveredCount... but that's a feature request, not a bug. :-)

A destination knowing how many messages you have in a queue is a bit of an anti-pattern. You should just receive a message asynchronously.

If you want to use JMX to receive your messages.. just use JDBC or JBoss Cache.

JMX is not made to be part of the life cycle of your consumer. It won't scale if you use JMX for every received message.

[clebert.suconic]

Carl Heymann wrote:

When querying message counts via jmx, I assume the request goes through a QueueControlImpl first (server-side), then eventually the call goes through QueueImpl.getInstantMessageCount(), which synchronizes on the queue itself. I assume this synchronized block is what Tyler was asking about.

Correct.. it won't scale!

[zohar_melamed]

JMX Counters are used to keep track of how the messaging part of the system is performing - thuis is completly orthogonal to message consumption/production. We keep the same counters for CPU , Network , Jetty Requests etc. Is  there is a better way - using message counters perhaps ?

[postmaxin]

Clebert,

     My destinations are not asking how many messages are in the queue. My reporting and alerting systems (eg; rrdtool and nagios) are. I don't think it is unreasonable for them to check on the queue sizes every 10 seconds and alert if there is trouble.