Lower than expected concurrency with message grouping
brettniven Apr 13, 2014 11:07 PMHi,
I'm experiencing lower than expected concurrency in a scenario that uses message grouping. I wondering whether I'm using one or more anti-patterns or whether this may be a JBoss/HornetQ integration issue. My scenario:
- JBoss 7.1.3.Final, HornetQ 2.2.21.Final
- Many producers send messages of varying sizes (let's say an average message size of 100KB)
- Messages are grouped by setting a JMSXGroupID
- Most messages are written to the filesystem's large messages dir (by HornetQ), as expected
- I am using MDB's as message consumers. These are 'slow consumers'. Default MDB instance size of 15.
- When there are >= 15 unique JMSXGroupID's in the queue, I expect concurrency of 15 (I expect 15 active MDB's)
Problem:
- When 'x' (let's say 100) messages with the same JMSXGroupID are at the top of the queue, I end up with a concurrency of 1 (a single active MDB), even if there are many more items in the queue with differing JMSXGroupID's
- It appears that only the first 'x' messages are 'considered for allocation' to consumers/MDB's
- When debugging and checking queue properties, I found that the JMSQueueControl.getDeliveringCount() returned a small figure (< 100), inferring that only that figure had been 'allocated to a consumer' (and all to the same consumer as they have the same message group)
How I've fixed it temporarily:
- I realised that it was not 'x messages' that were considered for allocation, but instead 'x bytes'. Proven by mocking a test with much smaller message sizes
- So, after reading up on Flow Control, I've increased consumer-window-size (100MB resolves my scenario)
- After doing so and re-running my test, I have an expected concurrency of 15 and JMSQueueControl.getDeliveringCount() returns the length of the queue - inferring that all messages have been 'allocated to consumers'
But this is undesirable for several reasons:
- The tests and tmp solution infer that before messages are allocated to a consumer (in my case, MDB's), the full message is read into memory.
- I had hoped that only message header properties would be read before allocating to a consumer and that the consumer would then only read the message into memory immediately prior to processing
- I ideally only want 15 full messages read into memory at any one time. In my example/test, I don't want the first 100 messages loaded into memory unnecessarily until immediately before they will be processed
- These are slow consumers so I would ideally like to set consumer-window-size to 0, meaning no buffer for each consumer
Questions:
- In a JBoss env, is each MDB considered a JMS consumer, or is there perhaps a single consumer that then delegates to the MDB's?
- i.e. When setting consumer-window-size to 100MB, am I allowing 100MB per MDB (of which there may be 15, so 1.5GB total) or 100MB per jvm?
- My tests suggest that there is a single consumer process somewhere. If all MDB's were considered JMS consumers, then the problem wouldn't have surfaced - as all 15 MDB's would have used the default consumer-window-size of 1MB and would have each had a message to process, even if the messages were way down the queue
- Is it true that the full message (including message body) is read into memory before allocating to consumers? This appears to be the case.
Am I using anti-patterns?
- Should I perhaps write message bodies to disk myself and keep the jms message body tiny - such as a mere path to the large message content? I had hoped to not have to do this seeing as HornetQ already has logic to write/read to/from the filesystem
- Should I perhaps not use MDB's and implement my own consumers?
Any tips/pointers/feedback much appreciated.
Regards,
Brett