Wildfly 10.1.0. - MessageListener start processing messages after cca 5 minutes

[rabbit10]

I am using Wildfly 10.1.0 with embedded activeMQ (messages on queue are marked PERSISTENT). When wildfly starts, my application immediately starts to generate messages on topics/queues. However, MessageListener doesn't start processing messages immediately, but after cca 5 minutes after wildfly started . I restarted wildlfy few times - every time this delay in message processing is cca 5 minutes. Problem is that there is lot of messages that get generated so activeMQ starts using its paging mechanism and storing messages on disk which ends up with messages just piling up on disk until disk runs out of space.

When I create simple listener outside our application - this listener starts processing messages immediately. What could potentially be reason for MessageListener to start processing messages 5 minutes after wildlfy is started? I see in wildfly server log that all Message Listeners are started.

Thank you in advance!

[jbertram]

Off the top of my head I can't think of any reason MessageListeners in the container would take 5 minutes to start actually receiving messages.  Have you acquired any thread dumps during the 5 minute delay?  Also, can you describe your application architecture a bit more?

Do you have a reproducible test-case you could share (preferably automated)?

[rabbit10]

Thank you Justin for quick reply!

I observed the 5 minute delay in processing through log files and CLI commands. However, once I turned off logging and tracked the message processing only through wildfly CLI commands, I haven't noticed this 5 minute delay.

However, I still have problem with messages piling up after wildlfy restart. I have producer that produces 100 messages every second for topic. The topic consumer processes this messages and publishes 100 messages to queue. Problem is that the queue consumer doesn't process these messages fast enough which results in messages piling up over time. This even happens when I comment all content of onMessage method on queue consumer.

However, when messages start piling up, if I stop our producer (while wildfly is runnning) and  wait till all messages on topic and queue are processed - then, after starting producer (while wildfly is runnning), messages will not pile up but will be processed in "no time". I thought problem is in the thread pool size for activemq connection factory and tried increasing min-pool-size to 20/50/100, but this didn't help. Do you have an idea what would be cause of this - activemq consumer not being able to process 100 messages per seconds after wildlfy restart?

[jbertram]

Can you describe your application architecture a bit more?  For example...What connection factory is the producer using?  Are your consumers MDBs?  If not, how are they written and what connection factories are they using?

I thought problem is in the thread pool size for activemq connection factory and tried increasing min-pool-size to 20/50/100, but this didn't help.

The min-pool-size and max-pool-size attributes of the pooled-connection-factory are not directly related to threads but rather to connection instances.  Also, it's not clear how you are using a pooled-connection-factory (if at all) so it's impossible to say whether or not this would actually have any impact.

Do you have an idea what would be cause of this - activemq consumer not being able to process 100 messages per seconds after wildlfy restart?

You haven't really provided enough information for me to form an opinion.

I asked this previously but didn't receive a direct response - Do you have a reproducible test-case you could share (preferably automated)?

[rabbit10]

Problem occurred when running on VM on my development machine. After deploying my application to dedicated machine the upper problem didn't occur again.

[mnovak]

Did you figure out what was causing the issue? Could you share your consumers and configuration of WF10?

Just as precaution of "out of disc space" symptom, you can set address-full-policy to BLOCK in address-settings (messaging-activemq). This will stop producers to send messages to queue/topic if max-size-bytes (10MB by default) is reached.