Fail to consume messages from remote machine

[concombremasqué]

Hi there,

I use ActiveMQ 5.3.0.3 (fuse distro) in FUSE ESB 4.1.0.2. If I launch my consumer program on the same machine as the broker (but in a different JVM) I can successfully consume the messages I've posted from my bundles in the ESB.

If I run my consumer program from another computer: I see on the ActiveMQ Web Console that I am listening on the queue, that messages are in the queue (pending) but they are never consumed!

My program is very simple (no message selector, just listening on the queue and consuming messages) and it works when run on the same machine.

Is there an option related to remote consuming of messages or a parameter in the configuration of ActiveMQ that I've missed?

Thanks in advance.

[gseben]

Hi,

Can you post your activemq.xml config file?

Thanks,

-Gio

[concombremasqué]

Thanks for looking at this.

Find attached my ActiveMQ configuration. This is the config I put into the 'deploy' folder of my ESB.

[gseben]

Your config looks fine. I'm not sure how a consumer could connect to a queue and not pick up messages sent to it...

Could you attach a screen shot that shows the consumer details and another with the queue details?

Thanks,

-Gio

[concombremasqué]

Here are the details you asked for (queue and consumer).

[concombremasqué]

Just for information:

For testing purpose, I've changed the broker from ActiveMQ to OpenJMS and the issue does not show up (i.e.: I can consume messages from a distant machine).

What conclusion can be drawn here? Is it a bug in ActiveMQ 5.3.0.3 (in this case it would be a huge bug and I cannot believe that) or a problem in my ActiveMQ configuration (but it seems fine: refer to my previous post to look at it)?

[concombremasqué]

After further investigations, it appears ActiveMQ is not the real issue as I manage to post/consume messages using distant machines and sample JMS sender/receiver test programs.

The issue is then related to the way my bundles, that make use of the camel activemq component, post JMS messages into the broker. There's something weird here...

Keep investigating...

[gseben]

From your screenshots it seems that messages are being dequeued from the workflow.input and enqueued in the DLQ.workflow.input queue. Have you tried receiving from the DLQ to see what the messages in there are?

Are you setting a time to live on these messages? If so, how long?

-Gio

[concombremasqué]

Hi Gio,

Yes messages are enqueued in DLQ because my consumer fails to "see" them in the workflow.input queue. Messages in the DLQ are all correct messages (content and structure ok).

And everything works fine when my consumer is on the same machine as the ESB+ActiveMQ. And no there is no TTL set on the messages.

What I said in another of my posts is that I have the feeling that there is some problem in the way my bundles post messages in the queue. The messages are posted using camel activemq component (I use camel 1.6.1.0). Is this possible that this strange behaviour could be due to some issue solved in later camel versions (1.6.1.1 or 1.6.1.2)?

What leads me to believe such a thing is that I sent some messages directly into ActiveMQ queue (not through my bundles that make use of camel-activemq) and in this case I am able to consume these messages from a distant machine.

Maybe I should forward this post and my thoughts in the Fuse Mediation Router sub-forum...

[concombremasqué]

Ok. I have some news.

I can observe that when I successfully consume messages (case where ESB+ActiveMQ and my consumer program are running on the same machine) the "Dispatched Queue" counter always remains set to 0 and the "Dequeues" counter is correct. Everything is fine in this case (refer to consumers_success.jpg screenshot).

When I run my consumer program from a different machine, then the program does not consume anything, yet messages are posted in the right queue. What can be seen then is that the "Dispatched Queue" counter is incremented whereas "Dequeues" counter is 0.

What's the meaning of that?

Again thanks as I am really stuck on this issue.

[gseben]

For an explanation on what the dispatch queue is check out: http://activemq.apache.org/scaling-the-depth-of-a-queue.html

This is not unique to the consumer but to the destination that the consumer is listening on.

The dispatch count keeps track of the number of messages that were sent to the consumer by the broker, while the dequeue count increases when the consumer sends and ack back to the broker for the message that it received.

So from the looks of it, the broker attempts to send a message to your consumer but for some reason the consumer finds that it cannot ack the messaged and it ends up in the DLQ. This looked to me like there was a time to live on the message where the consumer gets a expired message.

What does your camel route look like?

-Gio

P.S: Use jconsole instead of the webconsole. You will get better visibility into things.

[concombremasqué]

My camel route:

<endpoint id="workflowInputEndpoint" uri="jms:queue:$" />

[concombremasqué]

As a side note I should also bring this to you:

If I bypass the camel-jms component and send the message into the queue using direct calls to the JMS API in a pojo called by my camel route then it works fine. Strange isn't it?

[concombremasqué]

It seems you are right regarding the TTL. But its value is huge enough to be able to pick the message (Header 'Expiration' is set to 1254499608359 in the last message sent in my queue).

How can I disable TTL in camel-jms? (I tried  in the JmsComponent bean declaration but it does not help)

[gseben]

That time stamp looks to be in epoch time. But it seems to have three extra digits at the end. Must be part of a convention used by AMQ, my guess is that it represents the the timezone. Anyway, if you remove the last three digits from it you get: 1254499608 = Fri 02 Oct 2009 12:06:48 PM EDT (my timezone).

I'm not sure how to disable this in camel.

Another thing to keep in mind is that if there is a time difference in the clocks between the two machines (unrelated to timezone), then even if this time is large it will make the message expire.

-Gio