Cluster message redistribution / lost messages
sv_srinivaas Feb 18, 2010 5:58 AMHI,
I have an issue with Hornetq2.0.0.GA and JBoss 5.1.0 GA where the messages are not picked sometimes by the consumer even though the redistribution is enabled. Sometimes the messages are lost when there are no consumers. This issue happens only in Linux but not in Windows.
I have a queue cluster of two nodes nodeq1 and nodeq2. I have another cluster of two nodes nodeb1 and nodeb2 where I have my MDBs deployed.
Steps to replicate the issue
Started both the queue nodes nodeq1 and nodeq2
Started one consumer node nodeb2 for MDB2 to consume from nodeq2
Sent 4 messages and all got consumed by MDB2 from nodeb2
Started nodeb1 also for MDB1 to consume from nodeq1 (and anyway MDB2 should consume from nodeq2)
Sent 4 messages and two got consumed by MDB1 and two by MDB2
Stopped nodeb1
Sent 4 messages and all 4 got consumed by MDB2
Stopped nodeb2 as well
Sent 4 messages, and I encountered 3 different scenarios
Scenario1: Sometimes all 4 messages were sent to nodeq2 and all remained unconsumed
Scenario2: Sometimes 2 messages were sent to nodeq1 and two messages to nodeq2 and all remained unconsumed
Scenario3: Sometimes 4 messages were sent to nodeq1 and ALSO 4 messages to nodeq2 and all of them were shown as consumed in jmx console even though there were no consumers to pick the message in both the queue nodes. Also the below exception was reported in both the queue logs for Scenario3 alone
2010-02-18 06:12:51,113 WARN [org.hornetq.core.client.impl.FailoverManagerImpl] (Thread-8 (group:HornetQ-client-global-threads-717964854)) Failed to connect to server.
2010-02-18 06:12:59,251 WARN [org.hornetq.core.postoffice.impl.PostOfficeImpl] (New I/O server worker #1-2) Duplicate message detected - message will not be routed
2010-02-18 06:12:59,859 WARN [org.hornetq.core.postoffice.impl.PostOfficeImpl] (New I/O server worker #1-2) Duplicate message detected - message will not be routed
2010-02-18 06:13:00,663 WARN [org.hornetq.core.postoffice.impl.PostOfficeImpl] (New I/O server worker #1-2) Duplicate message detected - message will not be routed
2010-02-18 06:13:01,488 WARN [org.hornetq.core.postoffice.impl.PostOfficeImpl] (New I/O server worker #1-2) Duplicate message detected - message will not be routed
Again Started nodeb2 for MDB2 to consume from nodeq2
Sent 4 messages and two were sent to nodeq1 and two to nodeq2. Only two messages from nodeq2 were consumed by MDB2 and remaning 4 messages in nodeq1 remanined unconsumed even though the redistubution is enabled in both the nodeq1 and nodeq2. (We have set forward-when-no-consumers to false and redistribution-delay to 0 in both the nodes)
I've also attached the xmls for all 4 nodes. pls help
-
hornetq-xmls.zip 23.7 KB