We have an application that uses queues to connect different components. All the queues sit on a JMS server with their producers and consumers on remote servers. It works fine when we have just one JMS server. Now we want to create a cluster of JMS servers for load balancing and fail over. JBoss Clustering is successfully working. However, messages always get stuck in the queues, also it seems to be random on which queue they get stuck in.

Here is our setup:

- two JMS servers A & B, JBoss AS 5.1.0.GA with JBoss Messaging 1.4.3.GA

- Unique jboss.messaging.ServerPeerID per node (1 and 2 respectively)

- Both servers have queues deployed in farm, with Clustered=true

- Singleton consumer for each queue with message selector, it connects to the queue with the clustered JNDI port (A:1100,B:1100)

- Clustered connection factory is used, with supportsFailOver, supportsLoadBalancing both set to true.

- Based on https://issues.jboss.org/browse/JBMESSAGING-1713, added in the dependency on the connection factory for ClusterPullConnectionFactory on JMS servers so that message sucker is enabled to pull messages from one JMS server to another

Also, we found that the consistent way to get around this issue is just to shut down one JMS server, all the stuck messages will be failed over to the other server and get processed fine.

After reading this article ((http://community.jboss.org/wiki/JBMCluster), especially the last two notes in the article, my understanding is that our scenario cannot be achieved with JBoss messaging cluster since our singleton consumer may not be able to get all the qualified messages from remote server. Am I understanding it correctly? I have been digging this issue for quite a while, any suggestions and clarification would be greatly appreciated. Thanks!!