Clustering differences versus JBoss MQ

[kennardconsulting]

Having just upgraded from JBoss 4.2.3.GA (which used JBoss MQ I believe) to JBoss 5.1.0.GA (which uses JBoss Messaging I believe), I am a bit confused as to how the clustered JMS works.

I tried using /ClusteredConnectionFactory, and it seems to work, but its behaviour seems to be clustered delivery of messages to local queues. This is resulting in one of the machines (the fastest one) consuming all its messages and then sitting idle while the slower machines finish their messages. Really what I want is a single queue and clustered consumption of messages by all machines.

How should I configure that?

Regards,

Richard.

[gaohoward]

Using /ClusteredConnectionFactory, each time you gets a connection, you connects to a different node (round robin load balancing by default).

By enable Message Suckers in the cluster, messages will be pulled off from nodes that have no local consumers to the nodes that have local consumers but no messages in queue.

[kennardconsulting]

gaohoward,

Thanks as always for your help. I don't think what you suggest is going to work because:

1. As soon as new messages get delivered to its local queue, it will stop sucking from other nodes. This is fine but it will mess up the delivery order of messages
2. If I turn on 'strict message ordering' presumably it will have to deliver everything to the one node?

However, I found this FAQ which seems close to what I want:

http://www.jboss.org/community/wiki/JBMHASingleton

Unfortunately this FAQ does not work!

Here are some of the problems:

1. There is no deploy/jboss-messaging.sar in JBoss 5.1.0.GA. I assume it is now called deploy/messaging?
2. deploy/messaging includes jms-ds.xml, which if you move into deploy-hasingleton then every cluster node except the primary will fail to start (citing 'DefaultJMSProvider not bound')

All this was much easier under JBoss 4.2.3.GA. Under its all folder it already had most of JBoss MQ installed under deploy-hasingleton but it still had jms-ds.xml (actually hajndi-jms-ds.xml) under deploy.

So I tried moving hajndi-jms-ds.xml from 4.2.3.GA to 5.1.0.GA, and it almost works but the 'server session pool for Message Driven Beans' bit fails to deploy. I tried deleting that and things start up, but the queues don't seem to pick up and begin processing messages after you bounce a cluster node.

Please help! I just want JBoss Messaging to perform like JBoss MQ did: a single HA queue with load-balanced consumers.

Richard.

[gaohoward]

Hi Richard,

The JBM is under deploy/messaging in AS 5.1 as you found.
For how to setup a HA-Singleton cluster, I suggest you post a question to the AS forum, where you can get a better help on this.

Howard

[kennardconsulting]

gaohoward,

Please, please - I am begging you!

I have spent several days on this and one of the main problems is that in all of the documentation and forum postings members of one team keep referring people to another team, who refer people to another team etc.

I understand different people have different areas of expertise, but someone needs to take ownership and provide a one-stop, coherent solution to HA-JMS.

Given that the FAQ that doesn't work is a JBoss Messaging one...

http://www.jboss.org/community/wiki/JBMHASingleton

...could you please take ownership of this issue and verify that, at the very least, your own FAQ works?

Richard.

[clebert.suconic]

JBossMessaging has automatic failover.

If you want to control the failover the same style you would do on JBossMQ, you need to disable failover and do the reconnects yourself.

[kennardconsulting]

Clebert,

Thanks for your input. I am fine to manually failover in my producer, as I control that code.

However the problem I'm having is that, upon restarting a cluster node, the MDBs do not 'wake up' and start consuming messages from the HA queue. The code to connect the MDBs is controlled by JBoss (I presume - it's all magic to me :), so I'm unclear how I can do the reconnects myself?

Regards,

Richard.

[kennardconsulting]

gaohoward,

I would gladly rework and update the Wiki FAQ for you if you can furnish me with a solution :)

Regards,

Richard.

[gaohoward]

Hi Richard, I'll take a look at the wiki. Thanks.

Howard

[kennardconsulting]

gaohoward,

Awesome. You are a legend. Look forward to hearing how you go. If it helps:

1. I made the changes described in the Wiki, then copied the 'all' folder to a new folder (I called it 'cluster2') and started a second instance of JBoss using 'run -c cluster2 -Djboss.service.binding.set=ports-01'. You will see the second instance fails to start because 'DefaultJMSProvider not bound'

2. DefaultJMSProvider is defined in jms-ds.xml. The Wiki instructs you to move this into deploy-hasingleton, which I believe is a mistake. In JBoss 4.2.3 the jms-ds.xml stays in the deploy folder (in the 'all' config)

3. You can try moving jms-ds.xml back to deploy, and things go better. However the MDBs no longer seem to 'wake up' upon bouncing a cluster node

Regards,

Richard.

[kennardconsulting]

Howard,

Hmmm. From watching my logs, I think the reason the MDBs aren't 'waking up' upon bouncing a node is because JBoss Messaging is still creating 3 local queues, not one HA singleton one.

This, despite the fact that destinations-service.xml is under deploy-hasingleton. Do I need to do something extra to make a single, HA queue?

Thanks,

Richard.

[timfox]

"gaohoward" wrote:

Using /ClusteredConnectionFactory, each time you gets a connection, you connects to a different node (round robin load balancing by default).

By enable Message Suckers in the cluster, messages will be pulled off from nodes that have no local consumers to the nodes that have local consumers but no messages in queue.

Actually that is not correct. In JBM 1.x messages will be redistributed from one node to another if JBM determines that consumers on one node are sitting idle, whilst there are messages and busy consumers on other nodes.

If you're seeing some consumers sitting idle while one node consumes all the messages your system is probably misconfigured.

[kennardconsulting]

Tim,

Thanks for helping out.

If I understand correctly, JBM 1.x load-balances at distribution time, not consumption time. In my case, I am sending 1000 messages to the queue in one initial batch.

I see an initial flurry while all 3 of my consumers consume messages, then the fastest one stops waiting for the slowest 2 to finish. It seems each of my consumers got given 333 messages to work on locally, and that consumers cannot consume each other's messages without introducing message suckers?

Thanks again.

Richard.

[timfox]

Actually JBM 1.x does both. (BTW there is a FAQ on this, but I didn't write it so can't vouch for it's accuracy).

When using a clustered connection factory, subsequent *connections* will normally be created to different nodes of the cluster, by default in a round robin fashion.

So connection 1 might be to node B, connection 2 might be to node A, and connection 3 might be to node C. So that determines where you're messages get initially sent.

Now, when the messages reach a node, if they're non persistent and you're using a topic, then they will be instantly sent off to all the nodes where you have consumers using JGroups.

If they're persistent and/or you're using queues then they'll initially go into the local queue.

Each message consumer by default has an internal buffer of messages that it "prefetches" from the queue (this is faster than it fetching a message each time).

The local queue will first attempt to send messages to the local queue until it determines it is "busy" - busy means its buffer is full. If all local consumers are determined to be busy, then messages are eligible to be redistributed to other nodes where consumers might be idle.

If the processing time for each of your messages is significant then it might make sense for you to reduce your prefetch size (see docs) since there is little or no advantage in consumers buffering messages. You should then see a smoother distribution of messages across different nodes in the cluster.

[kennardconsulting]

Tim,

What you say is very interesting. Could you please elaborate on what you mean when you say:

> The local queue will first attempt to send messages to the local queue
> until it determines it is busy

Do you mean the 'local queue will send messages to the prefetch buffer'?

> If the processing time for each of your messages is significant

What kind of time is considered significant?

Regards,

Richard.