ClusteredConnectionFactory vs JmsXA

[ejb3workshop]

My application consists of a number of EJB3 MDBs, which pull messages off a number of queues and topics. The application is deployed on a cluster of 2 identical nodes. I have installed JBM 1.4(SP3) on JB 4.2.2.

When I inject the ClusteredXAConnectionFactory the application work, but the messages are not treated i the same transaction as the rest of the work done in my MDB. For example, I would like to persist an Entity as well as forward a message to anther MDB. The other MDB is dependent on the entity being persistent and available in the database.

However with ClusteredXAConnectionFactory it seems that message is sent outside the transactional context. Rather frequently it happens that the message arrives at the next bean, but the persistence has't finnished. The messages are however distributed evenly across both nodes.

After a little research online, I found several posts suggestion to use JmsXA instead of ClusteredXAConnectionFactory. This resolved my transaction problem nicely, however since the switch the messages are no longer distributed evenly. It seems to only use the other node, when all MDBs in the pool on one node are exhausted.

I am wondering if this is by design, as clustered XA transactions must be quite expensive, surely. So maybe messages are only distributed to the other nodes when one nodes bean pool is exhausted.

Please let me know if this is the correct behaviour, or what I should look at to correct the distribution.

[timfox]

"ejb3workshop" wrote:

However with ClusteredXAConnectionFactory it seems that message is sent outside the transactional context. Rather frequently it happens that the message arrives at the next bean, but the persistence has't finnished. The messages are however distributed evenly across both nodes.

This is a classic misconception with using XA.

If you do something like:

a) write a row in a DB
b) send a message

In an ejb and use XA to enlist both those operations in the same global tx, then on commit, you might expect a) to happen before b) right?

Wrong. The XA spec makes no guarantees of the order or concurrency of when the two things occur. All it says is they will both happen.

That's just the way XA works, nothing to do with JBM.

So if you're application expects to find the row in the db when it consumes the message , that won't be guaranteed to work.

[ejb3workshop]

I see your point now. Do you have any further information on why the message distribution when using JmsXA differs from ClusteredXAConnectionFactory ?

I am observing that messages are only distributed to other nodes in the cluster once the MDB pool is exhausted. Is this the correct / intended behaviour ?

Are there any configuration options to alter the message distribution between nodes ?

Thanks in advance

[timfox]

You shouldn't use a clustered connection factory with MDBs - if you look - the default configuration uses a non clustered connection factory.

Using a clustered connection factory means subsequent create connection attempts will round robin - this is typically not what you want with MDBs.

MDBs should always consume from the local machine.

This would explain what you are seeing.

[timfox]

I believe there have been several other threads on this topic in this forum in the past.

[ejb3workshop]

"timfox" wrote:

Using a clustered connection factory means subsequent create connection attempts will round robin - this is typically not what you want with MDBs.

But this is exactly what we want. We would like our messages to be distributed using RoundRobin accross the nodes in the cluster. Does that mean I should change the default connection factory used in hajndi-jms-ds.xml to ClusteredXAConnectionFactory?

[timfox]

I think you're misunderstanding how clustering works.

Let me give an overview again. This has also been covered on (many) other threads

If you use a connection factory with load balancing = true to *create* a connection to the cluster then subsequent create connection attempts will connect to different nodes in the cluster in a round robin fashion.

So connection 1 on node 2, connection 2 on node 3, connection 3 on node 1, connection 4 on node 2 etc. The first node is chosen randomly.

When you use one of those connections to *send* messages then all the messages sent by that connection will go to the node that connection is connected to. Different messages sent *do not* get sent to different nodes. That would be extremely hard to do and still maintain transactional semantics without being *extremely* slow.

Ok, that's for sending messages, how about consuming messages?

If you create a connection, this time to *consume* messages using a connection factory with load balancing = true, then, again, subsequent connection are created to different nodes of the cluster in a round robin fashion.

So if you deployed, say, 3 different MDBs on node A with a connection factory with load balancing = true, then they would actually be connected to and consuming from *different nodes of the cluster*.

For an MDB installation this is typically not what you want. With MDBs you normally deploy the same set of MDBs on each node of the cluster.

So node A, B, and C all have the same set of MDBs. In this scenario it makes no sense for the MDBs one node A to actually be consuming from the nodes B and C. It is far more efficient for the MDBs on node A to be only consuming from node A, the MDBs on node B to be consuming from node B etc.

As long as you have a good distribution of connections *sending* messages across the cluster, then this is the optimal configuration. You're not wasting network bandwidth moving messages between nodes when there's a consumer on the local node that could quite happily consume the message.

Now, JBM goes one step further. It has something called "message distribution".

Consider the case where consumers on node A are slower than consumers on node B. In that case it may make sense to move messages from A to B in order to utilise the cluster effectively.

JBM does this automatically. It will transparently shift messages from a queue on node A to a queue on node B according to load. You don't need to connect to the other node directly to do this. Just connect to the local node, JBM does the rest.

I hope that helps clear up the confusion.

As I say, there have been lots of other threads like this in the past - it's worth taking a look.

[ejb3workshop]

In our case we have one message coming into the cluster from an external JMS client. This message is then processed by a chain of MDB. Each node in the cluster has the same MDBs deployed. (Like you said).

Our problem is that one incoming message results in several (sometimes 100 / 1000) individual messages to be send to the other MDBs in the chain. So one message comes in, and many go to the next MDB. The current behavior we are observing is hat messages only get distributed to other nodes in the cluster, when the local bean pool is exhausted.

I appreciate the overhead of always distributing messages to other nodes, but would like to verify that this behavior is as intended and if there are any configuration options I can use to alter it.

[timfox]

"ejb3workshop" wrote:

Our problem is that one incoming message results in several (sometimes 100 / 1000) individual messages to be send to the other MDBs in the chain. So one message comes in, and many go to the next MDB. The current behavior we are observing is hat messages only get distributed to other nodes in the cluster, when the local bean pool is exhausted.

Right, but isn't that an optimal use of resources?

If you have a consumer on each node, then it always makes sense to send the work to the local consumer if it is idle, and only send it to a remote consumer if it is busy. This is what JBM does automatically.

If you send work to another node when it can be processed on the local node that's an inefficient use of resources (extra network round trip etc).

[ejb3workshop]

That depends how one looks at it. I completely accept your argument on using resources optimally. However the bean pool size might not be the best measure. If we deploy our application onto a cluster of nodes using single core CPU, a smaller bean pool is required to get optimal distribution, while if each node were a fancy quad core, a large bean pool would work just as well. I was hoping for an option to always distribute the work on a round robin basis. This would allow me to configure a standard bean pool, aiming to prevent overloading the system, rather then tuning the been pool size according to the hardware configuration of the cluster.

Maybe I misunderstand how load and idle periods are determined. Just from my observation it seems to be driven by the pool size.

[timfox]

JBM knows nothing about MDBs, MDBs and all their configuration params are part of the app server, and not part of JBoss Messaging.

All JBM knows about is message consumers, and has no idea of whether the consumer is an MDB or some other kind of consumer.

JBM determines if a consumer is busy on whether its buffer is full.

If you round robin on consumption too, it won't help you since those machines with smaller or fewer cpus will get, on average the same number of consumers on them, exactly as if you consuming from the local node. So I don't think that helps you.