Synchronization between clusters

[pedrokowalski]

Howdy Guys,

I'm just getting known with Infinispan (by the way - kudos for getting started guide and many well-written articles!) I just want to know if the following scenario is possible with Infinispan:

I have Cluster A which uses UDP transport protocol (defined in JGroups). I have also Cluster B which uses TCP transport protocol.

Is there any way I can seamlessly synchronize changes made to the cache on Cluster A with Cluster B? The requirement for supporting both TCP and UDP is important, as some of the clients (nodes) depends on either TCP or UDP only. Nevertheless, I would like to make sure that changes made by UDP-only clients to Cluster B will be available for TCP-clients in Cluster A.

If some unified JGroups configuration is not able to make it works - can I somehow achieve the same functionality by e.g. creating an additional node which handles the synchronization?

Is there a way to make this "additonal node" capable for being in the XA transaction?

Thanks in advance for any help!

I'm using Infinispan 5.1.15 and both clusters work in replication mode.

[galder.zamarreno]

We're working on a global solution as part of https://community.jboss.org/docs/DOC-17546, but the 1st alpha of this won't be ready for a few weeks.

There's some hacky workarounds you could use, but I'd wait for the alpha release which should be coming soon...

The JIRA to monitor is: https://issues.jboss.org/browse/ISPN-1824

[pedrokowalski]

Galder, thanks for your reply. Good to know that in the nearest future this will be easily achievable :-)

Is it safe to use the RELAY protocol for Infinispan replication cluster mode?

I've tried to achieve something similar to the bridge concept but rather on a naive level: cluster {A, B, C} using UDP transport and cluster {C, D, E} using TCP transport. The C is a node of the both clusters - it listens for changes in cluster {A, B, C} and puts the changes in cluster {C, D, E}. However, I don't think that it'll work if the {A, B, C} and {C, D, E} should be in different networks, right?

When it comes to the transactions, there is following excerpt in the page you showed: "in the case of the transactions, it would be the commit message that triggers the sending of transaction information to the remote site". So if a transaction in cluster C1 has commited only then I send changes to the cluster C2. What happens if the C2 fails to add the entries to the cache? Does the 'commit message' referrs to the 1PC or the prepare-commit of 2PC?

I'm just asking because I would like to be sure that any changes made to cache in Cluster C1 will be consistent with Cluster C2. If the transaction will be rolled back within the C1, the C2 won't be even notified by it and this seems a simple case. But what when a transaction in C1 is NOT rolled back (it commits successfully) but for whatever reason the cache changes cannot be made and replicated to Cluster C2. Then, the global transaction should be rolled back (neither C1 nor C2 has the entries modified by the transaction). Will this case be also supported?

Thanks in advance!

[mircea.markus]

Is it safe to use the RELAY protocol for Infinispan replication cluster mode?

It is technically named RELAY2 (relay was the first version of this protocol used as POC). It will work with replicated caches, yes. It  is actually possible to have a replicated cache in site A which backups to a distributed cache in site B. 

I've tried to achieve something similar to the bridge concept but rather on a naive level: cluster {A, B, C} using UDP transport and cluster {C, D, E} using TCP transport. The C is a node of the both clusters - it listens for changes in cluster {A, B, C} and puts the changes in cluster {C, D, E}. However, I don't think that it'll work if the {A, B, C} and {C, D, E} should be in different networks, right?

it will work with {A,B,C} in one network/location (e.g. LON) and {C,D,E} in a different location (e.g. NYC). The updated design document for x-site replication can be found here: https://community.jboss.org/wiki/DesignForCrossSiteReplication

When it comes to the transactions, there is following excerpt in the page you showed: "in the case of the transactions, it would be the commit message that triggers the sending of transaction information to the remote site". So if a transaction in cluster C1 has commited only then I send changes to the cluster C2. What happens if the C2 fails to add the entries to the cache? Does the 'commit message' referrs to the 1PC or the prepare-commit of 2PC?

The given document was revisited and the final design is here: https://community.jboss.org/wiki/DesignForCrossSiteReplication.

You'd be basically allowed to have the 2nd cluster be a full tx participant, i.e. answer to both prepare and commit. There's no mutch point in doing x-site 2PC if the 2nd cluster is passive: there won't be any writes going in it and the prepare phase(i.e. when locks are acquired) must succeed. 

Re: what happens if the changes cannot be committed to the 2nd cluster: that would be configurable - we might fail the transaction or allow it to succed. If later, then the admin of the sites must reconciliate the state.

I'm just asking because I would like to be sure that any changes made to cache in Cluster C1 will be consistent with Cluster C2. If the transaction will be rolled back within the C1, the C2 won't be even notified by it and this seems a simple case. But what when a transaction in C1 is NOT rolled back (it commits successfully) but for whatever reason the cache changes cannot be made and replicated to Cluster C2. Then, the global transaction should be rolled back (neither C1 nor C2 has the entries modified by the transaction). Will this case be also supported?

yes, (se above) this will be configurable.

May I suggest to give a read to nbew design document[1] and let us know any concerns you might have.

[1] https://community.jboss.org/wiki/DesignForCrossSiteReplication.

[pedrokowalski]

Thanks a lot Mircea - things are clearer right now.

To explain why I want to achieve such thing is that I have the following constraints:

- nodes within the cluster must use UDP for transport,

- the JTA/XA transaction support must exist between clients and the cluster,

- some clients are not allowed to use UDP and must therefore fallback to TCP instead.

So, given such constraints I was thinking about using the Hot Rod protocol. Unfortunately (https://community.jboss.org/message/754516#754516) XA transactions doesn't work with Hot Rod. Ok, so the next idea was to treat client as a cache node (which gives me the JTA/XA support) and use the TCP bridge between the client-node and the cluster (which can still run in UDP).

And thit is the moment I think I'm overcomplicating things here... I mean - are my constraints really so uncommon (the support for Tx and clients that cannot use the UDP transport)?

What do the others do in such case? Do they revert to the TCP cluster and sacrifice stability and performance? Am I missing something or do I have to wait for the next Infinispan/JGroups releases to solve this problem? (I'm currently also required to use Infinispan 5.1.5)

Thanks once again!

[belaban]

Piotr,

you may want to take a look at RAC (https://community.jboss.org/wiki/RAC): it provides almost the same guarantees as transactional replication across sites, but is much faster, especially if you have high latency on the WAN link connecting the sites.

[belaban]

Note that RAC is the working title only, I'm trying to find a better name...

[pedrokowalski]

Thanks Bela!

I will take a look at it.

BTW. The name recalls me of Oracle RAC ;-)

[belaban]

That's why I'm looking for another name...

[moia]

Have you tried RemoteCacheStore?

I've been looking through the docs recently and this seems like something you want to look into.

Best regards,

Mikolaj

[pedrokowalski]

Thanks Mikołaj for your input.

As far as I can see from the Javadoc (http://docs.jboss.org/infinispan/4.1/apidocs/org/infinispan/loaders/remote/RemoteCacheStore.html) it seems that RCS works with HotRod. This means that I can only use the TCP connection, while one of the requirements is to use the UDP transport protocol as well, so I'm not sure if it's achievable in this way.

Cheers!

[moia]

Indeed, I missed the requirement, that you need to communicate between clusters with both TCP and UDP.