How to replicate cache in different Blade in a Data Center?

[udit-mishra-5113a21a]

Our current architecture comprises of two identical configurations in a data center, lets say B1 and B2 (Blades)

Everything is duplicated i.e. Application, DB server etc

Initially B1 functions and B2 remains in passive mode.

If the entire B1 gets down, B2 can take over.

Our Applications and DBs are configured that way, like in SQL server, it first commits the transaction in B2 and then in B1

Now, as per the above architecture, we would like to duplicate our cluster in both B1 and B2 .

What we want is that at any point of time, the cluster of B2 should contain the identical data as of B1 , i.e. if I put one entry in cache and if it goes in B1NodeA, then it should also be put in B2NodeA.

In short, we want distributed cache in a cluster, yet it should be replicated in the different Blade

First question, is it possible?

If yes, how can we accomplish that?

[nadirx]

Sure you can.

Look at cross-site replication: http://infinispan.org/docs/8.2.x/user_guide/user_guide.html#CrossSiteReplication

[udit-mishra-5113a21a]

Is it available from 8.2.x onwards?

[nadirx]

It's been available since Infinispan 6.0

[udit-mishra-5113a21a]

Hello nadirx,

Sorry to say, but the reference seems outdated. I could not find mentioned infinispan.xml. Could you provide me an updated reference with the exact tags and files mentioned, please?

[nadirx]

That reference is for embedded mode. In server mode the jgroups transport needs to be configured in a slightly different way. Look in the server distribution under docs/examples/configs for the clustered-xsite.xml file for an example on how to configure xsite replication

[udit-mishra-5113a21a]

Thank you nadirx , I went through the specified example. I tell you step by step what I did, may be then you could tell me what I did wrong.

I have two DC at two sub domains , first is UDIT and the second is JBOSS (will be used as Backup Site).

Changes made in UDIT's domain.xml

(a) Added <channel name="xsite" stack="tcp"/>

(b) Made relay entry as

                <relay site="UDIT">

                      <remote-site name="JBOSS" channel="xsite"/>

                </relay>

(c) Then Added JBOSS in the Back up site as

              <distributed-cache name="default" mode="SYNC" segments="20" owners="2" remote-timeout="30000" start="EAGER">

                        <locking acquire-timeout="30000" concurrency-level="1000" striping="false"/>

                        <transaction mode="NONE"/>

                        <backups>

                                <backup site="JBOSS" strategy="SYNC" enabled="true" />

                        </backups>

              </distributed-cache>

Changes made in JBOSS's domain.xml

(a) Added <channel name="xsite" stack="tcp"/>

(b) Made relay entry as

               <relay site="JBOSS">

                      <remote-site name="UDIT" channel="xsite"/>

                </relay>

(c) Then said this is a remote cache

                         <distributed-cache name="default" mode="SYNC" segments="20" owners="2" remote-timeout="30000" start="EAGER">

                                <locking acquire-timeout="30000" concurrency-level="1000" striping="false"/>

                                <transaction mode="NONE"/>

                                <backup-for remote-cache="default" remote-site="UDIT"/>

                         </distributed-cache>

Now, I started DC on UDIT. It started fine, no messages for joining bridge come however.

Then I started DC on JBOSS. It also started fine, with a message -  [Server:ServerOne] 15:33:10,186 INFO  [org.jgroups.protocols.relay.RELAY2] (Timer-2,Jboss1:ServerOne) _Jboss1:ServerOne:JBOSS: joined bridge cluster 'xsite'

Then I wrote code to put cache in UDIT and get from JBOSS, but it gives me null. The error on UDIT's console is - [Server:ServerOne] 16:08:06,373 WARN  [org.infinispan.xsite.BackupSenderImpl] (HotRodServerWorker-8-1) ISPN000202: Problems backing up data for cache default to site JBOSS: java.lang.NullPointerException

However on Server Management Console, the default cache is shown as Remotely backed up

What did I miss out?

[udit-mishra-5113a21a]

Today I see fresh error message -

[Server:ServerOne] 11:11:01,220 ERROR [org.jgroups.protocols.relay.RELAY2] (HotRodServerWorker-8-2) Udit:ServerOne: no route to JBOSS: dropping message

[Server:ServerOne] 11:11:11,222 WARN  [org.infinispan.xsite.BackupSenderImpl] (HotRodServerWorker-8-2) ISPN000202: Problems backing up data for cache default to site JBOSS: org.infinispan.util.concurrent.TimeoutException: Timed out after 10 seconds waiting for a response from JBOSS (sync, timeout=10000)

I am sure, some communication details is missing, but whats that, I have little idea. Kindly help me guys nadirx , galder.zamarreno

[udit-mishra-5113a21a]

They both join the bridge cluster 'xsite' , but the node UDIT is not able to backup data to JBOSS.

I do not understand what I have missed..

[udit-mishra-5113a21a]

Hey nadirx , is there any well documented resource for implementing cross site replication with infinispan 8.2 for server mode? I am stuck and can not find resources

[pruivo]

Hi Udit,

Can you please attach your domain.xml? and, if possible, the logs.

Thanks

Pedro

[udit-mishra-5113a21a]

Hello Pedro,

I have attached my domain.xml file

Below are the console log when I put in UDIT and try to get from JBOSS -

[Server:ServerOne] 16:30:47,433 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-5) DGENDPT10002: REST mapped to /rest

[Server:ServerOne] 16:30:47,590 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: Infinispan Server 8.2.0.Final (WildFly Core 2.0.10.Final) started in 21242ms - Started 208 of 264 services (154 services are lazy, passive or on-demand)

[Server:ServerOne] 16:31:00,551 INFO  [org.jgroups.protocols.relay.RELAY2] (Timer-2,Udit:ServerOne) _Udit:ServerOne:UDIT: joined bridge cluster 'xsite'

[Server:ServerOne] 16:33:48,785 ERROR [org.jgroups.protocols.relay.RELAY2] (HotRodServerWorker-8-1) Udit:ServerOne: no route to JBOSS: dropping message

[Server:ServerOne] 16:33:58,789 WARN  [org.infinispan.xsite.BackupSenderImpl] (HotRodServerWorker-8-1) ISPN000202: Problems backing up data for cache default to site JBOSS: org.infinispan.util.concurrent.TimeoutException: Timed out after 10 seconds waiting for a response from JBOSS (sync, timeout=10000)

[Server:ServerOne] 16:34:35,157 ERROR [org.jgroups.protocols.relay.RELAY2] (HotRodServerWorker-8-2) Udit:ServerOne: no route to JBOSS: dropping message[Server:ServerOne] 16:34:45,158 WARN  [org.infinispan.xsite.BackupSenderImpl] (HotRodServerWorker-8-2) ISPN000202: Problems backing up data for cache default to site JBOSS: org.infinispan.util.concurrent.TimeoutException: Timed out after 10 seconds waiting for a response from JBOSS (sync, timeout=10000)

[udit-mishra-5113a21a]

Somebody please help. nadirx could you? please?

[pruivo]

Hi Udit,

It is a wrong configuration for the network. From the previous messages, I assume both sites (jboss and udit) and in the same network.

If that is the case, each of them must have different multicast address/ports otherwise they wouldn't see each other as a separate site.

The configuration parameters are located in the domain.xml file, in the end: <socket-binding name="jgroups-mping"...> and <socket-binding name="jgroups-udp"...>. You need to change in one of yours sites the multicast-address or the multicast-port.

Cheers,

Pedro

[n.dobryukha]

Hi Pedro,

I'm also interesting in this case. I tried to reproduce Udit's config with replaced parameters "jgroups-mping" and "jgroups-udp" from 234.99.54.14 to 234.99.54.15 in one of sites. It didn't help. I've got the same error:

[Server:server-two] 00:41:06,234 ERROR [org.jgroups.protocols.relay.RELAY2] (HotRodServerWorker-9-1) master:server-two: no route to BACKUP: dropping message
[Server:server-two] 00:41:16,237 WARN  [org.infinispan.xsite.BackupSenderImpl] (HotRodServerWorker-9-1) ISPN000202: Problems backing up data for cache libraryCache to site BACKUP: org.infinispan.util.concurrent.TimeoutException: Timed out after 10 seconds waiting for a response from BACKUP (sync, timeout=10000)