4 Replies Latest reply on Sep 30, 2015 2:23 AM by nadirx

    How to configure cross site replication with Infinispan 7.2.1

    mgeorg
    mgeorg Sep 14, 2015 6:52 AM

      Hello,

       

      I'm trying to test cross side replication between 2 clusters of three nodes each with Infinispan 7.2.1 and the underlying JGroups communication library,

      which is bundled with Infinispan 7.2.1 .

      The configuration is as follows:

          CSR-1 = first site / cluster with three nodes running on three different computers (c0, c1, c2).

                  The Infinispan configuration is in file 'infinispan-cache-CSR1.xml', the JGroups configuration in file

                  'jgroups-udp-CSR1.xml'. Every node in the cluster reads the same configuration files. The multicast address

                  and port for cluster communication is '228.6.7.8:46655' for every node. The attributes for the RELAY2 protocoll

                  are 'site="CSR-1" config=".../jgroups-relay2.xml" relay_multicasts="true" '

                 

          CSR-2 = second site / cluster with three nodes running on three different computers (c0, c1, c2).

                  The Infinispan configuration is in file 'infinispan-cache-CSR2.xml', the JGroups configuration in file

                  'jgroups-udp-CSR2.xml'. Every node in the cluster reads the same configuration files. The multicast address

                  and port for cluster communication is '229.6.7.8:46660' for every node. The attributes for the RELAY2 protocoll

                  are 'site="CSR-2" config=".../jgroups-relay2.xml" relay_multicasts="true" '

         

          For both sides, the 'jgroups-relay2.xml' file defines a global cluster for the inter site communication, the configuration

          is in file 'jgroups-relay2-tcp.xml'.

       

      At the moment, the cache in site 'CSR-2'('KeyValueProviderCSR2') is the backup cache for the cache in site 'CSR-1' ('KeyValueProviderCSR1').

      This is configured in 'infinispan-cache-CSR1.xml' resp. 'infinispan-cache-CSR2.xml' . When running both clusters and sending data to node

      c0 of site CSR-1, I got the error message

          "no route to CSR-2:drop the message"

      for every key / value pair which has to be backuped to site CSR-2.

      What is wrong with the configuration. I have tried to follow the documentation for cross site replication, but it doesn't work.

      The configuration files are attached.

       

      Thanks,

      Michael

      • 348 Views
      • Categories: Community Contributed Articles
      • Tags:
        • 1. Re: How to configure cross site replication with Infinispan 7.2.1
          mgeorg
          mgeorg Sep 22, 2015 4:42 AM (in response to mgeorg)

          Hi,

           

          no one in the community who has used cross site replication so far ?

          I have tried the "Draw" example described in the JGroups documentation (http://www.jgroups.org/manual-3.x/html/user-advanced.html#Relay2Advanced),

          but configuring straight like the doc says doesn't work also. I have attached the files.

           

          Greets,

          Michael

            • Actions
          • 2. Re: How to configure cross site replication with Infinispan 7.2.1
            nadirx
            nadirx Sep 22, 2015 10:30 AM (in response to mgeorg)

            Sorry Michael,

            this slipped under the radar. Can you please post some DEBUG logs of the nodes starting ?

              • Actions
            • 3. Re: How to configure cross site replication with Infinispan 7.2.1
              mgeorg
              mgeorg Sep 23, 2015 4:47 AM (in response to nadirx)

              Hi Tristan,

               

              I have attached some logs from the start phase of each node. You will find the configuration used in the first post.

               

              Thanks,

              Michael

                • Actions
              • 4. Re: How to configure cross site replication with Infinispan 7.2.1
                nadirx
                nadirx Sep 30, 2015 2:23 AM (in response to mgeorg)

                Hi Michael,

                 

                sorry this took so long. First off: the infinispan configuration for both sites looks good, as well as the local transport for each site which is using UDP. It is evident from the logs that the individual site clusters are forming correctly:

                 

                CSR1:

                10:35:25 D DBG0000 Topology change detected! EventImpl{type=TOPOLOGY_CHANGED, pre=false, cache=Cache 'KeyValueProviderCSR1'@c0-10836, consistentHashAtStart=DefaultConsistentHash{ns=60, owners = (2)[c2-8259: 30+30, c1-12385: 30+30]}, consistentHashAtEnd=DefaultConsistentHash{ns=60, owners = (3)[c2-8259: 20+20, c1-12385: 20+20, c0-10836: 20+20]}, unionConsistentHash=null, newTopologyId=4}

                 

                CSR2:

                10:35:33 D DBG0000 Updating cluster-wide stable topology for cache KeyValueProviderCSR2, topology = CacheTopology{id=4, rebalanceId=2, currentCH=DefaultConsistentHash{ns=60, owners = (3)[c0-43509: 20+20, c2-11026: 20+20, c1-21023: 20+20]}, pendingCH=null, unionCH=null, actualMembers=[c0-43509, c2-11026, c1-21023]}

                 

                The bridge however is not working correctly. From the hostnames involved it looks like you're attempting to create the two sites on the same three boxes, but you are not using separate network interfaces for the TCP transport which means that both sites are attempting to bind port 7800 on the same address on the same boxes which is not possible.

                If you want to keep the physical layout, you will need to create an additional virtual network interface and make it part of the same network. For example, supposing your c1, c2, c3 hosts are 192.168.0.[1-3] create interfaces on each of those boxes assigned to 192.168.0.[4-6] and assign them to host names c4, c5 and c6 respectively.

                In that way c1, c2, c3 will bind to port 7800 on the first interface and c4, c5, c6 will bind to port 7800 on the second interface. Obviously in the respective TCPPING configuration of each site you will need to point to the other side hosts/ips.

                 

                I hope this helps

                 

                Tristan

                  • Actions