7 Replies Latest reply on Jul 26, 2012 11:55 AM by dex80526

    consistent hash of distriubtion with relay behaviour

    dex80526

      Acooring to Jgroups document (http://www.jgroups.org/manual/html/user-advanced.html#RelayAdvanced), the change to a local cluster will be replicated by the RELAY to other cluster.  But, in my testing, the behaviour is different (detailed below). I used gui-demo code from Infinispan distribution. The observed behaviour is indeed like the 2 local cluster formed ONE big cluster.

       

      Here is my configuration.

      1) nyc cluster 1 (2 nodes with distribution and numOwners = 2); zrh cluster 2 (single node cluster, distribution and numbOwners=2)

      2) I use TCP for both local and bridge

       

      After I ran Infinispan demo instance on 3 different nodes to form 2 local clusters and bridge them with RELAY, the GUI shows 2 node from site nyc and one from zrh, but only one Coordinator.

       

      Then, I tried to add data entries from node1 (nyc), the data will be distributed to antoher node, only one (based the hash order). This could be the second node node2 (nyc) or node(zrh), but, the data never apper on all 3 nodes.

       

      This seems to me is the expected behavior of all 3 node form a single cluster with DISTRIBUTION and numOwners=2.

       

       

      Expected behavior of RELAY: the data should appear on all 3 node in this case.

       

      Do I misss somthing here? or it is a bug?

       

       

      Here are the configuration:

       

       

      relay1.xml:

      <infinispan

            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

            xsi:schemaLocation="urn:infinispan:config:5.1 http://www.infinispan.org/schemas/infinispan-config-5.1.xsd"

            xmlns="urn:infinispan:config:5.1">

       

         <global>

            <transport clusterName="c1"

                      machineId="c1-node1"

                      siteId="nyc"

                      rackId="r1" nodeName="hanode1"

            >

               <properties>

                  <property name="configurationFile" value="./jgroups-tcp-relay1.xml" />

               </properties>

            </transport>

            <globalJmxStatistics enabled="true"/>

         </global>

       

         <default>

           <locking

               isolationLevel="READ_COMMITTED"

               lockAcquisitionTimeout="20000"

               writeSkewCheck="false"

               concurrencyLevel="5000"

               useLockStriping="false"

            />

            <deadlockDetection enabled="true" spinDuration="1000"/>

       

            <jmxStatistics enabled="true"/>

        </default>

        <namedCache name="keychain" >

              <clustering mode="distribution">

               <l1 enabled="false" lifespan="10000"/>

               <hash numOwners="2" rehashWait="5000" rehashRpcTimeout="10000" />

             <async/>

            </clustering>  

        </namedCache>

      </infinispan>

       

      jgroups-tcp-relay1.xml:

      ....

         <TCPPING timeout="3000"

              initial_hosts="${jgroups.tcpping.initial_hosts:node1[7800],node2[7800]}"

                  port_range="1"

                  num_initial_members="2"

                  ergonomics="false"

              />

         <MERGE2 max_interval="30000"

                 min_interval="10000"/>

         <FD_SOCK/>

         <FD timeout="3000" max_tries="3"/>

         <VERIFY_SUSPECT timeout="1500"/>

         <pbcast.NAKACK

               use_mcast_xmit="false"

               retransmit_timeout="300,600,1200,2400,4800"

               discard_delivered_msgs="false"/>

         <UNICAST2 timeout="300,600,1200"/>

         <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"

                        max_bytes="400000"/>

         <pbcast.GMS print_local_addr="false" join_timeout="7000" view_bundling="true"/>

         <UFC max_credits="2000000" min_threshold="0.10"/>

         <MFC max_credits="2000000" min_threshold="0.10"/>

         <FRAG2 frag_size="60000"/>

         <RELAY site="nyc" bridge_name="dr_bridge" level="TRACE" bridge_props="./tcp.xml"  />

      </config>

       

      tcp.xml:

      ....

         <TCPPING timeout="3000"

                  initial_hosts="${jgroups.tcpping.initial_hosts:node1[8900], node2[8900], node3[8900]}"

       

                  port_range="1"

                  num_initial_members="3"

                  ergonomics="false"

        />

         <MERGE2 max_interval="30000"

                 min_interval="10000"/>

         <FD_SOCK/>

         <FD timeout="3000" max_tries="3"/>

         <VERIFY_SUSPECT timeout="1500"/>

         <pbcast.NAKACK

               use_mcast_xmit="false"

               retransmit_timeout="300,600,1200,2400,4800"

               discard_delivered_msgs="false"/>

         <UNICAST2 timeout="300,600,1200"/>

         <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"

                        max_bytes="400000"/>

         <pbcast.GMS print_local_addr="false" join_timeout="7000" view_bundling="true"/>

         <UFC max_credits="2000000" min_threshold="0.10"/>

         <MFC max_credits="2000000" min_threshold="0.10"/>

         <FRAG2 frag_size="60000"/>

      </config>

        • 1. Re: consistent hash of distriubtion with relay behaviour
          an1310

          No, this is exactly how it should work.  Remember, RELAY is a bridge protocol.  The view will still be nodes from both sites, and ISPN will treat this as it would any other cluster.  The only difference is at the transport (JGroups) layer and how data gets from site A to site B. 

          • 2. Re: consistent hash of distriubtion with relay behaviour
            dex80526

            thanks for the respose. Could you take a look at my other questions posted on another thread (relay across 2 sites)?

            • 3. Re: consistent hash of distriubtion with relay behaviour
              dex80526

              Erik:  If the consistent hash works as you stated, then the relay does NOT work. 

               

              For example, in my setting, the changes made to the local cluster at NYC (2 nodes), will not relay to remote cluster at ZRH, since the changes will only send to 2 local nodes as we configure numOwner=2.

               

              In addition, this way (consistent hash across global view) will not scale to large clusters.

              • 4. Re: consistent hash of distriubtion with relay behaviour
                an1310

                Hi Dex,

                 

                There's a bit of a disconnect here.  The nodes ISPN chooses is based off the hash code of the key and the portion of the hash space each node (+ any virtual nodes) takes.  So it's entirely possible that the key hashes to your local nodes in NYC.

                 

                I think what you want, in this case, is the Topology-Aware Consistent Hash, which I didn't see in your ISPN configuration.  See http://docs.jboss.org/infinispan/5.1/configdocs/ and look under "hash".  Then you can tune the site IDs in your NYC and ZRH configurations.  See if that helps out.

                • 5. Re: consistent hash of distriubtion with relay behaviour
                  dex80526

                  Erik: thanks for the info. Yes, the TACH makes different.

                   

                  Is there any performance impact on the local cluster if the network connectivity between 2 cluster is crappy?    In other words,will the local cluster's operation is going to be dragging down by the SLOW replicationr to the remont  cluster?

                   

                  I am trying to use RELAY to make another cluster as a backup/DR site.

                   

                  Do you know any one is doing this in production?

                  • 6. Re: consistent hash of distriubtion with relay behaviour
                    an1310

                    Oh, absolutely, especially if you use transactions and sync commit.  I don't feel that a single cluster spanning data centers is an acceptable solution.  Fortunately, the 5.2 release will have a cross-data center implementation.  Read about it here -- https://community.jboss.org/wiki/DesignForCrossSiteReplication

                    • 7. Re: consistent hash of distriubtion with relay behaviour
                      dex80526

                      I am talking 2 clusters across data centers. That is the exact reason which I am looking for RELAY to bridge 2 clusters, instead of using one cluster spaning 2 data centers.  Are you saying that REALY in 5.1.x release does not support bridging 2 clusters across 2 data centers. 

                       

                      My experience with RELAY in 5.1.5.Final is that the relay (data replication across 2 clusters) seems to work. But, I see a lot limitations, for example, I can not configure RELAY only for certain named caches (not for others) with a cache manager.

                       

                      Performace imapct to the local cluster is a major concern.  I do not see if the messages to be replicated to other cluster are queued or not, since there is no way to configure how a local cluster to handle the messages if the network, for example, has long latency, etc.

                       

                      thanks.