3 Replies Latest reply on Feb 18, 2014 9:17 AM by pruivo

    Two Infinispan clusters formed instead of one, inconsistent data

    tromper11

      Hi,

       

      There are 6 nodes for a distributed cache using 3 physical boxes:

      - nodes1-4 are on physical_boxA

      - node5    is  on physical_boxB

      - node6    is  on physical_boxC

       

      All data is persisted by a shared MySQL v5 server.

       

      Problems:

       

      1) After startup we see (using JMX) two clusters instead of one --

      • clusterA size=4 and containing nodes1-4
      • clusterB size=2 containing node5, node6.

       

      2) Even if we have 2 clusters (instead of one), put operations on clusterA are somehow seen by clusterB, but after remove on clusterA, clusterB still contains the old data;

       

      We're using the following config:

       

      <?xml version="1.0" encoding="UTF-8"?>

      <infinispan        xmlns="urn:infinispan:config:5.3">

       

          <global>

              <globalJmxStatistics enabled="true" allowDuplicateDomains="true"/>

              <transport nodeName="Node" clusterName="STORE_TEST">

              </transport>

          </global>

       

          <namedCache name="LOCKS">

              <clustering mode="distribution">

                  <sync/>

              </clustering>

              <eviction maxEntries="50000" strategy="LRU"/>

          </namedCache>

       

          <namedCache name="DATA">

              <clustering mode="distribution">

                  <sync/>

              </clustering>

       

              <eviction maxEntries="999999" strategy="LRU"/>

              <loaders passivation="false" shared="true" preload="false">

                  <loader class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore" fetchPersistentState="true" ignoreModifications="false" purgeOnStartup="false">

                      <properties>

                          <property name="stringsTableNamePrefix" value="data_res"/>

                          <property name="idColumnName" value="ind"/>

                          <property name="idColumnType" value="VARCHAR(255)"/>

       

                          <property name="dataColumnName" value="data"/>

                          <property name="dataColumnType" value="BLOB"/>

       

                          <property name="timestampColumnName" value="ver_time_stamp"/>

                          <property name="timestampColumnType" value="BIGINT"/>

       

                          <property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>

                          <property name="connectionUrl" value="jdbc:mysql://................."/>

                          <property name="driverClass" value="com.mysql.jdbc.Driver"/>

       

                          <property name="dropTableOnExit" value="false"/>

                          <property name="createTableOnStart" value="false"/>

                          <property name="databaseType" value="MYSQL"/>

                      </properties>

                  </loader>

              </loaders>

          </namedCache>

       

       

          <namedCache name="CON_DATA">

              <clustering mode="distribution">

                  <sync/>

              </clustering>

       

              <eviction maxEntries="999999" strategy="LRU"/>

              <loaders passivation="false" shared="true" preload="false">

                  <loader class="org.infinispan.loaders.jdbc.stringbased.JdbcStringBasedCacheStore" fetchPersistentState="true" ignoreModifications="false" purgeOnStartup="false">

                      <properties>

                          <property name="stringsTableNamePrefix" value="con_data"/>

                          <property name="idColumnName" value="ind"/>

                          <property name="idColumnType" value="VARCHAR(255)"/>

       

                          <property name="dataColumnName" value="data"/>

                          <property name="dataColumnType" value="BLOB"/>

       

                          <property name="timestampColumnName" value="ver_time_stamp"/>

                          <property name="timestampColumnType" value="BIGINT"/>

       

                          <property name="connectionFactoryClass" value="org.infinispan.loaders.jdbc.connectionfactory.PooledConnectionFactory"/>

                          <property name="connectionUrl" value="jdbc:mysql:/.............."/>

                          <property name="driverClass" value="com.mysql.jdbc.Driver"/>

       

                          <property name="dropTableOnExit" value="false"/>

                          <property name="createTableOnStart" value="false"/>

                          <property name="databaseType" value="MYSQL"/>

                      </properties>

                  </loader>

              </loaders>

       

          </namedCache>

      </infinispan>

       

       

      This drives us crazy, any help'd be greatly appreciated.

       

      Thanks,

        • 1. Re: Two Infinispan clusters formed instead of one, inconsistent data
          pruivo

          Hi tromper11,

           

          1)

          it looks a network problem to me. please double check if the instance in physical_boxA are binding to the correct network interface. Also, JGroups uses ip multicast to perform the discover a nodes, so if you don't have ip multicast available in your cluster, you need to provide a custom JGroups configuration file. If you need help with this let me know.

           

          2)

          a put operation is seen by the 2 clusters because they are pointing to the same database. In more detail: in clusterA, a put operation is persistence to the database; in clusterB, you perform a get operation. however, if the key is not found in memory, it will try to load it from the database. the load succeeds because it was written by clusterA.

          On other hand, the remove operation does not remove the data from clusterA because it keeps the data in memory until the eviction is triggered (after the threshold or manually). Since they are disjoint clusters, clusterB will never receive the remove operation to remove the data from memory (hmmm... am I clear?)

           

          Let me know if I wasn't clear.

           

          Cheers,

          Pedro

          • 2. Re: Two Infinispan clusters formed instead of one, inconsistent data
            tromper11

            Hi,

             

            Your explanation is perfectly clear; we'll check network.

             

            I thought that remove actually removes item regardless of eviction/expiration policy, are you sure the item remains in the cluster?

             

            Do you think that  maxEntries=999999 is fine and not too much?

             

            Thanks,

            • 3. Re: Re: Two Infinispan clusters formed instead of one, inconsistent data
              pruivo

              Hi,

              I thought that remove actually removes item regardless of eviction/expiration policy, are you sure the item remains in the cluster?

              yes, the remove operation removes the data regardless of eviction/expiration from memory and database. What I meant is if you have two separated clusters, A and B, if you remove a key in clusterA, B still keeps it.

               

              Do you think that  maxEntries=999999 is fine and not too much?

              It totally depends of your application, but the simplified formula would be heap size (i.e. available memory) > maxEntries * average entry size

               

              Cheers,

              Pedro