Infinispan rehash problem

[liliyancn]

The Infinispan rehash problem is when it comes to rehash (after node added or deleted), in this period, I can't get whole data back from the cluster.

I start 4 servers, make them as a cluster, then inject 10000 objects from node One. It costs 45843ms to inject those objects in my environment, and it costs 6907ms to loop back all those objects, till now every thing is OK.

After I kill one node (it's not node One), the node One server outputs:

13:50:03,769 INFO  [JGroupsTransport] Received new cluster view: [bnijms-60505|4] [bnijms-60505, iptv-97-17615, iptv-100-39927]
13:50:03,773 INFO  [DistributionManagerImpl] Detected a view change.  Member list changed from [bnijms-60505, iptv-srm1-3188, iptv-97-17615, iptv-100-39927] to [bnijms-60505, iptv-97-17615, iptv-100-39927]
13:50:03,773 INFO  [DistributionManagerImpl] This is a LEAVE event!  Node iptv-srm1-3188 has just left
13:50:03,774 INFO  [DistributionManagerImpl] I bnijms-60505 am participating in rehash

This is very obvious that the cluster view has changed and node One named bnijms-60505 is participating in rehash.

Now I loop all objects from cache region, can only retrieves back 2479 objects from 10000 objects in the cluster. And I continue trying, after 2 minutes and 21 seconds, I can loop back 8862 objects. Finally, after 2 minutes and 37 seconds, I can loop back all 10000 objects.

The loop objects code likes this:

        for (int i = from ; i <= to; i++) {
            String key = getKey(i);
           
            BigObject obj = (BigObject)localCache.get(key);
            if(obj != null) {
                count++;
            }
        }

So, I don't know what's the problem of getting object when rehash.

[vblagojevic]

Richard,

Can you attach your configuration file and bigger snippet of this loop code?

Regards,

Vladimir

[liliyancn]

Infinispan Configuration :

<?xml version="1.0" encoding="UTF-8"?>
<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xmlns="urn:infinispan:config:4.0">
  <global>
    <!--
      use chn-cluster for changbai, ana-cluster for annapurna
    -->
    <transport
      transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport"
      clusterName="iptv-cluster">
      <!--
        use udp.xml for now
      -->
      <properties>
        <property name="configurationFile" value="stacks/udp.xml" />
      </properties>
    </transport>
  </global>

  <default>
    <locking isolationLevel="READ_COMMITTED" lockAcquisitionTimeout="30000" useLockStriping="false"/>
    <lazyDeserialization enabled="true"/>
  </default>

  <namedCache name="cache_nojta_sync">
      <clustering mode="d">
          <l1 enabled="false"/>
          <hash numOwners="3"/>
          <sync/>
      </clustering>
  </namedCache>

</infinispan>

[liliyancn]

Inject objects into cache code :

        public String getKey(boolean isSmallObject, String prefix, int number) {

            String key = "";

            if (isSmallObject) {

                key = "S<" + number + "|" + prefix + "|" +  (number*3) + "|" + (number*7) + ">";

            } else {

                key = "B[" + number + "|" + prefix + "|" +  (number*3) + "|" + (number*7) + "]";

            }

            return key;

        }

        Context context = new InitialContext();

        DefaultCacheManager cacheManager = (DefaultCacheManager)context.lookup("java:CacheManager");

        Cache cache = cacheManager.getCache("cache_nojta_sync");

        for(int i = startNum; i <= endNum; i++) {

            SmallObject obj = new SmallObject();

            obj.setName(getKey(isSmallObject, prefix, i));

            obj.setValue("ABC");

            cache.put(obj.getName(), obj);

        }

Loop back objects code :

        Context context = new InitialContext();

        DefaultCacheManager cacheManager = (DefaultCacheManager)context.lookup("java:CacheManager");

        Cache cache = cacheManager.getCache("cache_nojta_sync");

        int count = 0;

        for (int i = from ; i <= to; i++) {

            String key = getKey(isSmallObject, prefix, i);

            SmallObject obj = (SmallObject)cache.get(key);

            if(obj != null) {

                count++;

            }

        }

        return count;

[vblagojevic]

Richard,

I do not understand how you "And I continue trying, after 2 minutes and 21 seconds, I can loop back 8862 objects." How do you "continue trying" ? You can send code example directly to me and I'll have a look.

Regards,

Vladimir

[mircea.markus]

This is related to https://jira.jboss.org/browse/ISPN-654 - I've tried to reproduce this a lot, but I was looking at the wrong thing!

Please confirm that this would be the expected behaviour:

- if a node craches and you read all the keys you added to the cluster immediately after, from another node then either

a) all the values are returned successfully or b) a TimeoutException is thrown at some point when reading if the given key was not transfered

What should never happen is to return null from a key that is in cache - that is incorrect.

[vblagojevic]

I looked around the code and I believe that this issue could have been potentially fixed in 4.2 Alpha3. Why don't you give it a try Richard?

[liliyancn]

"continue trying" means "put objects into cache" and "loop back objects from cache" are put into two webservcies respectively. So, I actually use SoapUI to PUT objects once and LOOP them repeatly.

As your suggestion, I will test this issue by 4.2 Alpha3, wish have a good result.

[mircea.markus]

This is now fixed: https://jira.jboss.org/browse/ISPN-654