8 Replies Latest reply on Nov 10, 2010 12:58 PM by Mircea Markus

    Infinispan rehash problem

    richard lee Newbie

      The Infinispan rehash problem is when it comes to rehash (after node added or deleted), in this period, I can't get whole data back from the cluster.


      I start 4 servers, make them as a cluster, then inject 10000 objects from node One. It costs 45843ms to inject those objects in my environment, and it costs 6907ms to loop back all those objects, till now every thing is OK.


      After I kill one node (it's not node One), the node One server outputs:


      13:50:03,769 INFO  [JGroupsTransport] Received new cluster view: [bnijms-60505|4] [bnijms-60505, iptv-97-17615, iptv-100-39927]
      13:50:03,773 INFO  [DistributionManagerImpl] Detected a view change.  Member list changed from [bnijms-60505, iptv-srm1-3188, iptv-97-17615, iptv-100-39927] to [bnijms-60505, iptv-97-17615, iptv-100-39927]
      13:50:03,773 INFO  [DistributionManagerImpl] This is a LEAVE event!  Node iptv-srm1-3188 has just left
      13:50:03,774 INFO  [DistributionManagerImpl] I bnijms-60505 am participating in rehash


      This is very obvious that the cluster view has changed and node One named bnijms-60505 is participating in rehash.


      Now I loop all objects from cache region, can only retrieves back 2479 objects from 10000 objects in the cluster. And I continue trying, after 2 minutes and 21 seconds, I can loop back 8862 objects. Finally, after 2 minutes and 37 seconds, I can loop back all 10000 objects.

       

      The loop objects code likes this:

       

              for (int i = from ; i <= to; i++) {
                  String key = getKey(i);
                 
                  BigObject obj = (BigObject)localCache.get(key);
                  if(obj != null) {
                      count++;
                  }
              }


      So, I don't know what's the problem of getting object when rehash.

        • 1. Re: Infinispan rehash problem
          Vladimir Blagojevic Master

          Richard,

           

          Can you attach your configuration file and bigger snippet of this loop code?

           

          Regards,

          Vladimir

          • 2. Re: Infinispan rehash problem
            richard lee Newbie

            Infinispan Configuration :

             

            <?xml version="1.0" encoding="UTF-8"?>
            <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xmlns="urn:infinispan:config:4.0">
              <global>
                <!--
                  use chn-cluster for changbai, ana-cluster for annapurna
                -->
                <transport
                  transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport"
                  clusterName="iptv-cluster">
                  <!--
                    use udp.xml for now
                  -->
                  <properties>
                    <property name="configurationFile" value="stacks/udp.xml" />
                  </properties>
                </transport>
              </global>

             

              <default>
                <locking isolationLevel="READ_COMMITTED" lockAcquisitionTimeout="30000" useLockStriping="false"/>
                <lazyDeserialization enabled="true"/>
              </default>

             

              <namedCache name="cache_nojta_sync">
                  <clustering mode="d">
                      <l1 enabled="false"/>
                      <hash numOwners="3"/>
                      <sync/>
                  </clustering>
              </namedCache>

            </infinispan>

            • 3. Re: Infinispan rehash problem
              richard lee Newbie

              Inject objects into cache code :

               

                      public String getKey(boolean isSmallObject, String prefix, int number) {

                          String key = "";

                          if (isSmallObject) {

                              key = "S<" + number + "|" + prefix + "|" +  (number*3) + "|" + (number*7) + ">";

                          } else {

                              key = "B[" + number + "|" + prefix + "|" +  (number*3) + "|" + (number*7) + "]";

                          }

               

                          return key;

                      }

               

                      Context context = new InitialContext();

                      DefaultCacheManager cacheManager = (DefaultCacheManager)context.lookup("java:CacheManager");

                      Cache cache = cacheManager.getCache("cache_nojta_sync");

               

                      for(int i = startNum; i <= endNum; i++) {

                          SmallObject obj = new SmallObject();

                          obj.setName(getKey(isSmallObject, prefix, i));

                          obj.setValue("ABC");

               

                          cache.put(obj.getName(), obj);

                      }

               

              Loop back objects code :

               

                      Context context = new InitialContext();

                      DefaultCacheManager cacheManager = (DefaultCacheManager)context.lookup("java:CacheManager");

                      Cache cache = cacheManager.getCache("cache_nojta_sync");

               

                      int count = 0;

                      for (int i = from ; i <= to; i++) {

                          String key = getKey(isSmallObject, prefix, i);

               

                          SmallObject obj = (SmallObject)cache.get(key);

                          if(obj != null) {

                              count++;

                          }

                      }

               

                      return count;

              • 4. Re: Infinispan rehash problem
                Vladimir Blagojevic Master

                Richard,

                 

                I do not understand how you "And I continue trying, after 2 minutes and 21 seconds, I can loop back 8862 objects." How do you "continue trying" ? You can send code example directly to me and I'll have a look.

                 

                Regards,

                Vladimir

                • 5. Re: Infinispan rehash problem
                  Mircea Markus Master

                  This is related to https://jira.jboss.org/browse/ISPN-654 - I've tried to reproduce this a lot, but I was looking at the wrong thing!

                  Please confirm that this would be the expected behaviour:

                  - if a node craches and you read all the keys you added to the cluster immediately after, from another node then either

                  a) all the values are returned successfully or b) a TimeoutException is thrown at some point when reading if the given key was not transfered

                  What should never happen is to return null from a key that is in cache - that is incorrect.

                  • 6. Re: Infinispan rehash problem
                    Vladimir Blagojevic Master

                    I looked around the code and I believe that this issue could have been potentially fixed in 4.2 Alpha3. Why don't you give it a try Richard?

                    • 7. Re: Infinispan rehash problem
                      richard lee Newbie

                      "continue trying" means "put objects into cache" and "loop back objects from cache" are put into two webservcies respectively. So, I actually use SoapUI to PUT objects once and LOOP them repeatly.

                       

                      As your suggestion, I will test this issue by 4.2 Alpha3, wish have a good result.