12 Replies Latest reply on Mar 18, 2009 11:25 AM by lovelyliatroim

    ExpirationPolicy & ClusteredCacheLoader

    lovelyliatroim

      Hi Guys,
      Im am using the clustered cache loader and in one of my regions i use the ExpirationPolicy.

      Im noticing something strange when i started playing with it, the eviction policy does not seem to work for data that is gravitated from a different node.

      Lets say i have a region configured called "a/b" with a max set of a 1000.

      Now when i add to the cache i place a path with values like so "a/b/c/d/e", now e is the node that i place the data item and also place the expiration time. When you do this you see the algorithm complains and you see logs like so

      14:03:51,921 [EvictionTimer-1] WARN eviction.ExpirationAlgorithm - Unable to remove nodes to reduce region size below 10000. Set expiration for nodes in this region
      


      Thats because I assume the nodes c and d dont have the expiration policy value set. (Thats fine).

      Now when I start 2 nodes, i ask node1 for "a/b/c/d/e", it doesnt have it so retrieves from the data source, i then add it to the cache in node1. I then start to see the above logs which is fine.


      Now i ask for the same data item from node2, the data is gravitated over via the clusteredcache loader and i return the value. Now i dont see any logs like above that tells me the eviction policy is started on node2 for that region. When the time has elapsed for the data item to be expired, it will expire on node1 but it does not expire on node2. When i add a different data item to the same region in node2 lets say "a/b/c/d/e2", i start seeing the logs, after the expiration time "e2" gets evicted but "e" still exists.

      Havent looked under the bonnet yet but it looks to me that data that is gravitated over via the clustered cache loader is not getting registered with the ExpirationPolicy algorithm, it simply thinks there is no data in this region to evict hence it doesnt start and when there is other data items in the region, it will evict them but the gravitated data item is still not getting cleaned up!!

      Using the 2.2GA version.

      Any insight??

      Thanks,
      LL





        • 1. Re: ExpirationPolicy & ClusteredCacheLoader
          lovelyliatroim

          Difference between adding locally and adding to the cache via the cache loader is that the addEvictionEntry in the ExpirationAlgorithm class is never called. This I assume is why it will never get evicted because it never gets registered.

          • 2. Re: ExpirationPolicy & ClusteredCacheLoader
            manik

             

            "lovelyliatroim" wrote:
            Difference between adding locally and adding to the cache via the cache loader is that the addEvictionEntry in the ExpirationAlgorithm class is never called. This I assume is why it will never get evicted because it never gets registered.


            Yes, this is possibly why.

            Specifically, do you use the DataGravitationInterceptor (used with buddy replication) or the ClusteredCacheLoader (not used with BR, but simply used as a cache loader impl) ?



            • 3. Re: ExpirationPolicy & ClusteredCacheLoader
              lovelyliatroim

               


              Specifically, do you use the DataGravitationInterceptor (used with buddy replication) or the ClusteredCacheLoader (not used with BR, but simply used as a cache loader impl) ?


              I use the ClusteredCacheLoader simply as a cache loader impl

              Configured like so

              <attribute name="CacheLoaderConfig" replace="false">
               <config>
               <cacheloader>
               <class>org.jboss.cache.loader.ClusteredCacheLoader</class>
               <properties>
               timeout=1500
               </properties>
               </cacheloader>
              
               </config>
               </attribute>
              


              • 4. Re: ExpirationPolicy & ClusteredCacheLoader
                manik

                Do you see this with any other cache loaders, e.g., a file cache loader? Try using a FCL, create a node, restart the cache, and access the node (to load it from the FCL) and see if it expires. This will help pinpoint the issue.

                • 5. Re: ExpirationPolicy & ClusteredCacheLoader
                  lovelyliatroim

                  Just another thing to note as well, my cache is set in REP_ASYNC mode but before i put anything into the cache i tell it that this is a local value.

                  treeCache.getInvocationContext().getOptionOverrides().setCacheModeLocal(true);
                  treeCache.put(nodePath,map);
                  


                  So this makes make my cache a LOCAL cache but configured with a ClusteredCacheLoader(We had discussed before ages ago).



                  Do you see this with any other cache loaders, e.g., a file cache loader? Try using a FCL, create a node, restart the cache, and access the node (to load it from the FCL) and see if it expires. This will help pinpoint the issue.


                  Will try and have a look at it today or tomorrow. Will post when i look at it again.

                  • 6. Re: ExpirationPolicy & ClusteredCacheLoader
                    lovelyliatroim

                    Sorry for the delay only getting back to this now.

                    Here is the a test case that i wrote

                     @Test public void testFileLoader() throws InterruptedException{
                     String persistentCache = CMDSJUnitUtil.getPersistentCache();
                     CacheFactory factory = new DefaultCacheFactory();
                     Cache cache = factory.createCache(persistentCache);
                     cache.create();
                     cache.start();
                    
                     Map<String, String> testData = new HashMap<String,String>();
                     testData.put("FirstName", "James");
                     testData.put("Surname", "Bond");
                     long expirationTime = 15000;
                     Fqn path = Fqn.fromString("/testData/bond/james");
                     cache.put(path, testData);
                     long future = System.currentTimeMillis() + expirationTime;
                     cache.put(path,ExpirationConfiguration.EXPIRATION_KEY, future);
                    
                     Thread.sleep(expirationTime + 5000);
                     Map data = cache.getData(path);
                     assertNull("Node should have been evicted "+data,data);
                    
                     //Re-add the node to test for restart case
                     cache.put(path, testData);
                     future = System.currentTimeMillis() + expirationTime;
                     cache.put(path,ExpirationConfiguration.EXPIRATION_KEY, future);
                    
                    
                     System.out.println("About to stop cache");
                     cache.stop();
                     cache.destroy();
                    
                     System.out.println("About to start cache");
                     cache.create();
                     cache.start();
                    
                     data = cache.getData(path);
                     System.out.println("Data = "+data);
                    
                     Thread.sleep(expirationTime + 5000);
                    
                     data = cache.getData(path);
                     assertNull("Node should have been evicted",data);
                     }
                    
                    


                    This interesting enough actually fails at the very first assert. The cache is configured to use the berkeley DB cache loader. Config is like so

                    <server>
                    
                     <!-- ==================================================================== -->
                     <!-- PERSISTENT CACHE CONFIG -->
                     <!-- ==================================================================== -->
                    
                     <mbean code="org.jboss.cache.jmx.CacheJmxWrapper"
                     name="jboss.cache:service=CMDSPersistentCache">
                    
                     <depends>jboss:service=Naming</depends>
                     <depends>jboss:service=TransactionManager</depends>
                    
                     <!--
                     Configure the TransactionManager
                    
                     <attribute name="TransactionManagerLookupClass">org.jboss.cache.transaction.GenericTransactionManagerLookup
                     </attribute>
                    -->
                    
                    
                    
                     <!--
                     Node locking level : SERIALIZABLE
                     REPEATABLE_READ (default)
                     READ_COMMITTED
                     READ_UNCOMMITTED
                     NONE
                     -->
                     <attribute name="IsolationLevel">NONE</attribute>
                    
                     <!--
                     Valid modes are LOCAL
                     REPL_ASYNC
                     REPL_SYNC
                     INVALIDATION_ASYNC
                     INVALIDATION_SYNC
                     -->
                     <attribute name="CacheMode">LOCAL</attribute>
                    
                     <!-- Name of cluster. Needs to be the same for all clusters, in order
                     to find each other
                     -->
                     <attribute name="ClusterName">Cluster</attribute>
                    
                     <!-- JGroups protocol stack properties NOT NEEDED since CacheMode is LOCAL -->
                    
                     <!--
                     The max amount of time (in milliseconds) we wait until the
                     state (ie. the contents of the cache) are retrieved from
                     existing members in a clustered environment
                     -->
                     <attribute name="StateRetrievalTimeout">20000</attribute>
                    
                     <!--
                     Number of milliseconds to wait until all responses for a
                     synchronous call have been received.
                     -->
                     <attribute name="SyncReplTimeout">20000</attribute>
                    
                     <attribute name="FetchInMemoryState">false</attribute>
                    
                    
                    
                     <!-- Max number of milliseconds to wait for a lock acquisition -->
                     <attribute name="LockAcquisitionTimeout">15000</attribute>
                    
                     <!-- Specific eviction policy configurations. -->
                     <attribute name="EvictionPolicyConfig">
                     <config>
                     <attribute name="wakeUpIntervalSeconds">2</attribute>
                     <attribute name="eventQueueSize">200000</attribute>
                     <attribute name="policyClass">org.jboss.cache.eviction.LRUPolicy</attribute>
                     <region name="/_default_">
                     <attribute name="maxNodes">100</attribute>
                     <attribute name="maxAgeSeconds">1</attribute>
                     <attribute name="timeToLiveSeconds">1</attribute>
                     </region>
                    
                     <region name="/testData" policyClass="org.jboss.cache.eviction.ExpirationPolicy" >
                     <attribute name="maxNodes">10</attribute>
                     </region>
                    
                     </config>
                     </attribute>
                    
                    
                     <attribute name="CacheLoaderConfig" replace="false">
                     <config>
                     <shared>false</shared>
                     <async>true</async>
                    
                     <cacheloader>
                     <class>org.jboss.cache.loader.bdbje.BdbjeCacheLoader</class>
                     <properties>
                     location=C:/flatfileDB/node8080
                     </properties>
                     </cacheloader>
                    
                     </config>
                     </attribute>
                    
                     </mbean>
                    </server>
                    
                    


                    Remove the cacheloader config from the cache configuration and everything runs as you would expect. Using the 2.2.x version of JBC.

                    Cheers,
                    LL



                    • 7. Re: ExpirationPolicy & ClusteredCacheLoader
                      manik

                      Yes, since evicting will only remove stuff from memory not from a cache loader.

                      If you want to remove stuff from the loader as well, I would recommend upgrading to JBC 3 and using a RemoveOnEvictActionPolicy (see sample configs that ship with JBC 3)

                      • 8. Re: ExpirationPolicy & ClusteredCacheLoader
                        lovelyliatroim

                         


                        Yes, since evicting will only remove stuff from memory not from a cache loader.

                        Ok thats fine for a file loader but when you use a clustered cache loader, if data gets gravitated over and is not registered for eviction which is my problem, what will remove these data items from memory??

                        As far as I can see, they will sit there forever even though their expiration time is up.

                        • 9. Re: ExpirationPolicy & ClusteredCacheLoader
                          manik

                          Is the expiry time set on all cache instances in the cluster?

                          • 10. Re: ExpirationPolicy & ClusteredCacheLoader
                            lovelyliatroim

                             


                            Is the expiry time set on all cache instances in the cluster?

                            Yes......both nodes use the same configuration file. See my very first post on the details.

                            • 11. Re: ExpirationPolicy & ClusteredCacheLoader
                              manik

                              More than the cfg file, is the expiry information set on the node on all cache instances?

                              • 12. Re: ExpirationPolicy & ClusteredCacheLoader
                                lovelyliatroim

                                Yes expiration key and value is set on the node. See my 2nd comment on this post as to why i think it doesnt get evicted from the cache. Expiration key and data item exist on the gravitated node but it looks like the gravitated node never gets registered thus never getting evicted.