3 Replies Latest reply on Dec 4, 2009 10:26 AM by Manik Surtani

    How to populate distributed cluster on startup?

    Juha Heljoranta Newbie

      Hi,

      I have an application which needs large amount of key-value pairs cached for performance reasons. However, there is one catch: all of the data must be preloaded into cache before serving client requests.

      So the challenge is: how to populate distributed cluster with large amount of data? I couldn't find any automated way to do this. The CacheLoaders seem to load everything into every node (e.g. clustering mode="replication").

      The best solution I managed to come up looks something like this:

      Central server:

      Configuration c = new Configuration();
      c.setCacheMode(Configuration.CacheMode.DIST_ASYNC);
      c.setNumOwners(2);
      
      CacheManager manager = new DefaultCacheManager(GlobalConfiguration.getClusteredDefault(), c);
      
      Cache cache = manager.getCache();
      
      // prevent individual nodes from throwing out-of-memory error
      waitUntilEnoughNodesAvailable();
      
      ...
      
      // populate cluster nodes
      while (hasMoreData()) {
       KeyValue kv = getNext();
       cache.put(kv.getKey(), kv.getData()); // insert
       cache.evict(kv.getKey()); // prevent server out-of-memory error
      }
      
      ...
      
      public void handle(Request request) {
       ...
       Key key = request.getKey();
       Value value = cache.get(key); // cache lookup
       ...
      }
      


      Grid nodes:
      Configuration c = new Configuration();
      c.setCacheMode(Configuration.CacheMode.DIST_ASYNC);
      c.setNumOwners(2);
      
      CacheManager manager = new DefaultCacheManager(GlobalConfiguration.getClusteredDefault(), c);
      
      Cache cache = manager.getCache();
      
      // serves cache put/get/remove requests in background
      while (true) {
       Thread.sleep(1000);
      }
      


      Sure, this design has a major flaw: enough nodes drop out and the rest of the nodes will throw out of memory error:) Plus there is a possibility of losing cached data if there are not enough copies in grid when a node(s) drops out.

      Other issue with the cache loaders is that they need to consult disk/database if the cache doesn't contain the key. Am I right?

      So, any thoughts on this?


        • 1. Re: How to populate distributed cluster on startup?
          Manik Surtani Master

          Hi there.

          The preload approach you have works, but a few comments:

          1. cache.evict() is unnecessary if you disable L1 caching
          2. you are connecting to the system using a client/server style approach, even though you are starting a cache as a peer. This could be unnecessary/wasteful. We have a more explicit client/server architecture in the works - ISPN-29. Until then the client could connect to the storage nodes using the REST API?
          3. To prevent OOMs, enable a CacheStore (e.g., FileCacheStore) and set an eviction policy (e.g., FIFO).
          4. Re: cache loaders consulting disk on a miss, yes this is correct.

          On loading data, where do you get this data from? Perhaps you could write a custom cache loader to fetch this data on demand, and attach this custom cache loader to the storage nodes? This would then be loaded lazily...

          1 of 1 people found this helpful
          • 2. Re: How to populate distributed cluster on startup?
            Juha Heljoranta Newbie

            Hmm.. it seems that the grid is loosing cache entries rapidly. Also, the cache performance is horrible.

            Either I am doing something wrong or there is a major bug. Any insight?

            I've tested with CR2 and with the latest snapshot.

            public class GridNode {
             public static void main(String[] args) throws Exception {
             Configuration c = new Configuration();
             c.setL1CacheEnabled(false);
             c.setCacheMode(Configuration.CacheMode.DIST_SYNC);
             // c.setInvocationBatchingEnabled(true);
            
             GlobalConfiguration gc = GlobalConfiguration.getClusteredDefault();
             // gc.setTransportProperties("configurationFile=my-tcp.xml");
             CacheManager manager = new DefaultCacheManager(gc, c);
            
             String name = ManagementFactory.getRuntimeMXBean().getName();
             Cache<String, byte[]> cache = manager.getCache("test");
            
             // serves cache put/get/remove requests in background
             while (true) {
             System.out.println(name + " cache size " + cache.size());
             Thread.sleep(4000);
             }
             }
            }
            


            public class GridClient {
            
             public static void main(String[] args) throws Exception {
             Configuration c = new Configuration();
             c.setL1CacheEnabled(false);
             c.setCacheMode(Configuration.CacheMode.DIST_SYNC);
             c.setEvictionStrategy(EvictionStrategy.FIFO);
             // c.setInvocationBatchingEnabled(true);
            
             GlobalConfiguration gc = GlobalConfiguration.getClusteredDefault();
             // gc.setTransportProperties("configurationFile=my-tcp.xml");
             CacheManager manager = new DefaultCacheManager(gc, c);
            
             Cache<String, byte[]> cache = manager.getCache("test");
            
             // wait until ready
             while (!ComponentStatus.RUNNING.equals(cache.getStatus())
             || manager.getMembers().size() < 3) {
             Thread.sleep(100L);
             }
            
             // reset cache
             cache.clear();
            
             // populate nodes with data
             int i = 0;
             while (true) {
             String key = Integer.toString(i);
            
             // System.out.println("put " + i);
             // cache.startBatch();
             cache.put(key, new byte[64]);
             // cache.endBatch(true);
            
             if (i++ % 3000 == 0) {
             System.out.println("server cache size: " + cache.size()
             + " total items: " + i);
             }
             Thread.sleep(1L);
            
             // verify existing data
             for (int j = 0; j < i; j++) {
             // System.out.println("get " + j);
             if (cache.get(Integer.toString(j)) == null) {
             System.out.println("ERROR: Grid lost an object: " + j);
             Thread.sleep(1000L);
             }
             }
             }
             }
            }
            


            • 3. Re: How to populate distributed cluster on startup?
              Manik Surtani Master

              If you are using eviction, then you should use a cache store as well otherwise you will have data loss. But TBH you shouldn't need eviction since you have disabled L1 caching, unless there is more data than (collective) memory.

              In terms of performance, your validation step (looping through all keys every time you add each key) will result in a lot of remote look-ups! I'm not surprised this is slow! :) How about validate *after* you have added all of your keys?

              1 of 1 people found this helpful