4 Replies Latest reply on Oct 27, 2010 9:55 AM by craigching

    Inconsistent performance when loading a cache

    craigching

      I'm finding that Infinispan's performance is mostly good, but I see "blips" of bad performance when creating lots of objects in a cache.  For instance, if I run the following code:

       

       

      System.out.println("Starting test.");
      long total_begin = System.currentTimeMillis();
      long begin = total_begin;
      cache.startBatch();
      long min = Integer.MAX_VALUE;
      long max = Integer.MIN_VALUE;
      for (int i = 0; i < 1000000; ++i) {
      String username = "User" + i;
      User u = new User(username, (i%2==0), username + " full name", username + "@test.com", "mypassword");
      long each = System.currentTimeMillis();
      cache.put(u.getUsername(), u);
      long current = System.currentTimeMillis() - each;
      if (current > max) {
      max = current;
      }
      if (current < min) {
      min = current;
      }
      if ((i + 1) % 1000 == 0) {
      cache.endBatch(true);
      System.out.println((i + 1) + " users in the cache.  " + (System.currentTimeMillis() - begin) + " ms. (min: " + min + ", max: " + max + ")");
      begin = System.currentTimeMillis();
      min = Integer.MAX_VALUE;
      max = Integer.MIN_VALUE;
      cache.startBatch();
      }
      }
      cache.endBatch(true);
      System.out.println("Total time for test: " + (System.currentTimeMillis() - total_begin) + " ms.");
      System.out.println("Cache size: " + cache.size());
      cache.stop();
      cache_mgr.stop();
      System.out.println("Cache is stopped, exiting.");

                     System.out.println("Starting test.");

                     long total_begin = System.currentTimeMillis();

                     long begin = total_begin;

       

                     cache.startBatch();

                    

                     long min = Integer.MAX_VALUE;

                     long max = Integer.MIN_VALUE;

       

                     for (int i = 0; i < 1000000; ++i) {

                          String username = "User" + i;

                         

                          User u = new User(username, (i%2==0), username + " full name", username + "@test.com", "mypassword");

       

                          long each = System.currentTimeMillis();

                          cache.put(u.getUsername(), u);

                          long current = System.currentTimeMillis() - each;

                         

                          if (current > max) {

                               max = current;

                          }

                         

                          if (current < min) {

                               min = current;

                          }

                         

                          if ((i + 1) % 1000 == 0) {

                               cache.endBatch(true);

                               System.out.println((i + 1) + " users in the cache.  " + (System.currentTimeMillis() - begin) + " ms. (min: " + min + ", max: " + max + ")");

                               begin = System.currentTimeMillis();

                               min = Integer.MAX_VALUE;

                               max = Integer.MIN_VALUE;

                               cache.startBatch();

                          }

                     }

                     cache.endBatch(true);

                     System.out.println("Total time for test: " + (System.currentTimeMillis() - total_begin) + " ms.");

                     System.out.println("Cache size: " + cache.size());

                     cache.stop();

                     cache_mgr.stop();

                     System.out.println("Cache is stopped, exiting.");

      I get output like this:
      821000 users in the cache.  189 ms. (min: 0, max: 2)
      822000 users in the cache.  188 ms. (min: 0, max: 1)
      823000 users in the cache.  265 ms. (min: 0, max: 92)
      824000 users in the cache.  2075028 ms. (min: 0, max: 2071584)
      825000 users in the cache.  1523 ms. (min: 0, max: 43)
      826000 users in the cache.  201 ms. (min: 0, max: 1)
      827000 users in the cache.  210 ms. (min: 0, max: 12)
      828000 users in the cache.  305 ms. (min: 0, max: 74)
      The "blip" I'm talking about is at 824000 users in the cache (there are other "blips" much earlier in the test, this one was just handy).  Just to clarify here too, 2075028 is the time it took to put the whole batch into the cache and max is the maximum time it took to put just one object in the cache.
      Anyone else see these "blips"?  I wouldn't be so concerned about this since loading a cache like this really isn't my use case, but I'm concerned that when I do get one loaded, it might take an inordinate amount of time to put just one object in the cache, and for a webapp, anything beyond a second or two tops is going to get flagged by someone.
        • 1. Re: Inconsistent performance when loading a cache
          sannegrinovero

          maybe enable logging of GC activity, to rule out the obvious..

          • 2. Re: Inconsistent performance when loading a cache
            craigching

            I'll try that, but I suspected that already and hooked up a jconsole at one point and didn't see gc going on.  I specify the heap settings at "-Xms512m -X1536m" but never go beyond 256m (I only keep 1000 entries in memory).  I'll double check that though.

            • 3. Re: Inconsistent performance when loading a cache
              craigching

              Actually, just to make sure I wasn't running into something stupid on my part, I switched from FileCacheStore to JdbmCacheStore and the performance is *much* better all around.  I wondered about that from the beginning, it seems to me that the FileCacheStore is a bit naive in creating tons of little files on the disk.

               

              Now I'm wondering if something like an H2 database on each node might be even better.  Anyone doing such a thing?

               

              Just to give a bit of background here, I was looking to replace our embedded JDBC database with something that had replication. H2 has replication, but it's recovery leaves a lot to be desired.  So I started looking into Infinispan and I like the cache, the replication, and the stores, I would *love* to not have to have a separate database to maintain.  It would really be cool if Infinispan came with a good, high-performance datastore, but developing such a thing probably isn't trivial.

              • 4. Re: Inconsistent performance when loading a cache
                craigching

                And just FYI in case anyone else wants to know, the JDBM cache store is almost twice as fast as H2, so I'd say it's a decent implementation.