10 Replies Latest reply on Nov 28, 2012 2:53 PM by hsn10

    Very slow FileCacheStore

    hsn10

      There must be some error in inf 5.1.6, i did my best to avoid fsynces and writing is still very slow. I guess it still does fsync after each write. 1000 writes takes 76 seconds. Final file is just 41 KB.

       

      <namedCache name="sipdb">

      <loaders passivation="false" shared="false" preload="false">

      <loader class="org.infinispan.loaders.file.FileCacheStore">

         <async enabled="true" modificationQueueSize="100"/>

         <properties>

            <property name="location" value="target/cache"/>

            <property name="fsyncMode" value="periodic"/>

            <property name="fsyncInterval" value="20000"/>

            <property name="streamBufferSize" value="16384"/>

         </properties>

      </loader>

      </loaders>

      </namedCache>

       

      code:

      static void fillCache(int records) {
      Cache <Integer, String> cache = manager.getCache("sipdb");
      for (int i=1;i<=records;i++) {
      cache.put(i, i+"@customer" + RandomStringUtils.randomNumeric(3) + ".sip.ourdomain.net");
      }
      }
        • 1. Re: Very slow FileCacheStore
          hsn10

          jdbm cache store is even slower. What do you use in production?

          • 2. Re: Very slow FileCacheStore
            mircea.markus

            Is this a performance degradation from a previous ISPN version or you've only tried 5.1.6.Final?

            Note that having the modificationQueueSize == 100 can also be a point of contention, as once the queue fills up the threads trying to add to it would block till objects are being persisted in an async manner. Can you increase it to a higher value (e.g. 10k) and give it a try?

            • 3. Re: Very slow FileCacheStore
              hsn10

              I only tried 5.1.6. I am new to this software.

               

              If i set queue size to 10k then its faster:

               

              4.9.2012 10:48:12 org.infinispan.factories.GlobalComponentRegistry start

              INFO: ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.6.FINAL

              4.9.2012 10:48:13 org.infinispan.loaders.jdbm.JdbmCacheStore openDatabase

              INFO: ISPN009001: JDBM database target\cache\sipdb2 opened

              1000 cache filled in 6.063 seconds

              4.9.2012 10:48:43 org.infinispan.loaders.decorators.AsyncStore stop

              ERROR: Async store executor did not stop properly

               

              But 6 seconds for just queue 1000 items is still a lot, it should be fraction of second.

               

              manager.stop(); does not wait till all async items are persisted - data loss.

              • 4. Re: Very slow FileCacheStore
                hsn10

                most likely lock contention between cache.put and async serializing. Are you using ConcurrentLinkedQueue ?

                • 5. Re: Very slow FileCacheStore
                  mircea.markus

                  But 6 seconds for just queue 1000 items is still a lot, it should be fraction of second.

                  Agreed, that doesn't relly make sense. Can you please uploadr your sample code + config to profile it a bit.

                  We're alo in the process of coming with a new FCS implementation[1], in case you feel like taking a look or even writting some code :-) 

                   

                  [1]

                  https://community.jboss.org/wiki/FileCacheStoreRedesign

                  • 6. Re: Very slow FileCacheStore
                    hsn10

                    I am not interested in writing FileCache storage. Its complex task and it should be delegated to external library. You can also take a peek on ehcache, it has some disk based serializer too.

                     

                    I am currently writing cache loader for jdbm3.

                    • 7. Re: Very slow FileCacheStore
                      mircea.markus

                      That would be a file cache store implementation by itself :-) Hope you'll contribute it back?

                      • 8. Re: Very slow FileCacheStore
                        hsn10

                        ISPN-2259 created

                        • 9. Re: Very slow FileCacheStore
                          hsn10

                          Can anybody skilled with profiling to look at https://issues.jboss.org/browse/ISPN-2259 ? There will be lock contention somewhere in write path.

                          • 10. Re: Very slow FileCacheStore
                            hsn10

                            I  profiled it and there are 2 lock contentions and 1 (probably) missdesign. Why you are reading from cachestore at write path?