6 Replies Latest reply on Oct 29, 2014 5:58 AM by rvansa

    Huge loading time for infinispan

    karminder.singh
    karminder.singh Oct 20, 2014 8:19 AM

      Hi,

      We are using infinispan in persisted mode.

      The size of persisted file is around 160GB. The time taken by infinispan to start is in hours.

      The configuration for "preload" is set to "false".

      Please help.

       

      Regards,

      Karminder singh

      • 321 Views
      • Tags:
        • 1. Re: Huge loading time for infinispan
          wdfink
          wdfink Oct 21, 2014 7:06 AM (in response to karminder.singh)

          Could you attach the configuration?

          Which mode you use?

          Do you have such big amount of data?

            • Actions
          • 2. Re: Re: Huge loading time for infinispan
            karminder.singh
            karminder.singh Oct 22, 2014 3:49 AM (in response to wdfink)

            Mode: We are using local mode. With Persistence and eviction.

            Attaching configuration file. Yes we have such huge data.

              • Actions
            • 3. Re: Re: Huge loading time for infinispan
              rvansa
              rvansa Oct 22, 2014 4:07 AM (in response to karminder.singh)

              Not that it would solve your problem, but the elements l1, hash and async configured for your (default) cache don't have any meaning in local mode. Otherwise your configuration looks OK, I don't see why you should experience any delay when starting the node.

               

              Could you share some thread dump by jstack got when the node appears to be starting?

                • Actions
              • 4. Re: Re: Huge loading time for infinispan
                karminder.singh
                karminder.singh Oct 22, 2014 7:05 AM (in response to rvansa)

                Following is the stack trace during initialization:

                Stack trace:

                sun.nio.ch.FileDispatcherImpl.pread0(Native Method)

                sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:52)

                sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:220)

                sun.nio.ch.IOUtil.read(IOUtil.java:197)

                sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:693)

                sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:679)

                org.infinispan.persistence.file.SingleFileStore.rebuildIndex(SingleFileStore.java:167)

                org.infinispan.persistence.file.SingleFileStore.start(SingleFileStore.java:119)

                org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:122)

                sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

                sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                java.lang.reflect.Method.invoke(Method.java:483)

                org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:183)

                org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)

                org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)

                org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)

                org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)

                - locked org.infinispan.factories.ComponentRegistry@3b0a5f9b

                org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216)

                org.infinispan.CacheImpl.start(CacheImpl.java:675)

                org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:553)

                org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:516)

                org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:398)

                com.cleartrail.infinispan.BasicCache.<init>(BasicCache.java:527)

                 

                If you need something more please ask.

                  • Actions
                • 5. Re: Huge loading time for infinispan
                  karminder.singh
                  karminder.singh Oct 28, 2014 12:51 PM (in response to rvansa)

                  Hi Radim,

                  Were the above logs of any help?

                    • Actions
                  • 6. Re: Re: Huge loading time for infinispan
                    rvansa
                    rvansa Oct 29, 2014 5:58 AM (in response to karminder.singh)

                    Hi, it seems my past response got lost.

                     

                    You're right, I forgot that single file store needs to load all *keys* into memory even if not preloading (it builds up an index of entry locations). This is a design specifics and cannot be worked around.

                     

                    And with 180 GB file and 50 MB/s reading speed for discs (I am not sure how fast is yours), 1 hour reading time is expected (although the values are not loaded into memory, buffering probably reads them from disc anyway).

                     

                    You can try LevelDB store, this does not create any in-memory index when starting. Infinispan recently got new store implementation, SoftIndexFileStore, but this wouldn't solve your problem either (the index needs to be rebuilt when starting).

                      • Actions