6 Replies Latest reply on Oct 29, 2014 5:58 AM by Radim Vansa

    Huge loading time for infinispan

    Karminder Singh Newbie

      Hi,

      We are using infinispan in persisted mode.

      The size of persisted file is around 160GB. The time taken by infinispan to start is in hours.

      The configuration for "preload" is set to "false".

      Please help.

       

      Regards,

      Karminder singh

        • 1. Re: Huge loading time for infinispan
          Wolf-Dieter Fink Master

          Could you attach the configuration?

          Which mode you use?

          Do you have such big amount of data?

          • 2. Re: Re: Huge loading time for infinispan
            Karminder Singh Newbie

            Mode: We are using local mode. With Persistence and eviction.

            Attaching configuration file. Yes we have such huge data.

            • 3. Re: Re: Huge loading time for infinispan
              Radim Vansa Master

              Not that it would solve your problem, but the elements l1, hash and async configured for your (default) cache don't have any meaning in local mode. Otherwise your configuration looks OK, I don't see why you should experience any delay when starting the node.

               

              Could you share some thread dump by jstack got when the node appears to be starting?

              • 4. Re: Re: Huge loading time for infinispan
                Karminder Singh Newbie

                Following is the stack trace during initialization:

                Stack trace:

                sun.nio.ch.FileDispatcherImpl.pread0(Native Method)

                sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:52)

                sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:220)

                sun.nio.ch.IOUtil.read(IOUtil.java:197)

                sun.nio.ch.FileChannelImpl.readInternal(FileChannelImpl.java:693)

                sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:679)

                org.infinispan.persistence.file.SingleFileStore.rebuildIndex(SingleFileStore.java:167)

                org.infinispan.persistence.file.SingleFileStore.start(SingleFileStore.java:119)

                org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:122)

                sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

                sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                java.lang.reflect.Method.invoke(Method.java:483)

                org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:183)

                org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)

                org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)

                org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)

                org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)

                - locked org.infinispan.factories.ComponentRegistry@3b0a5f9b

                org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216)

                org.infinispan.CacheImpl.start(CacheImpl.java:675)

                org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:553)

                org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:516)

                org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:398)

                com.cleartrail.infinispan.BasicCache.<init>(BasicCache.java:527)

                 

                If you need something more please ask.

                • 5. Re: Huge loading time for infinispan
                  Karminder Singh Newbie

                  Hi Radim,

                  Were the above logs of any help?

                  • 6. Re: Re: Huge loading time for infinispan
                    Radim Vansa Master

                    Hi, it seems my past response got lost.

                     

                    You're right, I forgot that single file store needs to load all *keys* into memory even if not preloading (it builds up an index of entry locations). This is a design specifics and cannot be worked around.

                     

                    And with 180 GB file and 50 MB/s reading speed for discs (I am not sure how fast is yours), 1 hour reading time is expected (although the values are not loaded into memory, buffering probably reads them from disc anyway).

                     

                    You can try LevelDB store, this does not create any in-memory index when starting. Infinispan recently got new store implementation, SoftIndexFileStore, but this wouldn't solve your problem either (the index needs to be rebuilt when starting).