5 Replies Latest reply on Sep 24, 2015 4:34 AM by sberthez

    Approximative count of items for a node to determine eviction strategy in Infinispan

    sberthez

      Hi,

       

      Is there any approximative count of items created for each node in order to have some informations to tune eviction strategy ? Does it existing an API i could use to determine this count ? I have a lot of nodes (nearly 600 000) to load, i cannot try for every possible count.

       

      Regards

        • 1. Re: Approximative count of items for a node to determine eviction strategy in Infinispan
          hchiorean

          Is there any approximative count of items created for each node in order to have some informations to tune eviction strategy ?

          I don't really understand the question. Each node you create via the JCR API translates into one document stored in ISPN. ModeShape doesn't create any "more items" behind the scenes (the only exception is when using features like JCR Versioning or Locking when additional data is stored in the system area). Moreover, the eviction size in general is not what it seems as it refers to size of the Map buckets not the number of entries in the cache (see Frequently Asked Questions). Feel free to ask the Infinispan community about more details regarding the eviction size.

           

          As far as ModeShape is concerned, determining an "optimal" eviction size is something you have to try experimenting with, depending on your volume of data and read/write intensiveness. There is no predefined algorithm which will give you this optimal value.

          • 2. Re: Approximative count of items for a node to determine eviction strategy in Infinispan
            sberthez

            Each node you create via the JCR API translates into one document stored in ISPN. ModeShape doesn't create any "more items" behind the scenes (the only exception is when using features like JCR Versioning or Locking when additional data is stored in the system area).

            It is basically what i wanted to know.

             

            Moreover, the eviction size in general is not what it seems as it refers to size of the Map buckets not the number of entries in the cache (see Frequently Asked Questions). Feel free to ask the Infinispan community about more details regarding the eviction size.

             

            Thanks for these clarifications I will dig into Infinispan FAQ.

            • 3. Re: Approximative count of items for a node to determine eviction strategy in Infinispan
              sberthez

              Also, sorry for this strange question, but is there any way to determine if all nodes are resident in memory (ie : in Infinispan cache memory) ? I have performance issues and i would like to know if it is caused or not by partial data in memory.

              • 4. Re: Approximative count of items for a node to determine eviction strategy in Infinispan
                hchiorean

                The only way to truly determine what's eating up your memory is to attach a profiler and investigate.

                 

                Now, the object types stored in Infinispan are SchematicEntryLiteral instances, so you could look for those while profiling. However, if you've configured a) a cache-store and b) eviction it's highly unlikely that the memory consumption comes from Infinispan not offloading the entries stored in memory.

                 

                I think a particular use case is much more relevant in terms of why memory is filling up. For example, if you have 300k nodes and you run a query which is supposed to return 100k of them and you iterate through the results, *all those nodes will be loaded in memory* so you must make sure you have enough memory to do that. Another example is if you have a parent node which has 100k children and you iterate through all its children via the same session. These will be brought into memory no matter what. Infinispan is merely ModeShape's persistent store but the way in which you use the JCR/ModeShape API is usually much more relevant in terms of memory consumption.

                • 5. Re: Approximative count of items for a node to determine eviction strategy in Infinispan
                  sberthez

                  Thanks. Since my question i have found how to enable statistic mode of Infinispan and i am able to watch store state in memory using JConsole. I have enable the "preload" setting of the JDBC cache store to load all nodes at startup. It is working fine, i can tune memory usage.

                   

                  I think a particular use case is much more relevant in terms of why memory is filling up. For example, if you have 300k nodes and you run a query which is supposed to return 100k of them and you iterate through the results, *all those nodes will be loaded in memory* so you must make sure you have enough memory to do that

                  Yes, it was part of my first design because i wanted to rely on "mode:unorderedLargeCollection" introduced by last ModeShape versions but there were bottlenecks at save time when Infinispan try to save this huge node into the store. I have changed this design to distribute documents into auto generated folders. Write/read result is very quick now but now i have a problem with searches because of big number of ancestor nodes. I try to resolve this new issue by adding an index for node/mixin type in order to remove the traverse path part from query engine, not sure if it is possible. Do you have any suggestion ?