-
1. Re: Approximative count of items for a node to determine eviction strategy in Infinispan
hchiorean Sep 23, 2015 4:09 AM (in response to sberthez)Is there any approximative count of items created for each node in order to have some informations to tune eviction strategy ?
I don't really understand the question. Each node you create via the JCR API translates into one document stored in ISPN. ModeShape doesn't create any "more items" behind the scenes (the only exception is when using features like JCR Versioning or Locking when additional data is stored in the system area). Moreover, the eviction size in general is not what it seems as it refers to size of the Map buckets not the number of entries in the cache (see Frequently Asked Questions). Feel free to ask the Infinispan community about more details regarding the eviction size.
As far as ModeShape is concerned, determining an "optimal" eviction size is something you have to try experimenting with, depending on your volume of data and read/write intensiveness. There is no predefined algorithm which will give you this optimal value.
-
2. Re: Approximative count of items for a node to determine eviction strategy in Infinispan
sberthez Sep 23, 2015 5:04 AM (in response to hchiorean)Each node you create via the JCR API translates into one document stored in ISPN. ModeShape doesn't create any "more items" behind the scenes (the only exception is when using features like JCR Versioning or Locking when additional data is stored in the system area).
It is basically what i wanted to know.
Moreover, the eviction size in general is not what it seems as it refers to size of the Map buckets not the number of entries in the cache (see Frequently Asked Questions). Feel free to ask the Infinispan community about more details regarding the eviction size.
Thanks for these clarifications I will dig into Infinispan FAQ.
-
3. Re: Approximative count of items for a node to determine eviction strategy in Infinispan
sberthez Sep 23, 2015 1:15 PM (in response to hchiorean)Also, sorry for this strange question, but is there any way to determine if all nodes are resident in memory (ie : in Infinispan cache memory) ? I have performance issues and i would like to know if it is caused or not by partial data in memory.
-
4. Re: Approximative count of items for a node to determine eviction strategy in Infinispan
hchiorean Sep 24, 2015 3:16 AM (in response to sberthez)The only way to truly determine what's eating up your memory is to attach a profiler and investigate.
Now, the object types stored in Infinispan are SchematicEntryLiteral instances, so you could look for those while profiling. However, if you've configured a) a cache-store and b) eviction it's highly unlikely that the memory consumption comes from Infinispan not offloading the entries stored in memory.
I think a particular use case is much more relevant in terms of why memory is filling up. For example, if you have 300k nodes and you run a query which is supposed to return 100k of them and you iterate through the results, *all those nodes will be loaded in memory* so you must make sure you have enough memory to do that. Another example is if you have a parent node which has 100k children and you iterate through all its children via the same session. These will be brought into memory no matter what. Infinispan is merely ModeShape's persistent store but the way in which you use the JCR/ModeShape API is usually much more relevant in terms of memory consumption.
-
5. Re: Approximative count of items for a node to determine eviction strategy in Infinispan
sberthez Sep 24, 2015 4:34 AM (in response to hchiorean)Thanks. Since my question i have found how to enable statistic mode of Infinispan and i am able to watch store state in memory using JConsole. I have enable the "preload" setting of the JDBC cache store to load all nodes at startup. It is working fine, i can tune memory usage.
I think a particular use case is much more relevant in terms of why memory is filling up. For example, if you have 300k nodes and you run a query which is supposed to return 100k of them and you iterate through the results, *all those nodes will be loaded in memory* so you must make sure you have enough memory to do that
Yes, it was part of my first design because i wanted to rely on "mode:unorderedLargeCollection" introduced by last ModeShape versions but there were bottlenecks at save time when Infinispan try to save this huge node into the store. I have changed this design to distribute documents into auto generated folders. Write/read result is very quick now but now i have a problem with searches because of big number of ancestor nodes. I try to resolve this new issue by adding an index for node/mixin type in order to remove the traverse path part from query engine, not sure if it is possible. Do you have any suggestion ?