ModeShape memory usage

[wesssel]

Hey all,

Using Modeshape 3.8 DV 6.1 version, we are seeing the following memory usage after around a day of usage. At the time the Modeshape repository was idle, but there has been content added in the previous 24 hours. We are wondering where these objects are being used, since it is not in our applications. No jcr sessions were active and this was taken right after a full garbage collection. Heap is 8gb, 1/3 of which is young and 2/3 is old.

https://www.dropbox.com/s/ax4fjfa0kg7dvke/ModeshapeMemUsage.png?dl=0

I read somewhere ModeShape tries to keep as much as it can in memory, so this should be cleared when memory is running low? Eviction is set to 40,000 entries.

[hchiorean]

I read somewhere ModeShape tries to keep as much as it can in memory, so this should be cleared when memory is running low? Eviction is set to 40,000 entries.

I'm curios where you read that, if you can point me to the source of the article I'd be happy to take a look.

ModeShape by itself (outside of what Infinispan does & caches) only has a small in-memory cache in the form of workspace caches. These workspace caches are essentially in-memory maps, configured by default as https://github.com/ModeShape/modeshape/blob/3.x/modeshape-jcr/src/main/resources/org/modeshape/jcr/default-workspace-cache-config.xml. As you see in that configuration, the number of entries which *should* be cached (I'm saying should because the operation is done essentially by an Infinispan cache) is really small and should have no impact on overall memory usage. Apart from this in-memory cache, ModeShape does not cache/explicitly hold onto anything else. However, as long as a session instance is referenceable from a GC root all the nodes/items manipulated by that session will not be GCed.

From the heap snapshot that you provided the large number of chars/Strings and LinkedListMultiMap$Entry objects is indicative of parents which have lots of children. This is something which has been intensively discussed in blog posts, JIRA issues and forum posts, so I will not go into all the details again. In short, if you have 1 node which  has 500k children and you load this node (for example in order to add a new child) ModeShape will load into memory *all its child references* resulting in creating the previously mentioned objects (just the child references, not the entire children). The child references objects are essentially [string,string] pairs held into a special Map. If your application uses these large, flat hierarchies, ModeShape will take up a lot of memory and there is no way around that. The only solution is to design your node hierarchy so that you avoid these types of structures.

If on the other hand you're performing an operation via a session - e.g load lots of nodes - and you don't hold onto that session and call session.logout() at the end of the operation, all the objects loaded/referred to by that session are/should be GCed, essentially freeing up the memory. If that is not happening, you have to profile your application locally and trace the GC roots of the objects which are not being reclaimed, investigating why this is happening.

[wesssel]

We have aimed to keep number of child nodes small, with around a maximum of 2000 children per node. Will investigate what is causing these objects to be held in memory.

[wesssel]

I have checked to see all JcrSessions are closed properly and indeed they are. After a full GC all JcrSession objects disappear, however a large number of objects related to Modeshape survive the GC.

Screenshot after full GC:

https://www.dropbox.com/s/9328vglxd4ax703/Selection_082.png?dl=0

Is there anything inside Modeshape that could keep hold of these objects? Indexing was not taking place at the time of the snapshot. I will try to figure out how to do better GC analysis and see what's being kept in memory.

[hchiorean]

The JcrSession objects (which are in turn held in a Map inside the JcrReposiory instance and released when calling session.logout) are the "main roots" in terms of object references. If those are GCed correctly anything that's tied directly to them will be GCed as well.

Is there anything inside Modeshape that could keep hold of these objects?

You have to figure out in our own system what these objects are and who is (still) referencing them after a GC. All the ModeShape specific objects are normally tied to & released when each JcrSession instance is being logged out (assuming your code doesn't have any additional references towards these sessions). However, there are other "moving parts" which ModeShape does not control - for example the Infinispan Cache and Hibernate Search tasks in case of indexing. Moreover, the Infinispan eviction settings *presumably* have an influence over memory usage, but from ModeShape's perspective Infinispan is a black box (remember that the max-entries settings on the Infinispan cache eviction does not refer the number of entries in the cache but rather something related to the number of buckets in their custom Map implementation - Infinispan Eviction Strategy - Stack Overflow)

All in all it's impossible to tell from looking at a heap snapshot why objects are still there.

[wesssel]

I think it says the number of entries will never be larger than max-entries, but eviction can happen before the max-entries has been reached.

[wesssel]

After taking a full heap dump I came across this:

https://www.dropbox.com/s/8lf0exrzkus1qso/Screenshot%20from%202015-05-14%2014%3A16%3A49.png?dl=0

I'm new to memory analysis but it seems there is a ConcurrentHashMap being used, instead of a BoundedConcurrentHashmap. I.e. there is an infinispan cache defined without eviction set, even though the Modeshape cache has eviction set, What other caches are being used in Modeshape (querying?) that could have an infinispan cache without eviction?

[hchiorean]

ModeShape does not use any explicit cache for querying if you've configured local, FS based index storage. The only caches that ModeShape uses is the a) the transient (non persistent) workspace cache and b) the main Infinispan cache for data storage. Any other caches you may see are not explicitly used/interacted with by ModeShape.

The modeshape-indexer thread (from the attached picture) is used  for reindexing, which can take place (depending on your configuration) either a) never or b) at repository startup or c) when explicitly called via the API workspace.reindex. If reindexing is performed and you have a lot of nodes *that will load all the nodes into the workspace ISPN cache* because there's simply no other way to do it. However, the workspace cache has eviction enabled, so that *should* work just like any other regular ISPN cache with eviction enabled.

[hchiorean]

It may happen (I need to investigate some more) that the EAP kit does not read the correct definition for the workspace cache, causing ModeShape to use a transient cache without any eviction configured (which might be what you're seeing). I will investigate this some more and if necessary raise a JIRA issue (and cross comment here).

In the meantime, you can try the following workaround in your EAP configuration (assuming the name of your repository is X and you're only using one workspace - default):

1) Define a cache-container used for workspaces:

<cache-container name="ws-cache-container" default-cache="X/system" module="org.modeshape">
                <local-cache name="X/system">
                    <eviction strategy="LRU" max-entries="100"/>
                    <expiration lifespan="10000" interval="1000" max-idle="5000"/>
                </local-cache>
                <local-cache name="X/default">
                    <eviction strategy="LRU" max-entries="100"/>
                    <expiration lifespan="10000" interval="1000" max-idle="5000"/>
                </local-cache>
          </cache-container>

2) Force the repository configuration to use this ws cache config:

  <repository name="X"...>
    <workspaces cache-container="ws-cache-container">
  ...

This should force ModeShape to use the specific caches for the system & default workspaces.

[wesssel]

I will try this ASAP.

[wesssel]

Caused by: org.modeshape.jcr.ConfigurationException: The 'system' workspace cannot be initialized because it is configured with a transactional cache. Workspace caches should not be transactional. Please check your configuration.

[hchiorean]

My bad, the ws caches should not be transactional. Just remove the <transaction config from above (you can also remove the <locking element). 100 is a small number for a real application, make sure you adjust the eviction/expiration size to suit your case.

Also see https://issues.jboss.org/browse/MODE-2466

[wesssel]

The initial impression is that this has solved our memory issue.

[wesssel]

This has definitely solved our memory issues, running for 8 days straight and garbage collections are going smooth.