Modeshape : CPU spikes every 30 seconds for 10 seconds without any software interaction

[spidex]

Hello

I am currently maintaining a software product that uses Modeshape 3.8 and Infinispan (5.2.7) for file storage and retrieval. I don't have much knowledge of Modeshape but have been reading the documentation to understand how it works.

We currently have an issue where we have 2 instances of our product which both use separate Modeshape repositories where the CPU on the machine that hosts them is spiking to 100% usage every 30 seconds for 10 seconds or so. The size of the library is around 400MB.

I have looked at one of these instances on my machine in a development environment and been able to reproduce a similar problem where the CPU spikes to about 20% usage ever 30 seconds for 10 seconds or so.

I have turned on full debug in log4j.xml for Modeshape but there doesn't seem to be any hint as to what is causing this or what it's doing. This process seems to be happening asynchronously without any user interaction to the product.

I have done some Java Profiling using VisualVM (see attachments) and found the process causing the CPU spike is called "datasabreReporisotry-FileCacheStore-0", On further inspection I think code being run is ogr.infinispan.loaders.file.FileCacheStore.loadBucket.

My question is what is happening during this process and is there any way we can prevent it?

I thought I may have been able to stop the asynchronous processing by removing the call to async() from the settings or disabling it but this doesn't seem to work. Does this relate more to when files are saved or retrieved?

ModeShape in Java applications - ModeShape 3 - Project Documentation Editor

<!--

               We could use "write-behind", which actually writes to the file system asynchronously,

               which can improve performance as seen by the JCR client.

               Plus changes are coalesced, meaning that if multiple changes are enqueued for the

               same node, only the last one is written. (This is good much of the time, but not

               always.)

             <async enabled="true" flushLockTimeout="15000" threadPoolSize="5"/>

             -->

Our current settings for Infinispan are detailed below.

// Create the Infinispan configuration via a Local Environment ...

                ConfigurationBuilder builder = new ConfigurationBuilder();

                Configuration cacheConfig = builder.transaction()

                        .transactionManagerLookup(getInfinispanTransactionmanager)

                        .transactionMode(TransactionMode.TRANSACTIONAL)

                        .autoCommit(true)

                        .lockingMode(LockingMode.PESSIMISTIC)

                        .loaders()

                        .passivation(false)

                        .shared(false)

                        .preload(false)

                        .addFileCacheStore()

                        .async()

                        .threadPoolSize(10)

                        .enabled(true)

                        .fetchPersistentState(false)

                        .purgeOnStartup(false)

                        .addProperty("location", bundle.getString("datasabre.repo") + "/Library/content")

                        .build();

If you need anymore information please let me know

Regards

John

[hchiorean]

I have done some Java Profiling using VisualVM (see attachments) and found the process causing the CPU spike is called "datasabreReporisotry-FileCacheStore-0", On further inspection I think code being run is ogr.infinispan.loaders.file.FileCacheStore.loadBucket.

Infinispan is used pretty much as black box by ModeShape. Any ISPN releated questions should be redirected to the ISPN community.

Infinispan 5.x is an extremely old and bug-ridden version of Infinispan. This also coupled with the fact that ModeShape 3.x has stopped in the community almost 2 years ago, makes me suggest to try moving to a newer version of ModeShape and Infinispan (the latest stable in the community is ModeShape 4.5 and Infinispan 7.2.4). If you can't move to a newer version, the ISPN FileCacheStore in Infinispan 5 is notoriously slow, so you should try using another cache store (perhaps BDB or JDBC).

Also, you should never use "async" when configuring ISPN for ModeShape. Using this option might cause ModeShape to read/use invalid(stale) data which can corrupt a repository.

[spidex]

Thanks for the quick response Horai

Sadly I don't think we can upgrade our version of Modeshape due to a dependency on Lucene 3.6

Re: Modeshape 3.8.0 final Spring 4.0.3 InfiniSpan 5.2.7 Lucene 4.10.0

I think we will need to go with the route of using JDBC as our cache store rather then Infinispan.

Could you point me in the right direction of how to do this? Is this much different in setup or performance then using Infinispan?

Regards

John

[hchiorean]

You can see how to configure a JDBC cache store here: modeshape/infinispan-jdbc.xml at 3.x · ModeShape/modeshape · GitHub or in one of our examples: https://github.com/ModeShape/modeshape-examples/tree/3.x/modeshape-jdbc-store-example

Performance is very much context dependent, so you will have to compare performance in your environment. Again, to figure out which is the most "performant" cache store in ISPN 5 (note you need to disambiguate this a bit: there is write performance/read performance/memory consumption, overall throughput etc) the best source of information is the ISPN community.

[spidex]

Thanks Horia

One more question is I may ask...

I decided to look at your first suggestion of upgrading Modeshape to 4.5 and Infinispan to 7.2.4 (I think we can work around the issue with Lucene).

As you mentioned (and in my testing) this seems to perform much better and the CPU spikes I mentioned seem to have gone.

ModeShape in Java applications - ModeShape 4 - Project Documentation Editor

Looking at the infinispan_configuration.xml the file structure seems to have changed since Modeshape 3.8 (or infinispan 5.2.7) from using multiple file to use a single .dat file store.

I presume this will still work o.k if we take a back up from Modeshape 3.8 and then import into Modeshape 4.5 when we migrate?

Backup and restore - ModeShape 4 - Project Documentation Editor

John

[hchiorean]

Looking at the infinispan_configuration.xml the file structure seems to have changed since Modeshape 3.8 (or infinispan 5.2.7) from using multiple file to use a single .dat file store.

not only has the ISPN configuration changed (schema wise) but also all of the ISPN cache stores are not binary compatible between 5.x and 7.x, meaning data stored in ISPN 5 cannot be read OOTB in ISPN 7. In the case of the FS cache store, AFAIK it has been completely rewritten in ISPN 7.

I presume this will still work o.k if we take a back up from Modeshape 3.8 and then import into Modeshape 4.5 when we migrate?

Backup and restore - ModeShape 4 - Project Documentation Editor

that's correct (pending there are no bugs in our backup / restore mechanism). You should be able to backup your 3.x repo (you'll have to this by writing & running a bit of code), change your configuration to 4.x, start a new empty repository and restore the 3.x backup (in 4.x you can do the restore via code, but also via the the REST service or the Web Explorer webapp).