Restore and OutOfmemory error

[ludopaquet]

Hi,

We are trying to backup and restore a filesystem based cache (we want eventually to defragment it).

The size of backup files is 379M.

However when we try to restore the repository we have an OutOfmemory (even with xmx 4g).

The local cache is configured as follow :

<eviction strategy="LRU" max-entries="500"/>

  <transaction

  transaction-manager-lookup="org.infinispan.transaction.lookup.DummyTransactionManagerLookup"

  mode="NON_XA" locking="PESSIMISTIC" />

  <persistence>

  <file-store fetch-state="false" shared="false" preload="false"

  purge="false" path="/data/store" />

  </persistence>

Can we configure the transaction Manager to avoid loading all documents in memory ?

Many thanks

[hchiorean]

DummyTransactionManager holds all state in memory and there's no way around that, except using another tx manager. So if you've profiled and that's the reason for the OOM, you should use another tx manager (for example Atomikos or JBoss).

[ludopaquet]

Thanks we are going to investigate further with others transactionManagers

Ludovic

[ludopaquet]

Do you think it could be a good feature request to have a commit every x documents ?

I can write a feature request on github if you think it could be interesting too.

[hchiorean]

What exactly is cause of the OOM ? (i.e. why can't the object references be collected during the restore process)

Regarding the feature, there are a couple of things to keep in mind:

1. ModeShape 5 will no longer be using Infinispan, so if the problem is related to ISPN it's something which may not happen with ModeShape 5

2. the entire restore operation has to be atomic. If any one operation fails during the restore operation, *everything* has to be correctly rolled back. Having multiple transactions complicates this (it's not impossibe to do, but the chances of leaving the repository inconsistent in case of a failure are greater)

If you want, feel free log a JIRA.

[ludopaquet]

Yes I think it's related to infinispan.

Actually during the restore process we have a lot of org.infinispan.schematic.internal.document.BasicDocument calling from org.infinispan.schematic.internal.CacheSchematicDb.put(Document) I think.

For the moment to restore the base we have no choice but to use an R3 Machine on Amazon...

[hchiorean]

The Basic document instances are ModeShape's (although the package name is misleading). The loop where documents are read and stored in the cache store is here: https://github.com/ModeShape/modeshape/blob/4.x/modeshape-jcr/src/main/java/org/modeshape/jcr/BackupService.java#L656

Since ISPN does have eviction enabled, I don't quite understand why the documents created in the above loop would all be hard-referenced, preventing GC. Can you look at incoming hard references at the end of the restore process (or at some point during the restore) and see where they are coming from ? As an alternative, if you can create & attach a mem snapshot of the restore process using Flight Recorder for example, I can look at it locally some more.

[ludopaquet]

Many thanks Horia for your help. Here the flight record.

Ludovic

[hchiorean]

It seems the current backup & restore approach won't really work for large number of documents (see [MODE-2587] Repository backup and restore should be performed in batches - JBoss Issue Tracker which we'll hopefully implement for 5.0).

[ludopaquet]

Thanks Horia for taking into account of our issue.