Infinispan Nullpointer in Modeshape 4.4.0 and Infinispan 7.2.3

[folch]

Hi,

We are using Modeshape 4.4.0.Final release with Infinispan 7.2.3 installed in TomEE and using Java 7.

The repository starts and works. However, if we add some concurrency (100 threads) adding and removing nodes we found, from time to time, a Nullpointer with the following stacktrace:

org.infinispan.interceptors.InvocationContextInterceptor handleAll

ERROR: ISPN000136: Execution error java.lang.NullPointerException

  at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8$LIRSEvictionPolicy.findIfEntriesNeedEvicting(BoundedEquivalentConcurrentHashMapV8.java:1397)

  at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8.compute(BoundedEquivalentConcurrentHashMapV8.java:3487)

  at org.infinispan.container.DefaultDataContainer$BoundedEquivalentConcurrentExtendedMap.putAndActivate(DefaultDataContainer.java:556)

  at org.infinispan.container.DefaultDataContainer.put(DefaultDataContainer.java:192)

  at org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:168)

  at org.infinispan.statetransfer.CommitManager.commit(CommitManager.java:98)

  at org.infinispan.interceptors.locking.ClusteringDependentLogic$LocalLogic.commitSingleEntry(ClusteringDependentLogic.java:255)

  at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.commitEntry(ClusteringDependentLogic.java:108)

  at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:371)

  at org.infinispan.interceptors.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:549)

  at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:348)

  at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:422)

  at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:453)

  at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:195)

  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)

  at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitNonTxDataWriteCommand(AbstractLockingInterceptor.java:88)

  at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitDataWriteCommand(NonTransactionalLockingInterceptor.java:40)

  at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:55)

  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)

  at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)

  at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:44)

  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)

  at org.infinispan.interceptors.CacheMgmtInterceptor.updateStoreStatistics(CacheMgmtInterceptor.java:191)

  at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:177)

  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)

  at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)

  at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)

  at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:44)

  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:71)

  at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)

  at org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1617)

  at org.infinispan.cache.impl.CacheImpl.putIfAbsentInternal(CacheImpl.java:1122)

  at org.infinispan.cache.impl.CacheImpl.putIfAbsent(CacheImpl.java:1112)

  at org.infinispan.cache.impl.CacheImpl.putIfAbsent(CacheImpl.java:1717)

  at org.infinispan.cache.impl.CacheImpl.putIfAbsent(CacheImpl.java:283)

  at org.modeshape.jcr.cache.document.WorkspaceCache.getNode(WorkspaceCache.java:244)

  at org.modeshape.jcr.cache.document.NodeCacheIterator.nextNode(NodeCacheIterator.java:103)

  at org.modeshape.jcr.cache.document.NodeCacheIterator.hasNext(NodeCacheIterator.java:70)

  at org.modeshape.jcr.query.NodeSequence$16.hasNext(NodeSequence.java:1450)

  at org.modeshape.jcr.query.NodeSequence$23.findNext(NodeSequence.java:1863)

  at org.modeshape.jcr.query.NodeSequence$23.hasNext(NodeSequence.java:1849)

  at org.modeshape.jcr.query.NodeSequence$23.findNext(NodeSequence.java:1863)

  at org.modeshape.jcr.query.NodeSequence$23.hasNext(NodeSequence.java:1849)

  at org.modeshape.jcr.query.NodeSequence$23.findNext(NodeSequence.java:1863)

  at org.modeshape.jcr.query.NodeSequence$23.hasNext(NodeSequence.java:1849)

  at org.modeshape.jcr.query.NodeSequence$SingleWidthBatch.<init>(NodeSequence.java:1920)

  at org.modeshape.jcr.query.NodeSequence.copy(NodeSequence.java:1907)

  at org.modeshape.jcr.query.engine.process.RestartableSequence.loadBatch(RestartableSequence.java:213)

  at org.modeshape.jcr.query.engine.process.RestartableSequence$1.nextBatch(RestartableSequence.java:92)

  at org.modeshape.jcr.query.engine.process.RestartableSequence.nextBatch(RestartableSequence.java:119)

  at org.modeshape.jcr.query.JcrQueryResult$QueryResultIterator.findNextBatch(JcrQueryResult.java:257)

  at org.modeshape.jcr.query.JcrQueryResult$QueryResultIterator.hasNext(JcrQueryResult.java:241)

  at com.hp.exstream.hes.repository.impl.ApplicationRepository.searchApplications(ApplicationRepository.java:99)
  ...

Attached you can find infinispan_config.xml and modeshape_config.json.

We've tried to disable Eviction (NONE) and also change it to LRU with same results.

Do you know what is happening? Are we doing something wrong? With Modeshape 3.8.0 the same code is working with the same load.

Thanks

[hchiorean]

This issue seems similar to [ISPN-5522] LIRS BoundedConcurrentHashMapv8 eviction for tree bin missing check - JBoss Issue Tracker which should've been fixed in 7.2.3.Final.

One possible reason why this keeps happening even after you've changed the eviction policy of the main cache, is because the ModeShape workspace internal cache also uses LIRS by default. So you can also try the workaround described in [MODE-2479] Upgrade to ISPN 7.2.3 or newer to avoid LIRS eviction bug - JBoss Issue Tracker - explicitly configuring the workspace cache by using a separate ISPN xml configuration file. You can really use the same values as the default: modeshape/default-workspace-cache-config.xml at modeshape-4.4.0.Final · ModeShape/modeshape · GitHub just select another eviction strategy.

ModeShape 4.5.0 will use Infinispan 7.2.4.Final (which is available), so my first suggestion is to try using that version. If the problem is still there, you should report it to the Infinispan folks (Communications - Infinispan) and try the workaround mentioned above.