Infinispan eviction bug in ISPN 7.2.0 / MS 4.3.0?

[wesssel]

[[0m^[[31m22:23:40,685 ERROR [com.onior.onem.AddService] (pool-5-thread-4) Exception in call()

^[[0m^[[31m22:23:40,686 ERROR [stderr] (pool-5-thread-4) java.lang.ClassCastException: java.lang.Object cannot be cast to org.infinispan.container.entries.InternalCacheEntry

^[[0m^[[31m22:23:40,686 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.container.DefaultDataContainer$DefaultEvictionListener.onEntryChosenForEviction(DefaultDataContainer.java:272)

^[[0m^[[31m22:23:40,686 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8.notifyListenerOfRemoval(BoundedEquivalentConcurrentHashMapV8.java:2524)

^[[0m^[[31m22:23:40,686 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8.replaceNode(BoundedEquivalentConcurrentHashMapV8.java:2619)

^[[0m^[[31m22:23:40,686 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8$LIRSEvictionPolicy.findIfEntriesNeedEvicting(BoundedEquivalentConcurrentHashMapV8.java:1392)

^[[0m^[[31m22:23:40,686 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.commons.util.concurrent.jdk8backported.BoundedEquivalentConcurrentHashMapV8.get(BoundedEquivalentConcurrentHashMapV8.java:2296)

^[[0m^[[31m22:23:40,686 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.container.DefaultDataContainer.get(DefaultDataContainer.java:154)

^[[0m^[[31m22:23:40,687 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.container.EntryFactoryImpl.getFromContainer(EntryFactoryImpl.java:266)

^[[0m^[[31m22:23:40,687 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.container.EntryFactoryImpl.wrapEntryForReading(EntryFactoryImpl.java:71)

^[[0m^[[31m22:23:40,687 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.interceptors.EntryWrappingInterceptor.visitDataReadCommand(EntryWrappingInterceptor.java:129)

^[[0m^[[31m22:23:40,687 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.interceptors.EntryWrappingInterceptor.visitGetKeyValueCommand(EntryWrappingInterceptor.java:120)

^[[0m^[[31m22:23:40,687 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

^[[0m^[[31m22:23:40,687 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)

^[[0m^[[31m22:23:40,687 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:70)

^[[0m^[[31m22:23:40,688 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:70)

^[[0m^[[31m22:23:40,688 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

^[[0m^[[31m22:23:40,688 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)

^[[0m^[[31m22:23:40,688 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:111)

^[[0m^[[31m22:23:40,688 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:86)

^[[0m^[[31m22:23:40,688 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

^[[0m^[[31m22:23:40,688 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)

^[[0m^[[31m22:23:40,689 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:346)

^[[0m^[[31m22:23:40,689 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:318)

^[[0m^[[31m22:23:40,689 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

^[[0m^[[31m22:23:40,689 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)

^[[0m^[[31m22:23:40,689 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.interceptors.CacheMgmtInterceptor.visitDataReadCommand(CacheMgmtInterceptor.java:103)

^[[0m^[[31m22:23:40,689 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:91)

^[[0m^[[31m22:23:40,690 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

^[[0m^[[31m22:23:40,690 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:97)

^[[0m^[[31m22:23:40,690 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:102)

^[[0m^[[31m22:23:40,690 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:71)

^[[0m^[[31m22:23:40,690 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:86)

^[[0m^[[31m22:23:40,690 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40)

^[[0m^[[31m22:23:40,690 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:336)

^[[0m^[[31m22:23:40,690 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:430)

^[[0m^[[31m22:23:40,690 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:422)

^[[0m^[[31m22:23:40,690 ERROR [stderr] (pool-5-thread-4)        at org.infinispan.schematic.internal.CacheSchematicDb.get(CacheSchematicDb.java:72)

^[[0m^[[31m22:23:40,691 ERROR [stderr] (pool-5-thread-4)        at org.modeshape.jcr.cache.document.LocalDocumentStore.get(LocalDocumentStore.java:71)

^[[0m^[[31m22:23:40,691 ERROR [stderr] (pool-5-thread-4)        at org.modeshape.jcr.cache.document.WorkspaceCache.documentFor(WorkspaceCache.java:179)

^[[0m^[[31m22:23:40,691 ERROR [stderr] (pool-5-thread-4)        at org.modeshape.jcr.cache.document.WorkspaceCache.documentFor(WorkspaceCache.java:198)

^[[0m^[[31m22:23:40,748 ERROR [stderr] (pool-5-thread-4)        at org.modeshape.jcr.cache.document.WorkspaceCache.getNode(WorkspaceCache.java:230)

^[[0m^[[31m22:23:40,748 ERROR [stderr] (pool-5-thread-4)        at org.modeshape.jcr.cache.document.AbstractSessionCache.getNode(AbstractSessionCache.java:240)

^[[0m^[[31m22:23:40,749 ERROR [stderr] (pool-5-thread-4)        at org.modeshape.jcr.cache.document.WritableSessionCache.getNode(WritableSessionCache.java:161)

^[[0m^[[31m22:23:40,749 ERROR [stderr] (pool-5-thread-4)        at org.modeshape.jcr.cache.document.AbstractSessionCache.getNode(AbstractSessionCache.java:245)

^[[0m^[[31m22:23:40,749 ERROR [stderr] (pool-5-thread-4)        at org.modeshape.jcr.JcrSession.cachedNode(JcrSession.java:626)

^[[0m^[[31m22:23:40,749 ERROR [stderr] (pool-5-thread-4)        at org.modeshape.jcr.JcrSession.node(JcrSession.java:656)

^[[0m^[[31m22:23:40,749 ERROR [stderr] (pool-5-thread-4)        at org.modeshape.jcr.JcrSession.node(JcrSession.java:675)

^[[0m^[[31m22:23:40,749 ERROR [stderr] (pool-5-thread-4)        at org.modeshape.jcr.JcrSession.getNode(JcrSession.java:859)

^[[0m^[[31m22:23:40,749 ERROR [stderr] (pool-5-thread-4)        at org.modeshape.jcr.JcrSession.getNode(JcrSession.java:842)

^[[0m^[[31m22:23:40,750 ERROR [stderr] (pool-5-thread-4)        at org.modeshape.jcr.JcrSession.getNode(JcrSession.java:126)

^[[0m^[[31m22:23:40,750 ERROR [stderr] (pool-5-thread-4)        at com.onior.onem.AddService$AddTask.call(AddService.java:103)

^[[0m^[[31m22:23:40,750 ERROR [stderr] (pool-5-thread-4)        at com.onior.onem.AddService$AddTask.call(AddService.java:80)

^[[0m^[[31m22:23:40,750 ERROR [stderr] (pool-5-thread-4)        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

^[[0m^[[31m22:23:40,750 ERROR [stderr] (pool-5-thread-4)        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

^[[0m^[[31m22:23:40,750 ERROR [stderr] (pool-5-thread-4)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

^[[0m^[[31m22:23:40,750 ERROR [stderr] (pool-5-thread-4)        at java.lang.Thread.run(Thread.java:745)

This occurs some time after starting 10 threads all adding nodes to modeshape in small batches. Anyone else see this behavior? Using WildFly 8.2.0.Final with Modeshape 4.3.0.Final with Jdbc cache store backend.

Seems I've found it:

https://issues.jboss.org/browse/ISPN-5522

[william.burns]

This issue was found previously [1].  It is fixed in [2] which will be included in 7.2.3.  If you run with LRU this issue won't happen.

[1] https://developer.jboss.org/message/932819

[2] [ISPN-5522] LIRS BoundedConcurrentHashMapv8 eviction for tree bin missing check - JBoss Issue Tracker

[wesssel]

How can I override the workspace cache in Modeshape 4.3's WildFly kit?

I tried the following, but it still seems to be using a cache with LIRS eviction causing exceptions:

<?xml version="1.0" encoding="UTF-8"?>

<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

        xsi:schemaLocation="urn:infinispan:config:7.0 http://www.infinispan.org/schemas/infinispan-config-7.0.xsd"

        xmlns="urn:infinispan:config:7.0">

        <cache-container default-cache="sample" statistics="false">

                <jmx duplicate-domains="true" />

                <local-cache name="sample">

                        <transaction mode="NON_XA" locking="PESSIMISTIC" />

                        <persistence passivation="false">

                                <string-keyed-jdbc-store xmlns="urn:infinispan:config:store:jdbc:7.0"

                                        fetch-state="false" read-only="false" purge="false">

                                        <data-source jndi-url="java:/jdbc/ODM" />

                                        <string-keyed-table prefix="ISPN"

                                                create-on-start="true" drop-on-exit="false">

                                                <id-column name="ID_COLUMN" type="VARCHAR(255)" />

                                                <data-column name="DATA_COLUMN" type="BYTEA" />

                                                <timestamp-column name="TIMESTAMP_COLUMN" type="NUMERIC" />

                                        </string-keyed-table>

                                <write-behind thread-pool-size="3"/>

                                </string-keyed-jdbc-store>

                        </persistence>

                        <eviction max-entries="10000" strategy="LRU"/>

                </local-cache>

                <local-cache name="default">                     <eviction max-entries="10000" strategy="LRU"/><expiration lifespan="120000" max-idle="6000"/>             </local-cache>

                <local-cache name="system/default">                     <eviction max-entries="10000" strategy="LRU"/><expiration lifespan="120000" max-idle="6000"/>             </local-cache>

                <local-cache name="sample/default">                     <eviction max-entries="10000" strategy="LRU"/><expiration lifespan="120000" max-idle="6000"/>             </local-cache>

        </cache-container>

</infinispan>

java.lang.ClassCastException: java.lang.Object cannot be cast to org.infinispan.container.entries.InternalCacheEntry

at org.infinispan.container.DefaultDataContainer$DefaultEvictionListener.onEntryChosenForEviction(DefaultDataContainer.java:272) [infinispan-core.jar:7.2.0.Final]

        at org.modeshape.jcr.cache.document.WorkspaceCache.evictChangedNodes(WorkspaceCache.java:296) [modeshape-jcr-4.3.0.Final.jar:4.3.0.Final]

[hchiorean]

Did you configure the workspaces element like https://github.com/ModeShape/modeshape/blob/modeshape-4.3.0.Final/integration/modeshape-jbossas-integration-tests/src/main/resources/kit/jboss-wf8/standalone/configuration/standalone-modeshape.xml#L501 ?

You need to first configure the workspaces XML element and then make sure that ISPN configuration file exists at that location. After that you can either define a global default like: modeshape/ws-cache-config.xml at modeshape-4.3.0.Final · ModeShape/modeshape · GitHub or on a per workspace basis where the name of each workspace cache follows the repositoryName/workspaceName pattern.

[wesssel]

Nope, I did not specify the cache configuration element so I guess it was using the default cache configuration which has LIRS. Thanks!

[wesssel]

Perhaps it's a good idea to post this on the front page of the Modeshape website, any repositories using the default workspace cache using LIRS will eventually throw these exceptions if eviction is triggered.

[hchiorean]

I've opened [MODE-2479] Upgrade to ISPN 7.2.3 or newer to avoid LIRS eviction bug - JBoss Issue Tracker to be able to track this issue. Once ISPN releases a version with a fix, we'll move to using that version.

In the meantime the workaround is to define a custom default ws cache which isn't using LIRS.

[rhauch]

Infinispan 7.2.3.Final is now available, and it includes the fix for ISPN-5522.