Infinispan Lucene directory and eviction

[bostjanl1]

Hi,

I am trying to use infinispan lucene directory as a backend for hibernate search based app. It works fine as long as I don't try to use eviction.

With eviction I always get the following exception:

2011-06-02 08:17:05,505 WARN EvictionManagerImpl:172 - ISPN00027: Could not acquire lock for eviction of _0.frq|10|DataEntry

java.lang.IllegalStateException: Transaction DummyTransaction{xid=DummyXid{, globalTransactionId = ByteArray{size=64, array=[48, 11, -49, 62, 48, 34, -13, 53, -98, -119, ..]}, branchQualifier = ByteArray{size=64, array=[20, 122, -73, -118, 71, 74, 48, 45, -81, 116, ..]}}, status=8} appears to no longer be valid!

          at org.infinispan.util.concurrent.locks.LockManagerImpl.lockAndRecord(LockManagerImpl.java:95)

          at org.infinispan.eviction.EvictionManagerImpl.acquireLock(EvictionManagerImpl.java:210)

          at org.infinispan.eviction.EvictionManagerImpl.onEntryEviction(EvictionManagerImpl.java:170)

          at org.infinispan.eviction.EvictionManagerImpl.onEntryEviction(EvictionManagerImpl.java:162)

          at org.infinispan.container.DefaultDataContainer$DefaultEvictionListener.onEntryEviction(DefaultDataContainer.java:201)

...

I have used the default infinispan config shipped with hibernate-search and added cache store and eviction parameters. My conf file is:

<?xml version="1.0" encoding="UTF-8"?>

<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

        xsi:schemaLocation="urn:infinispan:config:5.0 http://www.infinispan.org/schemas/infinispan-config-5.0.xsd"

        xmlns="urn:infinispan:config:5.0">

    <!-- *************************** -->

    <!-- System-wide global settings -->

    <!-- *************************** -->

    <global>

        <!-- Duplicate domains are allowed so that multiple deployments with default configuration

            of Hibernate Search applications work - if possible it would be better to use JNDI to share

            the CacheManager across applications -->

        <globalJmxStatistics enabled="true" cacheManagerName="Lucene" allowDuplicateDomains="true"/>

        <!-- If the transport is omitted, there is no way to create distributed or clustered

            caches. There is no added cost to defining a transport but not creating a cache that uses one,

            since the transport is created and initialized lazily. -->

        <transport clusterName="Lucene" distributedSyncTimeout="50000">

            <!-- Note that the JGroups transport uses sensible defaults if no configuration

                property is defined. See the JGroupsTransport javadocs for more flags -->

        </transport>

        <!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER, DONT_REGISTER.

            Hibernate Search takes care to stop the CacheManager so registering is not needed -->

        <shutdown hookBehavior="DONT_REGISTER"/>

    </global>

    <!-- *************************** -->

    <!-- Default "template" settings -->

    <!-- *************************** -->

    <default>

        <locking lockAcquisitionTimeout="20000" writeSkewCheck="false" concurrencyLevel="500" useLockStriping="false"/>

        <lazyDeserialization enabled="false"/>

        <!-- Invocation batching is required for use with the Lucene Directory -->

        <invocationBatching enabled="true"/>

        <!-- This element specifies that the cache is clustered. modes supported: distribution

            (d), replication (r) or invalidation (i). Don't use invalidation to store Lucene indexes (as

            with Hibernate Search DirectoryProvider). Replication is recommended for best performance of

            Lucene indexes, but make sure you have enough memory to store the index in your heap.

            Also distribution scales much better than replication on high number of nodes in the cluster. -->

        <clustering mode="replication">

            <!-- Prefer loading all data at startup than later -->

            <stateRetrieval timeout="20000" logFlushTimeout="30000" fetchInMemoryState="true"

                    alwaysProvideInMemoryState="true"/>

            <!-- Network calls are synchronous by default -->

            <sync replTimeout="20000"/>

        </clustering>

        <jmxStatistics enabled="true"/>

        <eviction maxEntries="-1" strategy="NONE"/>

        <expiration maxIdle="-1"/>

    </default>

    <!-- ******************************************************************************* -->

    <!-- Individually configured "named" caches.                                         -->

    <!--                                                                                 -->

    <!-- While default configuration happens to be fine with similar settings across the -->

    <!-- three caches, they should generally be different in a production environment.   -->

    <!--                                                                                 -->

    <!-- Current settings could easily lead to OutOfMemory exception as a CacheStore     -->

    <!-- should be enabled, and maybe distribution is desired.                           -->

    <!-- ******************************************************************************* -->

    <!-- *************************************** -->

    <!--  Cache to store Lucene's file metadata  -->

    <!-- *************************************** -->

    <namedCache name="LuceneIndexesMetadata">

        <!--<eviction maxEntries="1000" strategy="LIRS"/>-->

        <loaders passivation="false" shared="true">

            <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="true">

                <properties>

                    <property name="location" value="/tmp/lucene_cache"/>

                </properties>

            </loader>

        </loaders>

        <clustering mode="replication">

            <stateRetrieval fetchInMemoryState="true" logFlushTimeout="30000"/>

            <sync replTimeout="25000"/>

        </clustering>

    </namedCache>

    <!-- **************************** -->

    <!--  Cache to store Lucene data  -->

    <!-- **************************** -->

    <namedCache name="LuceneIndexesData">

        <eviction maxEntries="1000" strategy="LIRS"/>

        <loaders passivation="false" shared="true">

            <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="true">

                <properties>

                    <property name="location" value="/tmp/lucene_cache"/>

                </properties>

            </loader>

        </loaders>

        <clustering mode="replication">

            <stateRetrieval fetchInMemoryState="true" logFlushTimeout="30000"/>

            <sync replTimeout="25000"/>

        </clustering>

    </namedCache>

    <!-- ***************************** -->

    <!--  Cache to store Lucene locks  -->

    <!-- ***************************** -->

    <namedCache name="LuceneIndexesLocking">

        <clustering mode="replication">

            <stateRetrieval fetchInMemoryState="true" logFlushTimeout="30000"/>

            <sync replTimeout="25000"/>

        </clustering>

    </namedCache>

</infinispan>

This is with hibernate-search 3.4 and infinispan 5.0.0.CR3. I have also tried it with infinispan 4.2.1 and get the same exception.

Any ideas?

Thanks,

best regards,

Bostjan

[sannegrinovero]

Hello Bostjan,

Your configuration file looks correct, assuming that you don't enable eviction on the LuceneIndexesMetadata cache, which is not possible and not needed as it contains very small data anyways.

I've been preparing some tests, and could finally reproduce your issue. I'm not sure yet about the cause, but opened an issue to track this; it seems I cant get the same result with any eviction strategy.. something is definitely wrong with the batching transaction manager.

Thank you!

[bostjanl1]

Brilliant, thanks Sanne, I'll keep watching the Jira issue.

Best regards,

Bostjan

[sannegrinovero]

As a reference for others: it's indeed a bug, but it doesn't affect functionality; it's a performance issue as it will log lots of exceptions but it's still a situation from which it recovers, and because of how the Lucene data is immutable the index is stored correctly even withouth owning the lock.