Hi Everyone,

I'm having issues getting my index to be persistant when using Infinispan. When not trying to use persistance it works just fine however as soon as I start trying to use a FileCacheStore for persistance I start getting exceptions.

It works for about 2 hours when this exception begins to be thrown:

[code][2011-08-29 11:30:53,425] ERROR FileCacheStore.java:317 Hibernate Search: indexwriter-154 ) ISPN000063: Exception while saving bucket Bucket{entries={_4o.fdt|M|cnwk.foreman.model.SoftwareDownload=ImmortalCacheEntry{key=_4o.fdt|M|cnwk.foreman.model.SoftwareDownload, value=ImmortalCacheValue{value=FileMetadata{lastModified=1314642653425, size=32768}}}}, bucketId='1509281792'}

java.io.FileNotFoundException: /var/opt/fullTextStore/LuceneIndexesMetadata/1509281792 (Too many open files)

  at java.io.RandomAccessFile.open(Native Method)

  at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)

  at org.infinispan.loaders.file.FileCacheStore$BufferedFileSync.createChannel(FileCacheStore.java:494)

  at org.infinispan.loaders.file.FileCacheStore$BufferedFileSync.write(FileCacheStore.java:472)

  at org.infinispan.loaders.file.FileCacheStore.updateBucket(FileCacheStore.java:315)

  at org.infinispan.loaders.bucket.BucketBasedCacheStore.insertBucket(BucketBasedCacheStore.java:137)

  at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:94)

  at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:49)

  at org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:195)

  at org.infinispan.interceptors.CacheStoreInterceptor.visitPutKeyValueCommand(CacheStoreInterceptor.java:210)

  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)

  at org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:82)

  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)

  at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133)

  at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:60)

  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)

  at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:214)

  at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:162)

  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)

  at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:114)

  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)

  at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104)

  at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:64)

  at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:60)

  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)

  at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:77)

  at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:60)

  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

  at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:274)

  at org.infinispan.CacheImpl.put(CacheImpl.java:515)

  at org.infinispan.CacheSupport.put(CacheSupport.java:51)

  at org.infinispan.lucene.InfinispanIndexOutput.close(InfinispanIndexOutput.java:206)

  at org.apache.lucene.util.IOUtils.closeSafely(IOUtils.java:80)

  at org.apache.lucene.index.FieldsWriter.close(FieldsWriter.java:111)

  at org.apache.lucene.index.FieldsWriter.abort(FieldsWriter.java:121)

  at org.apache.lucene.index.StoredFieldsWriter.abort(StoredFieldsWriter.java:90)

  at org.apache.lucene.index.DocFieldProcessor.abort(DocFieldProcessor.java:71)

  at org.apache.lucene.index.DocumentsWriter.abort(DocumentsWriter.java:421)

  at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:729)

  at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2042)

  at org.hibernate.search.backend.impl.lucene.works.AddWorkDelegate.performWork(AddWorkDelegate.java:76)

  at org.hibernate.search.backend.impl.batchlucene.DirectoryProviderWorkspace.doWorkInSync(DirectoryProviderWorkspace.java:96)

  at org.hibernate.search.backend.impl.batchlucene.DirectoryProviderWorkspace$AsyncIndexRunnable.run(DirectoryProviderWorkspace.java:144)

  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

  at java.lang.Thread.run(Thread.java:680)[/code]

I know that the error says I am opening too many files for my OS but I have already doubled the limit and it's still giving me the error.

Here is my configuration:

hibernate.cfg.xml:

[code] <property name="hibernate.search.default.directory_provider">infinispan</property>

        <property name="hibernate.search.infinispan.configuration_resourcename">infinispan.cfg.xml</property>

[/code]

infinispan.cfg.xml

[code]<?xml version="1.0" encoding="UTF-8"?>

<infinispan

        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

        xsi:schemaLocation="urn:infinispan:config:4.2 http://www.infinispan.org/schemas/infinispan-config-4.2.xsd"

        xmlns="urn:infinispan:config:4.2">

    <!-- *************************** -->

    <!-- System-wide global settings -->

    <!-- *************************** -->

    <global>

        <!-- Duplicate domains are allowed so that multiple deployments with default configuration

            of Hibernate Search applications work - if possible it would be better to use JNDI to share

            the CacheManager across applications -->

        <globalJmxStatistics

                enabled="true"

                cacheManagerName="HibernateSearch"

                allowDuplicateDomains="true"/>

        <!-- If the transport is omitted, there is no way to create distributed or clustered

            caches. There is no added cost to defining a transport but not creating a cache that uses one,

            since the transport is created and initialized lazily. -->

        <transport

                clusterName="HibernateSearch-Infinispan-cluster"

                distributedSyncTimeout="50000">

            <!-- Note that the JGroups transport uses sensible defaults if no configuration

                property is defined. See the JGroupsTransport javadocs for more flags -->

        </transport>

        <!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER, DONT_REGISTER.

            Hibernate Search takes care to stop the CacheManager so registering is not needed -->

        <shutdown

                hookBehavior="DONT_REGISTER"/>

    </global>

    <!-- *************************** -->

    <!-- Default "template" settings -->

    <!-- *************************** -->

    <default>

        <locking

                lockAcquisitionTimeout="20000"

                writeSkewCheck="false"

                concurrencyLevel="500"

                useLockStriping="false"/>

        <lazyDeserialization

                enabled="false"/>

        <!-- Invocation batching is required for use with the Lucene Directory -->

        <invocationBatching

                enabled="true"/>

        <!-- This element specifies that the cache is clustered. modes supported: distribution

            (d), replication (r) or invalidation (i). Don't use invalidation to store Lucene indexes (as

            with Hibernate Search DirectoryProvider). Replication is recommended for best performance of

            Lucene indexes, but make sure you have enough memory to store the index in your heap.

            Also distribution scales much better than replication on high number of nodes in the cluster. -->

        <clustering

                mode="replication">

            <!-- Prefer loading all data at startup than later -->

            <stateRetrieval

                    timeout="60000"

                    logFlushTimeout="30000"

                    fetchInMemoryState="true"

                    alwaysProvideInMemoryState="true"/>

            <!-- Network calls are synchronous by default -->

            <sync

                    replTimeout="20000"/>

        </clustering>

        <jmxStatistics

                enabled="true"/>

        <eviction

                maxEntries="-1"

                strategy="NONE"/>

        <expiration

                maxIdle="-1"/>

    </default>

    <!-- ******************************************************************************* -->

    <!-- Individually configured "named" caches.                                         -->

    <!--                                                                                 -->

    <!-- While default configuration happens to be fine with similar settings across the -->

    <!-- three caches, they should generally be different in a production environment.   -->

    <!--                                                                                 -->

    <!-- Current settings could easily lead to OutOfMemory exception as a CacheStore     -->

    <!-- should be enabled, and maybe distribution is desired.                           -->

    <!-- ******************************************************************************* -->

    <!-- *************************************** -->

    <!--  Cache to store Lucene's file metadata  -->

    <!-- *************************************** -->

    <namedCache name="LuceneIndexesMetadata">

        <clustering mode="replication">

            <stateRetrieval

                    fetchInMemoryState="true"

                    logFlushTimeout="30000"/>

            <sync replTimeout="25000"/>

        </clustering>

        <loaders preload="true">

            <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="true">

                <properties>

                    <property name="location" value="/var/opt/fullTextStore"/>

                </properties>

            </loader>

        </loaders>

    </namedCache>

    <!-- **************************** -->

    <!--  Cache to store Lucene data  -->

    <!-- **************************** -->

    <namedCache name="LuceneIndexesData">

        <clustering mode="replication">

            <stateRetrieval

                    fetchInMemoryState="true"

                    logFlushTimeout="30000"/>

            <sync

                    replTimeout="25000"/>

        </clustering>

        <loaders>

            <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="true">

                <properties>

                    <property name="location" value="/var/opt/fullTextStore"/>

                </properties>

            </loader>

        </loaders>

    </namedCache>

    <!-- ***************************** -->

    <!--  Cache to store Lucene locks  -->

    <!-- ***************************** -->

    <namedCache

            name="LuceneIndexesLocking">

        <clustering

                mode="replication">

            <stateRetrieval

                    fetchInMemoryState="true"

                    logFlushTimeout="30000"/>

            <sync

                    replTimeout="25000"/>

        </clustering>

    </namedCache>

</infinispan>[/code]

and lastly here is my indexing code:

[code]ftSession.createIndexer().batchSizeToLoadObjects(25).cacheMode(CacheMode.NORMAL).threadsToLoadObjects(5).threadsForIndexWriter(3).startAndWait();[/code]

I am using Hibernate Search 3.4.0, Lucene: 3.1.0, Infinispan 5.0.0.Final and jGroups: 2.12.1.3.

These are the approxamate statistics from my currently open files:

80% open files belong to java

5% belong to Google (chrome is my default browser)

5% belong to intelliJ

and 10% belong to various system tools (finder, etc)

Of the files opened by java

2-3% are various .jar files and other java files

95% belong to the cachestore

If anyone has any insight I would greatly appreciate it, I am new to infinispan and am fairly lost on this.

Thanks!