1 Reply Latest reply on Aug 30, 2011 5:13 AM by Sanne Grinovero

    Issues with implementing persistance with Hibernate Search

    Todd Underwood Newbie

      Hi Everyone,

       

       

      I'm having issues getting my index to be persistant when using Infinispan. When not trying to use persistance it works just fine however as soon as I start trying to use a FileCacheStore for persistance I start getting exceptions.

       

       

      It works for about 2 hours when this exception begins to be thrown:

       

       

      [code][2011-08-29 11:30:53,425] ERROR FileCacheStore.java:317 Hibernate Search: indexwriter-154 ) ISPN000063: Exception while saving bucket Bucket{entries={_4o.fdt|M|cnwk.foreman.model.SoftwareDownload=ImmortalCacheEntry{key=_4o.fdt|M|cnwk.foreman.model.SoftwareDownload, value=ImmortalCacheValue{value=FileMetadata{lastModified=1314642653425, size=32768}}}}, bucketId='1509281792'}

      java.io.FileNotFoundException: /var/opt/fullTextStore/LuceneIndexesMetadata/1509281792 (Too many open files)

        at java.io.RandomAccessFile.open(Native Method)

        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:216)

        at org.infinispan.loaders.file.FileCacheStore$BufferedFileSync.createChannel(FileCacheStore.java:494)

        at org.infinispan.loaders.file.FileCacheStore$BufferedFileSync.write(FileCacheStore.java:472)

        at org.infinispan.loaders.file.FileCacheStore.updateBucket(FileCacheStore.java:315)

        at org.infinispan.loaders.bucket.BucketBasedCacheStore.insertBucket(BucketBasedCacheStore.java:137)

        at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:94)

        at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:49)

        at org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:195)

        at org.infinispan.interceptors.CacheStoreInterceptor.visitPutKeyValueCommand(CacheStoreInterceptor.java:210)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)

        at org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:82)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)

        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:133)

        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:60)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)

        at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:214)

        at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:162)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)

        at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:114)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)

        at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:104)

        at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:64)

        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:60)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:119)

        at org.infinispan.interceptors.BatchingInterceptor.handleDefault(BatchingInterceptor.java:77)

        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:60)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

        at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:274)

        at org.infinispan.CacheImpl.put(CacheImpl.java:515)

        at org.infinispan.CacheSupport.put(CacheSupport.java:51)

        at org.infinispan.lucene.InfinispanIndexOutput.close(InfinispanIndexOutput.java:206)

        at org.apache.lucene.util.IOUtils.closeSafely(IOUtils.java:80)

        at org.apache.lucene.index.FieldsWriter.close(FieldsWriter.java:111)

        at org.apache.lucene.index.FieldsWriter.abort(FieldsWriter.java:121)

        at org.apache.lucene.index.StoredFieldsWriter.abort(StoredFieldsWriter.java:90)

        at org.apache.lucene.index.DocFieldProcessor.abort(DocFieldProcessor.java:71)

        at org.apache.lucene.index.DocumentsWriter.abort(DocumentsWriter.java:421)

        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:729)

        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2042)

        at org.hibernate.search.backend.impl.lucene.works.AddWorkDelegate.performWork(AddWorkDelegate.java:76)

        at org.hibernate.search.backend.impl.batchlucene.DirectoryProviderWorkspace.doWorkInSync(DirectoryProviderWorkspace.java:96)

        at org.hibernate.search.backend.impl.batchlucene.DirectoryProviderWorkspace$AsyncIndexRunnable.run(DirectoryProviderWorkspace.java:144)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

        at java.lang.Thread.run(Thread.java:680)[/code]

       

       

      I know that the error says I am opening too many files for my OS but I have already doubled the limit and it's still giving me the error.

       

       

      Here is my configuration:

       

       

      hibernate.cfg.xml:

      [code] <property name="hibernate.search.default.directory_provider">infinispan</property>

              <property name="hibernate.search.infinispan.configuration_resourcename">infinispan.cfg.xml</property>

      [/code]

       

       

      infinispan.cfg.xml

      [code]<?xml version="1.0" encoding="UTF-8"?>

      <infinispan

              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

              xsi:schemaLocation="urn:infinispan:config:4.2 http://www.infinispan.org/schemas/infinispan-config-4.2.xsd"

              xmlns="urn:infinispan:config:4.2">

       

       

          <!-- *************************** -->

          <!-- System-wide global settings -->

          <!-- *************************** -->

       

       

          <global>

       

       

              <!-- Duplicate domains are allowed so that multiple deployments with default configuration

                  of Hibernate Search applications work - if possible it would be better to use JNDI to share

                  the CacheManager across applications -->

              <globalJmxStatistics

                      enabled="true"

                      cacheManagerName="HibernateSearch"

                      allowDuplicateDomains="true"/>

       

       

              <!-- If the transport is omitted, there is no way to create distributed or clustered

                  caches. There is no added cost to defining a transport but not creating a cache that uses one,

                  since the transport is created and initialized lazily. -->

              <transport

                      clusterName="HibernateSearch-Infinispan-cluster"

                      distributedSyncTimeout="50000">

                  <!-- Note that the JGroups transport uses sensible defaults if no configuration

                      property is defined. See the JGroupsTransport javadocs for more flags -->

              </transport>

       

       

              <!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER, DONT_REGISTER.

                  Hibernate Search takes care to stop the CacheManager so registering is not needed -->

              <shutdown

                      hookBehavior="DONT_REGISTER"/>

       

       

          </global>

       

       

          <!-- *************************** -->

          <!-- Default "template" settings -->

          <!-- *************************** -->

       

       

          <default>

       

       

              <locking

                      lockAcquisitionTimeout="20000"

                      writeSkewCheck="false"

                      concurrencyLevel="500"

                      useLockStriping="false"/>

       

       

              <lazyDeserialization

                      enabled="false"/>

       

       

              <!-- Invocation batching is required for use with the Lucene Directory -->

              <invocationBatching

                      enabled="true"/>

       

       

              <!-- This element specifies that the cache is clustered. modes supported: distribution

                  (d), replication (r) or invalidation (i). Don't use invalidation to store Lucene indexes (as

                  with Hibernate Search DirectoryProvider). Replication is recommended for best performance of

                  Lucene indexes, but make sure you have enough memory to store the index in your heap.

                  Also distribution scales much better than replication on high number of nodes in the cluster. -->

              <clustering

                      mode="replication">

       

       

                  <!-- Prefer loading all data at startup than later -->

                  <stateRetrieval

                          timeout="60000"

                          logFlushTimeout="30000"

                          fetchInMemoryState="true"

                          alwaysProvideInMemoryState="true"/>

       

       

                  <!-- Network calls are synchronous by default -->

                  <sync

                          replTimeout="20000"/>

              </clustering>

       

       

              <jmxStatistics

                      enabled="true"/>

       

       

              <eviction

                      maxEntries="-1"

                      strategy="NONE"/>

       

       

              <expiration

                      maxIdle="-1"/>

       

       

          </default>

       

       

          <!-- ******************************************************************************* -->

          <!-- Individually configured "named" caches.                                         -->

          <!--                                                                                 -->

          <!-- While default configuration happens to be fine with similar settings across the -->

          <!-- three caches, they should generally be different in a production environment.   -->

          <!--                                                                                 -->

          <!-- Current settings could easily lead to OutOfMemory exception as a CacheStore     -->

          <!-- should be enabled, and maybe distribution is desired.                           -->

          <!-- ******************************************************************************* -->

       

       

          <!-- *************************************** -->

          <!--  Cache to store Lucene's file metadata  -->

          <!-- *************************************** -->

          <namedCache name="LuceneIndexesMetadata">

       

       

              <clustering mode="replication">

                  <stateRetrieval

                          fetchInMemoryState="true"

                          logFlushTimeout="30000"/>

                  <sync replTimeout="25000"/>

              </clustering>

              <loaders preload="true">

                  <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="true">

                      <properties>

                          <property name="location" value="/var/opt/fullTextStore"/>

                      </properties>

                  </loader>

              </loaders>

          </namedCache>

       

       

          <!-- **************************** -->

          <!--  Cache to store Lucene data  -->

          <!-- **************************** -->

          <namedCache name="LuceneIndexesData">

       

       

              <clustering mode="replication">

                  <stateRetrieval

                          fetchInMemoryState="true"

                          logFlushTimeout="30000"/>

                  <sync

                          replTimeout="25000"/>

              </clustering>

              <loaders>

                  <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="true">

                      <properties>

                          <property name="location" value="/var/opt/fullTextStore"/>

                      </properties>

                  </loader>

              </loaders>

          </namedCache>

       

       

          <!-- ***************************** -->

          <!--  Cache to store Lucene locks  -->

          <!-- ***************************** -->

          <namedCache

                  name="LuceneIndexesLocking">

              <clustering

                      mode="replication">

                  <stateRetrieval

                          fetchInMemoryState="true"

                          logFlushTimeout="30000"/>

                  <sync

                          replTimeout="25000"/>

              </clustering>

          </namedCache>

       

       

      </infinispan>[/code]

       

       

      and lastly here is my indexing code:

      [code]ftSession.createIndexer().batchSizeToLoadObjects(25).cacheMode(CacheMode.NORMAL).threadsToLoadObjects(5).threadsForIndexWriter(3).startAndWait();[/code]

       

       

      I am using Hibernate Search 3.4.0, Lucene: 3.1.0, Infinispan 5.0.0.Final and jGroups: 2.12.1.3.

       

       

      These are the approxamate statistics from my currently open files:

       

      80% open files belong to java

      5% belong to Google (chrome is my default browser)

      5% belong to intelliJ

      and 10% belong to various system tools (finder, etc)

       

      Of the files opened by java

      2-3% are various .jar files and other java files

      95% belong to the cachestore

       

      If anyone has any insight I would greatly appreciate it, I am new to infinispan and am fairly lost on this.

       

      Thanks!