1 2 Previous Next 15 Replies Latest reply: Apr 15, 2013 2:27 PM by kjordan2001 RSS

    Read Past EOF errors with Infinispan - Hibernate Search

    smnarayam Newbie

      We're currently attempting to setup a 10 node Infinispan distributed cluster, used as a HibernateSearch backend. Our stack is Hibernate 3.6.4, HibernateSearch 3.4.1, Infinispan 4.2.1 Final, JGroups 2.11.1 Final and Lucene 3.1.0. Our setup is a JMS Master / Slave configuration. We are using JNDI to lookup the Infinispan CacheManager. Following the Hibernate Search recommendation, we have the Metadata & Lock Caches replicated, and we are distributing the Data cache with hash = 3.

       

      Under low usage, it holds up fine. However, whenever we start to stress the system, we start seeing a lot of 'read past EOF' errors on the master during index writes.

       

      We have tried various options, like switching cache store to Jdbm (based on the last comment in this post - https://forum.hibernate.org/viewtopic.php?p=2449501), async write-behind to the cache store, exclusive_index_use set to false, and with all 3 caches distributed. Switching the Metadata / Lock caches to replicated and using exclusive_index_use gave us positive improvements (got us over "java.io.IOException: No sub-file with id .fnm found (files: [.fdt, .fdx])" exceptions), but we continue to see these ''read past eof' errors.

       

      We believe we must be misconfigured somewhere. Can anyone recommend a fix?

       

      The errors we see are -
      04-02-2012 20:45:07 ERROR Hibernate Search: Directory writer-1 impl.LogErrorHandler: Exception occurred java.io.IOException: read past EOF

      java.io.IOException: read past EOF

          at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:207)

          at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:39)

          at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:92)

          at org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:181)

          at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:339)

          at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71)

          at org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:118)

          at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:578)

          at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:684)

          at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:659)

          at org.apache.lucene.index.BufferedDeletes.applyDeletes(BufferedDeletes.java:283)

          at org.apache.lucene.index.BufferedDeletes.applyDeletes(BufferedDeletes.java:191)

          at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3358)

          at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3296)

          at org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:3159)

          at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3232)

          at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3214)

          at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3198)

          at org.hibernate.search.backend.Workspace.commitIndexWriter(Workspace.java:220)

          at org.hibernate.search.backend.impl.lucene.PerDPQueueProcessor.run(PerDPQueueProcessor.java:109)

          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)

          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)

          at java.util.concurrent.FutureTask.run(FutureTask.java:138)

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

          at java.lang.Thread.run(Thread.java:662)


      Another variation we see is -

      04-02-2012 14:37:43 ERROR Lucene Merge Thread #8 impl.LogErrorHandler: Exception occurred org.apache.lucene.index.MergePolicy$MergeException: java.io.IOExcept

      org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: read past EOF

          at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:517)

          at org.hibernate.search.backend.impl.lucene.overrides.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:49)

          at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)

      Caused by: java.io.IOException: read past EOF

          at org.apache.lucene.store.BufferedIndexInput.readBytes(BufferedIndexInput.java:138)

          at org.apache.lucene.index.SegmentReader$Norm.bytes(SegmentReader.java:409)

          at org.apache.lucene.index.SegmentReader$Norm.bytes(SegmentReader.java:404)

          at org.apache.lucene.index.SegmentReader.norms(SegmentReader.java:1084)

          at org.apache.lucene.index.SegmentMerger.mergeNorms(SegmentMerger.java:636)

          at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:112)

          at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3938)

          at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3614)

          at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)

          at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)

       

      Our Infinispan configuration is -

       

      <?xml version="1.0" encoding="UTF-8"?>

      <infinispan

          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

          xsi:schemaLocation="urn:infinispan:config:4.2 http://www.infinispan.org/schemas/infinispan-config-4.2.xsd"

          xmlns="urn:infinispan:config:4.2">

         <!-- *************************** -->

          <!-- System-wide global settings -->

          <!-- *************************** -->

            <global>

                <!-- Duplicate domains are allowed so that multiple deployments with default configuration

                  of Hibernate Search applications work - if possible it would be better to use JNDI to share

                  the CacheManager across applications -->

              <globalJmxStatistics

                  enabled="true"

                  cacheManagerName="HibernateSearch"

                  allowDuplicateDomains="true" />

       

              <!-- If the transport is omitted, there is no way to create distributed or clustered

                  caches. There is no added cost to defining a transport but not creating a cache that uses one,

                  since the transport is created and initialized lazily. -->

              <transport

                  clusterName="HibernateSearch-Infinispan-cluster-MT"

                  distributedSyncTimeout="90000">

                  <properties>

                      <property name="configurationFile" value="infinispan-tcp.cfg.xml" />

                  </properties>

                  <!-- Note that the JGroups transport uses sensible defaults if no configuration

                      property is defined. See the JGroupsTransport javadocs for more flags -->

              </transport>

       

              <!-- Used to register JVM shutdown hooks. hookBehavior: DEFAULT, REGISTER, DONT_REGISTER.

                  Hibernate Search takes care to stop the CacheManager so registering is not needed -->

              <shutdown

                  hookBehavior="DONT_REGISTER" />

            </global>

       

          <!-- *************************** -->

          <!-- Default "template" settings -->

          <!-- *************************** -->

            <default>

                <locking

                  lockAcquisitionTimeout="20000"

                  writeSkewCheck="false"

                  concurrencyLevel="5000"

                  useLockStriping="false" />

       

              <lazyDeserialization

                  enabled="false" />

       

              <!-- Invocation batching is required for use with the Lucene Directory -->

              <invocationBatching

                  enabled="true" />

       

              <!-- This element specifies that the cache is clustered. modes supported: distribution

                  (d), replication (r) or invalidation (i). Don't use invalidation to store Lucene indexes (as

                  with Hibernate Search DirectoryProvider). Replication is recommended for best performance of

                  Lucene indexes, but make sure you have enough memory to store the index in your heap.

                  Also distribution scales much better than replication on high number of nodes in the cluster. -->

              <clustering

                  mode="d">

                  <sync replTimeout="90000" />

                  <l1 enabled="false" />

              </clustering>

       

              <jmxStatistics

                  enabled="true" />

       

              <eviction

                  maxEntries="-1"

                  strategy="NONE" />

       

              <expiration

                  maxIdle="-1" />

          </default>

       

          <!-- ******************************************************************************* -->

          <!-- Individually configured "named" caches.                                         -->

          <!--                                                                                 -->

          <!-- While default configuration happens to be fine with similar settings across the -->

          <!-- three caches, they should generally be different in a production environment.   -->

          <!--                                                                                 -->

          <!-- Current settings could easily lead to OutOfMemory exception as a CacheStore     -->

          <!-- should be enabled, and maybe distribution is desired.                           -->

          <!-- ******************************************************************************* -->

       

          <!-- *************************************** -->

          <!--  Cache to store Lucene's file metadata  -->

          <!-- *************************************** -->

          <namedCache

              name="LuceneIndexesMetadata">

              <clustering

                  mode="replication">

                  <stateRetrieval fetchInMemoryState="true" logFlushTimeout="30000" />

                  <sync replTimeout="90000" />

                  <l1 enabled="false" />

              </clustering>

              <loaders shared="true" preload="true">

                  <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="false" ignoreModifications="false" purgeOnStartup="false">

                      <properties>

                          <property name="location" value="/usr/local/tc/.index/metadata" />

                      </properties>

                  </loader>

              </loaders>

          </namedCache>

       

          <!-- **************************** -->

          <!--  Cache to store Lucene data  -->

          <!-- **************************** -->

          <namedCache

              name="LuceneIndexesData">

              <clustering

                  mode="d">

                  <hash numOwners="3" />

                  <sync replTimeout="90000" />

                  <l1 enabled="false" />

              </clustering>

              <loaders shared="true" preload="true">

                  <loader class="org.infinispan.loaders.file.FileCacheStore" fetchPersistentState="false" ignoreModifications="false" purgeOnStartup="false">

                      <properties>

                          <property name="location" value="/usr/local/tc/.index/data" />

                      </properties>

                  </loader>

              </loaders>

          </namedCache>

       

          <!-- ***************************** -->

          <!--  Cache to store Lucene locks  -->

          <!-- ***************************** -->

          <namedCache

              name="LuceneIndexesLocking">

              <clustering

                  mode="replication">

                  <stateRetrieval fetchInMemoryState="true" logFlushTimeout="30000" />

                  <sync replTimeout="90000" />

                  <l1 enabled="false" />

              </clustering>

          </namedCache>

         

      </infinispan>

       

      Our hibernate configuration on the master -

       

              <property name="hibernate.search.default.directory_provider">infinispan</property>

              <property name="hibernate.search.infinispan.cachemanager_jndiname">InfinispanCacheManager</property>

              <property name="hibernate.search.infinispan.chunk_size">40960</property>

              <property name="hibernate.search.default.exclusive_index_use">true</property>

       

      Hibernate configuration on the slaves -

       

              <property name="hibernate.search.default.directory_provider">infinispan</property>

              <property name="hibernate.search.infinispan.cachemanager_jndiname">InfinispanCacheManager</property>

              <property name="hibernate.search.infinispan.chunk_size">40960</property>

              <property name="hibernate.search.worker.backend">jms</property>

              <property name="hibernate.search.worker.jndi.class">org.apache.naming.java.javaURLContextFactory</property>               

              <property name="hibernate.search.worker.jms.connection_factory">ConnectionFactory</property>

              <property name="hibernate.search.worker.jms.queue">/queue/hibernateSearchQueue</property>


        • 1. Re: Read Past EOF errors with Infinispan - Hibernate Search
          Sanne Grinovero Master

          Hi,

          this was being caused by subtle race conditions and small bugs in both Lucene and Infinispan, but it's a long time ago, these versions are quite old; it would require several backports in both projects to fix, which is not practical: I'd advise to upgrade.

           

          I've got very good results from testing the latest stable versions:

          • Apache Lucene 3.5
          • Infinispan 5.1.3.FINAL
          • Hibernate Search 4.1.0.Final

           

          You would get Hibernate ORM 4.1.2.Final as well, which includes a lot of improvements; I realize it's quite some work to update it all, but there are many other reasons for this to be a good idea.

           

          If you really can't upgrade, or can't now, you can workaround the problem by using a chunk_size
          which is large enough to contain the full segment. It will require some experimentation to find a good value for you, as if you set it way too high you'll be consuming more memory than necessary.

          • 2. Re: Read Past EOF errors with Infinispan - Hibernate Search
            smnarayam Newbie

            Hi Sanne,

             

            Thanks for your fast response.

             

            Unfortunately, we are not in a position to upgrade right now. We'll experiment with the chunk_size, and report back our findings.

             

            Thanks.

            • 3. Re: Read Past EOF errors with Infinispan - Hibernate Search
              Sanne Grinovero Master

              thanks, please keep me updated.

              • 4. Re: Read Past EOF errors with Infinispan - Hibernate Search
                Francisco Lozano Newbie

                I am having a similar problem but running with the latest versions (Lucene 3.5.0, Infinispan 5.1.2/5.1.3 - tried both). No hibernate search, just pure lucene usage here.

                 

                My configuration for tests is:

                 

                <?xml version="1.0" encoding="UTF-8"?>

                <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                          xsi:schemaLocation="urn:infinispan:config:5.1 http://www.infinispan.org/schemas/infinispan-config-5.1.xsd"

                          xmlns="urn:infinispan:config:5.1">

                          <global>

                          </global>

                          <default>

                                    <locking lockAcquisitionTimeout="20000" />

                                    <invocationBatching enabled="true" />

                                    <eviction maxEntries="-1" strategy="NONE" />

                                    <expiration maxIdle="-1" />

                          </default>

                 

                 

                          <namedCache name="indexMetadata">

                                    <loaders passivation="false" shared="false" preload="false">

                                              <loader class="org.infinispan.loaders.jdbm.JdbmCacheStore"

                                                        fetchPersistentState="true" purgeSynchronously="false"

                                                        purgeOnStartup="false">

                                                        <async enabled="true" threadPoolSize="10"/>

                                                        <properties>

                                                                  <property name="location" value="/tmp/infinispan/index_metadata" />

                                                        </properties>

                                              </loader>

                                    </loaders>

                          </namedCache>

                 

                 

                          <namedCache name="indexChunks">

                                    <eviction wakeUpInterval="5000" maxEntries="100000" strategy="LIRS" />

                                    <loaders shared="false" preload="false">

                                              <loader class="org.infinispan.loaders.jdbm.JdbmCacheStore"

                                                        fetchPersistentState="true" purgeSynchronously="false"

                                                        purgeOnStartup="false">

                                                        <async enabled="true" threadPoolSize="10"/>

                                                        <properties>

                                                                  <property name="location" value="/tmp/infinispan/index_chunks" />

                                                        </properties>

                                              </loader>

                                    </loaders>

                          </namedCache>

                 

                 

                          <namedCache name="indexDistLocks">

                          </namedCache>

                 

                 

                </infinispan>

                 

                I had to add the async option because without it the write performance was so slow that it was totally unusable.

                 

                EDIT: Now I'm able to reproduce everywhere. If I remove the async clause it doesn't reproduce.

                 

                The 1st time I open an index in a set of tests, it works well. Then I shutdown everything (whole Spring context), and start everything for the next test, and when I try to get the index I get:

                 

                java.io.IOException: Read past EOF

                at org.infinispan.lucene.SingleChunkIndexInput.readByte(SingleChunkIndexInput.java:76) ~[infinispan-lucene-directory-5.1.3.FINAL.jar:5.1.3.FINAL]

                at org.apache.lucene.store.ChecksumIndexInput.readByte(ChecksumIndexInput.java:41) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

                at org.apache.lucene.store.DataInput.readInt(DataInput.java:84) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

                at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:272) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

                at org.apache.lucene.index.IndexFileDeleter.<init>(IndexFileDeleter.java:182) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

                at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1178) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

                ...

                 

                Or similar stuff, like:

                java.io.FileNotFoundException: Error loading medatada for index file: _6.fnm|M|idx12345678

                          at org.infinispan.lucene.InfinispanDirectory.openInput(InfinispanDirectory.java:282) ~[infinispan-lucene-directory-5.1.3.FINAL.jar:5.1.3.FINAL]

                          at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:74) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

                          at org.apache.lucene.index.IndexWriter.getFieldInfos(IndexWriter.java:1222) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

                          at org.apache.lucene.index.IndexWriter.getCurrentFieldInfos(IndexWriter.java:1238) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

                          at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1171) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

                 

                I have tried with BerkeleyDB backend and I have the same result.

                 

                I setup the caches in Spring with:

                 

                <bean factory-bean="infinispanDefaultCacheManager" factory-method="getCache"

                id="infinispanIndexMetadataCache">

                <constructor-arg value="indexMetadata" />

                </bean>

                 

                 

                <bean factory-bean="infinispanDefaultCacheManager" factory-method="getCache"

                id="infinispanIndexChunksCache">

                <constructor-arg value="indexChunks" />

                </bean>

                 

                 

                <bean factory-bean="infinispanDefaultCacheManager" factory-method="getCache"

                id="infinispanIndexDistLocksCache">

                <constructor-arg value="indexDistLocks" />

                </bean>

                 

                 

                <bean factory-bean="infinispanDefaultCacheManager" factory-method="getCache"

                id="infinispanIndexDatabaseCache">

                <constructor-arg value="indexDatabase" />

                </bean>

                 

                 

                <bean class="org.infinispan.manager.DefaultCacheManager" id="infinispanDefaultCacheManager"

                init-method="start" destroy-method="stop">

                <constructor-arg value="infinispan-test-bdb.xml" />

                </bean>

                 

                Shall I do some kind of shutdown hook or anything like that?

                • 5. Re: Read Past EOF errors with Infinispan - Hibernate Search
                  Francisco Lozano Newbie

                  I have checked and added <eviction to every <namedCache>, and it happens something interesting:

                  - If I put a very small maxEntries value (eg: 5), it reproduces immediately. If I put a very large maxEntries value (eg: 100000) it does not reproduce - which makes me think that the problem is that the async cache store is not properly "flushed" before shutdown. I have added the init-method="start" destroy-method="stop" to the DefaultCacheManager definition, but maybe I'm missing something :?

                   

                  EDIT:

                  I have fixed the problem above. After each test, the CacheManager is properly closed and I get in the logs:

                  INFO o.i.eviction.PassivationManagerImpl.passivateAll - ISPN000029: Passivating all entries to disk

                  INFO o.i.eviction.PassivationManagerImpl.passivateAll - ISPN000030: Passivated 0 entries in 2 milliseconds

                  INFO o.i.eviction.PassivationManagerImpl.passivateAll - ISPN000029: Passivating all entries to disk

                  INFO o.i.eviction.PassivationManagerImpl.passivateAll - ISPN000030: Passivated 4 entries in 1 milliseconds

                  INFO o.i.eviction.PassivationManagerImpl.passivateAll - ISPN000029: Passivating all entries to disk

                  INFO o.i.eviction.PassivationManagerImpl.passivateAll - ISPN000030: Passivated 4 entries in 0 milliseconds


                  But this was not the cause of the problem, I still have messages like:

                  java.io.FileNotFoundException: Error loading medatada for index file: segments_1|M|idx12345678

                            at org.infinispan.lucene.InfinispanDirectory.openInput(InfinispanDirectory.java:282) ~[infinispan-lucene-directory-5.1.3.FINAL.jar:5.1.3.FINAL]

                            at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:265) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

                            at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:363) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

                            at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:754) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

                            at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:593) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

                            at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:359) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

                            at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1148) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

                   

                   

                  And:

                   

                   

                  java.io.FileNotFoundException: Error loading medatada for index file: _0.fnm|M|idx12345678

                            at org.infinispan.lucene.InfinispanDirectory.openInput(InfinispanDirectory.java:282) ~[infinispan-lucene-directory-5.1.3.FINAL.jar:5.1.3.FINAL]

                            at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:74) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

                            at org.apache.lucene.index.IndexWriter.getFieldInfos(IndexWriter.java:1222) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

                            at org.apache.lucene.index.IndexWriter.getCurrentFieldInfos(IndexWriter.java:1238) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51]

                            at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1171) ~[lucene-core-3.5.0.jar:3.5.0 1204988 - simon - 2011-11-22 14:46:51].

                   

                   

                  And a new one:

                   

                  INFO o.i.eviction.PassivationManagerImpl.passivateAll - ISPN000029: Passivating all entries to disk

                  INFO o.i.eviction.PassivationManagerImpl.passivateAll - ISPN000030: Passivated 0 entries in 0 milliseconds

                  INFO o.i.eviction.PassivationManagerImpl.passivateAll - ISPN000029: Passivating all entries to disk

                  INFO o.i.eviction.PassivationManagerImpl.passivateAll - ISPN000030: Passivated 361 entries in 0 milliseconds

                  ERROR o.i.loaders.decorators.AsyncStore.run - ISPN000051: Unexpected error

                  org.infinispan.CacheException: Unable to acquire lock on update map

                            at org.infinispan.loaders.decorators.AsyncStore.acquireLock(AsyncStore.java:293) ~[infinispan-core-5.1.3.FINAL.jar:5.1.3.FINAL]

                            at org.infinispan.loaders.decorators.AsyncStore.access$900(AsyncStore.java:86) ~[infinispan-core-5.1.3.FINAL.jar:5.1.3.FINAL]

                            at org.infinispan.loaders.decorators.AsyncStore$AsyncProcessor.innerRun(AsyncStore.java:336) ~[infinispan-core-5.1.3.FINAL.jar:5.1.3.FINAL]

                            at org.infinispan.loaders.decorators.AsyncStore$AsyncProcessor.run(AsyncStore.java:312) ~[infinispan-core-5.1.3.FINAL.jar:5.1.3.FINAL]

                            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) ~[na:1.6.0_31]

                            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) ~[na:1.6.0_31]

                            at java.lang.Thread.run(Thread.java:662) ~[na:1.6.0_31]

                   

                   

                  Right now my infinispan configuration file looks like (single-node setup for tests):

                   

                  <?xml version="1.0" encoding="UTF-8"?>

                  <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                            xsi:schemaLocation="urn:infinispan:config:5.1 http://www.infinispan.org/schemas/infinispan-config-5.1.xsd"

                            xmlns="urn:infinispan:config:5.1">

                            <global>

                            </global>

                            <default>

                                      <locking lockAcquisitionTimeout="20000" />

                                      <invocationBatching enabled="true" />

                                      <eviction maxEntries="-1" strategy="NONE" />

                                      <expiration maxIdle="-1" />

                            </default>

                   

                   

                            <namedCache name="indexMetadata">

                                      <eviction wakeUpInterval="5000" maxEntries="1000" strategy="LRU" />

                                      <loaders shared="false" preload="true" passivation="true">

                                                <loader class="org.infinispan.loaders.bdbje.BdbjeCacheStore"

                                                          fetchPersistentState="true" purgeSynchronously="false"

                                                          purgeOnStartup="false">

                                                          <async enabled="true" threadPoolSize="10" />

                                                          <properties>

                                                                    <property name="location" value="/tmp/infinispan/index_metadata" />

                                                          </properties>

                                                </loader>

                                      </loaders>

                            </namedCache>

                   

                   

                            <namedCache name="indexChunks" >

                                      <eviction wakeUpInterval="5000" maxEntries="1000" strategy="LRU" />

                                      <loaders shared="false" preload="false" passivation="true">

                                                <loader class="org.infinispan.loaders.bdbje.BdbjeCacheStore"

                                                          fetchPersistentState="true" purgeSynchronously="false"

                                                          purgeOnStartup="false">

                                                          <async enabled="true" threadPoolSize="10" />

                                                          <properties>

                                                                    <property name="location" value="/tmp/infinispan/index_chunks" />

                                                          </properties>

                                                </loader>

                                      </loaders>

                            </namedCache>

                   

                   

                            <namedCache name="indexDistLocks">

                            </namedCache>

                   

                   

                  </infinispan>

                  • 6. Re: Read Past EOF errors with Infinispan - Hibernate Search
                    Francisco Lozano Newbie

                    Sorry to update this thread so frequently, but I hope it's useful for someone else...

                     

                    I just changed my "purgeSynchronously" to true for every cache, and added flushLockTimeout="60000" to every <async /> clause and the problem seems to be solved it still reproduces - or, at least, does not reproduce in my tests.

                    (Actually the problem seems only solved with BerkeleyDB, but with Jdbm it still fails until I disable also the "preload" of the index metadata - I'll open a new thread for that to avoid confusion).

                    • 7. Re: Read Past EOF errors with Infinispan - Hibernate Search
                      smnarayam Newbie

                      Hi Sanne,

                       

                      We were unable to get past the 'read past eof' errors with a distributed cluster. Finally, we reverted back to a replicated setup. With replication, we are able to run the same stress tests without any errors.

                       

                      For now, we are planning to move ahead with replication. In the coming months, we plan to upgrade Infinispan, Lucene etc, and we will try distribution again after the upgrade.

                       

                      Thanks.

                      • 8. Re: Read Past EOF errors with Infinispan - Hibernate Search
                        Sanne Grinovero Master

                        right that seems a good choice. Please let me know if you have any problem after upgrading!

                        Sanne

                        • 9. Re: Read Past EOF errors with Infinispan - Hibernate Search
                          Francisco Lozano Newbie

                          Hi Sanne,

                           

                          Actually I am reproducing the same issue with 5.1.3 / lucene 3.5.0. I created a test in git for it: https://github.com/flozano/infinispan-lucene-eof-problem. May I ask you to take a look and tell me if I'm doing something wrong or if I'm hitting a bug actually?

                          • 10. Re: Read Past EOF errors with Infinispan - Hibernate Search
                            Sanne Grinovero Master

                            Hi Francisco,

                            thanks for the tests, that's very interesting and I'm glad to look into it in the weekend, unfortunately I can't before that as I'm away.

                            • 11. Re: Read Past EOF errors with Infinispan - Hibernate Search
                              Francisco Lozano Newbie

                              That'd be great Sanne, thanks a lot!

                              • 12. Re: Read Past EOF errors with Infinispan - Hibernate Search
                                Sanne Grinovero Master

                                Hi Francisco,

                                sorry for the delay, I've been very busy and then.. I forgot.

                                 

                                I've opened https://issues.jboss.org/browse/ISPN-2046 to track this directly.

                                 

                                Your test is amazing: it has been very helpfull to reproduce it. The good news is it helped be to immediately spot a just-introduced problem ISPN-2049, but then I realized the branch you where testing wasn't affected by this.

                                 

                                The main problem, as you had guessed already, lies with async stores:  ISPN-1174.

                                Until that is solved, it's not safe to use an async CacheLoader with data strictures which can't deal with it; Lucene definitely not.

                                • 13. Re: Read Past EOF errors with Infinispan - Hibernate Search
                                  Francisco Lozano Newbie

                                  Hi Sanne,

                                   

                                  Thanks a lot for taking care of this. I also lost track of this problem it was an experiment, not a project-with-deadline.

                                   

                                  I see the JIRA is scheduled for fix in 5.2.0.Final - that would be great, looks like a very complex issue...

                                  • 14. Re: Read Past EOF errors with Infinispan - Hibernate Search
                                    kjordan2001 Newbie

                                    I get a similar error with and without async turned on in 5.2.4 when I try to start up a 3rd node:

                                    java.io.IOException: Read past EOF

                                            at org.infinispan.lucene.SingleChunkIndexInput.readByte(SingleChunkIndexInput.java:77)

                                            at org.apache.lucene.store.DataInput.readVInt(DataInput.java:107)

                                            at org.apache.lucene.index.FieldInfos.read(FieldInfos.java:314)

                                            at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:74)

                                            at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:80)

                                            at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116)

                                            at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:94)

                                            at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:105)

                                            at org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:27)

                                            at org.apache.lucene.index.DirectoryReader$1.doBody(DirectoryReader.java:78)

                                            at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:709)

                                            at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:72)

                                            at org.apache.lucene.index.IndexReader.open(IndexReader.java:256)

                                            at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.readerFactory(SharingBufferR

                                    eaderProvider.java:143)

                                            at org.hibernate.search.indexes.impl.SharingBufferReaderProvider$PerDirectoryLatestReader.<in

                                    it>(SharingBufferReaderProvider.java:217)

                                            at org.hibernate.search.indexes.impl.SharingBufferReaderProvider.createReader(SharingBufferRe

                                    aderProvider.java:120)

                                     

                                    Sometimes I also get:

                                    Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.infinispan.lucene.locking.BaseLuceneLock@1caf7e42

                                            at org.apache.lucene.store.Lock.obtain(Lock.java:84)

                                            at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1098)

                                            at org.hibernate.search.store.impl.DirectoryProviderHelper.initializeIndexIfNeeded(DirectoryProviderHelper.java:157)

                                     

                                    All the nodes seem to be configured correctly as I can start 2 up but no matter which one the 3rd one is, it gets one of those errors.

                                     

                                    I have it configured with JGroups JDBC and a JDBC CacheStore.  So far Infinispan has been pretty aggravating for me as I also had a very hard time getting it set up due to errors that shouldn't have happened since I used an equivalent configuration and then it worked fine (and never got it working with S3 as a store): https://community.jboss.org/thread/222803

                                     

                                    For the second error:

                                    2013-04-11 16:52:12,501 [org.infinispan.lucene.locking.BaseLuceneLock] TRACE: Lock: write.lock not aquired for index: <myindex>, was taken already.

                                     

                                    Which doesn't make any sense since no writing was being done to the index at that point in time.  I even cleared out the backing store for it to make sure no corruption occured before starting the nodes.  Node 1 recreated the index.  Then I started Node 2 and that went fine.  But then Node 3 gives an error always.

                                    1 2 Previous Next