6 Replies Latest reply on Apr 5, 2013 10:17 PM by clementp

    Configuration for an All PostgreSQL repository storage

    bwallis42
    bwallis42 Apr 4, 2013 11:06 PM

      I'm having some difficulty getting this to work.

       

      This may be related to the issues highlighted in this thread by Horia to do with indexing caches but I think there is also an underlying problem (ISPN-604) that is going to make this impossible to get working in a production environment.

       

      I am getting concerned that the timeframe to a usable transactional modeshape/infinispan/JDBC database combination is going to be too far into the future for my project. I would really like to know if what I am trying to do is achievable and if so what configuration I should be using.

       

      What my eventual configuration is meant to achieve is

       

      1. a small cluster of servers (2-3) running my application which is storing all document data and metadata into a JCR repository that is replicated across all the nodes.
      2. All data written to the repository is immediately persisted to secondary storage (ie: write through caches)
      3. Data written to a repository on one node is immediately available on the other nodes
      4. As I have other data stores used in the application (Hibernate/JPA and HornetQ at least) we need to use XA transactions. We absolutely have to maintain data consistency in the face of failures, this is a key requirement.
      5. Storage of 50 million documents with a size around 20TB in all (this is what our current system can do on a single server)

       

      As far as I understand the persistence options for infinispan, the only choice for the cache loader that supports transactions is to use a relational database. The choices seem to be the various JDBC based loaders and a decent relational DB underneath (PostgreSQL is our preferred DB at the moment, oracle an option) or the BerkeleyDB one (but that has limitations in the replication area and it's datasheet does not mention XA support).

       

      There is a comment in the infinispan documentation at the end of the page on cache loaders and stores that is a concern

       

      When a cache is transactional and a cache loader is present, the cache loader won't be enlisted in the transaction in which the cache is part. That means that it is possible to have inconsistencies at cache loader level: the transaction to succeed applying the in-memory state but (partially) fail applying the changes to the store. Manual recovery would not work with caches stores.

       

      I'm assuming that this is a reference to the long standing bug ISPN-604 that has been a concern to me for quite a while. Currently scheduled for a fix in infinispan 5.3.0 which is unlikely to be in an EAP release of JBoss anytime soon. (I actually find it hard to believe that this is not a critical bug rather than a feature request).

       

      My first step is to get the persistence of the repository working on a single server to a PostgreSQL database for both the repository data and indexes using write through JDBC stores for the caches *with transactions enabled*. Thus far I have failed.

       

      My current configuration is (using NON_XA, but similar results with FULL_XA)

       

       
      {code}
              <subsystem xmlns="urn:jboss:domain:infinispan:1.4">
       
      ... hibernate stuff
       
       
                  <cache-container name="modeshapeWorkspaceCache" default-cache="workspaceCache">
                      <local-cache name="workspaceCache">
                          <transaction mode="NON_XA"/>
                          <eviction strategy="LRU" max-entries="100"/>
                          <expiration max-idle="5000" lifespan="10000" interval="1000"/>
                      </local-cache>
                      <local-cache name="testRepository/default">
                          <transaction mode="NON_XA"/>
                          <eviction strategy="LRU" max-entries="100"/>
                          <expiration max-idle="5000" lifespan="10000" interval="1000"/>
                      </local-cache>
                  </cache-container>
                  <cache-container name="modeshapeCache">
                      <local-cache name="testRepositoryCache" batching="false">
                          <transaction mode="NON_XA"/>
                          <eviction strategy="LIRS" max-entries="512"/>
                          <string-keyed-jdbc-store datasource="java:jboss/datasources/JcrDataDS" passivation="false" purge="false">
                              <property name="databaseType">
                                  postgres
                              </property>
                              <property name="createTableOnStart">
                                  true
                              </property>
                              <string-keyed-table prefix="stringbased">
                                  <id-column name="id" type="VARCHAR(200)"/>
                                  <data-column name="datum" type="BYTEA"/>
                                  <timestamp-column name="version" type="BIGINT"/>
                              </string-keyed-table>
                          </string-keyed-jdbc-store>
                      </local-cache>
                  </cache-container>
                  <cache-container name="modeshapeBinaryCache">
                      <local-cache name="testRepository_binary_data" batching="false">
                          <transaction mode="NON_XA"/>
                          <eviction strategy="LIRS" max-entries="5"/>
                          <binary-keyed-jdbc-store datasource="java:jboss/datasources/JcrDataDS" passivation="false" purge="false">
                              <property name="databaseType">
                                  postgres
                              </property>
                              <property name="createTableOnStart">
                                  true
                              </property>
                              <binary-keyed-table prefix="binarybased">
                                  <id-column name="id" type="VARCHAR(200)"/>
                                  <data-column name="datum" type="BYTEA"/>
                                  <timestamp-column name="version" type="BIGINT"/>
                              </binary-keyed-table>
                          </binary-keyed-jdbc-store>
                      </local-cache>
                      <local-cache name="testRepository_binary_metadata" batching="false">
                          <transaction mode="NON_XA"/>
                          <eviction strategy="LIRS" max-entries="2000"/>
                          <binary-keyed-jdbc-store datasource="java:jboss/datasources/JcrDataDS" passivation="false" purge="false">
                              <property name="databaseType">
                                  postgres
                              </property>
                              <property name="createTableOnStart">
                                  true
                              </property>
                              <binary-keyed-table prefix="binarybased">
                                  <id-column name="id" type="VARCHAR(200)"/>
                                  <data-column name="datum" type="BYTEA"/>
                                  <timestamp-column name="version" type="BIGINT"/>
                              </binary-keyed-table>
                          </binary-keyed-jdbc-store>
                      </local-cache>
                  </cache-container>
                  <cache-container name="modeshapeIndexCache">
                      <local-cache name="testRepository_index_locks" batching="false">
                          <transaction mode="NON_XA"/>
                          <binary-keyed-jdbc-store datasource="java:jboss/datasources/JcrDataDS" passivation="false" purge="false">
                              <property name="databaseType">
                                  postgres
                              </property>
                              <property name="createTableOnStart">
                                  true
                              </property>
                              <binary-keyed-table prefix="binarybased">
                                  <id-column name="id" type="VARCHAR(200)"/>
                                  <data-column name="datum" type="BYTEA"/>
                                  <timestamp-column name="version" type="BIGINT"/>
                              </binary-keyed-table>
                          </binary-keyed-jdbc-store>
                      </local-cache>
                      <local-cache name="testRepository_index_data" batching="false">
                          <transaction mode="NON_XA"/>
                          <eviction strategy="LIRS" max-entries="2000"/>
                          <binary-keyed-jdbc-store datasource="java:jboss/datasources/JcrDataDS" passivation="false" purge="false">
                              <property name="databaseType">
                                  postgres
                              </property>
                              <property name="createTableOnStart">
                                  true
                              </property>
                              <binary-keyed-table prefix="binarybased">
                                  <id-column name="id" type="VARCHAR(200)"/>
                                  <data-column name="datum" type="BYTEA"/>
                                  <timestamp-column name="version" type="BIGINT"/>
                              </binary-keyed-table>
                          </binary-keyed-jdbc-store>
                      </local-cache>
                      <local-cache name="testRepository_index_metadata" batching="false">
                          <transaction mode="NON_XA"/>
                          <eviction strategy="LIRS" max-entries="2000"/>
                          <binary-keyed-jdbc-store datasource="java:jboss/datasources/JcrDataDS" passivation="false" purge="false">
                              <property name="databaseType">
                                  postgres
                              </property>
                              <property name="createTableOnStart">
                                  true
                              </property>
                              <binary-keyed-table prefix="binarybased">
                                  <id-column name="id" type="VARCHAR(200)"/>
                                  <data-column name="datum" type="BYTEA"/>
                                  <timestamp-column name="version" type="BIGINT"/>
                              </binary-keyed-table>
                          </binary-keyed-jdbc-store>
                      </local-cache>
                  </cache-container>
              </subsystem>
              <subsystem xmlns="urn:jboss:domain:modeshape:1.0">
                  <repository name="testRepository" cache-name="testRepositoryCache" cache-container="modeshapeCache">
                      <workspaces cache-container="modeshapeWorkspaceCache">
                          <workspace name="default"/>
                      </workspaces>
                      <indexing systemContentMode="SYNC"/>
                      <cache-index-storage data-cache-name="testRepository_index_data" metadata-cache-name="testRepository_index_metadata" lock-cache-name="testRepository_index_locks" cache-container="modeshapeIndexCache"/>
                      <cache-binary-storage min-value-size="0" min-string-size="256" data-cache-name="testRepository_binary_data" metadata-cache-name="testRepository_binary_metadata" cache-container="modeshapeBinaryCache"/>
                  </repository>
                  <webapp name="modeshape-rest.war"/>
                  <webapp name="modeshape-cmis.war"/>
                  <webapp name="modeshape-webdav.war"/>
              </subsystem>
      {code}
       
       

       

      This just does not work at all. As soon as I access the repository via the REST api (http://localhost:8080/modeshape-rest/testRepository) I start getting errors like the following

       

      {code}
      13:20:59,923 ERROR [org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory] (Hibernate Search: Index updates queue processor for index nodeinfo-1) ISPN008018: Sql failure retrieving connection from datasource: java.sql.SQLException: javax.resource.ResourceException: IJ000460: Error checking for a transaction
                at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:147)
                at org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory.getConnection(ManagedConnectionFactory.java:93) [infinispan-cachestore-jdbc-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.loaders.jdbc.binary.JdbcBinaryCacheStore.loadBucket(JdbcBinaryCacheStore.java:277) [infinispan-cachestore-jdbc-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.loaders.bucket.BucketBasedCacheStore.removeLockSafe(BucketBasedCacheStore.java:106) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.loaders.bucket.BucketBasedCacheStore.removeLockSafe(BucketBasedCacheStore.java:49) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.loaders.LockSupportCacheStore.remove(LockSupportCacheStore.java:230) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.container.DefaultDataContainer$DefaultEvictionListener.onEntryRemoved(DefaultDataContainer.java:245) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.remove(BoundedConcurrentHashMap.java:1735) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.util.concurrent.BoundedConcurrentHashMap.remove(BoundedConcurrentHashMap.java:2225) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.container.DefaultDataContainer.remove(DefaultDataContainer.java:178) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:198) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.locking.ClusteringDependentLogic$AllNodesLogic.commitEntry(ClusteringDependentLogic.java:160) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:271) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:373) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:258) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.EntryWrappingInterceptor.visitCommitCommand(EntryWrappingInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitCommitCommand(AbstractTxLockingInterceptor.java:100) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.NotificationInterceptor.visitCommitCommand(NotificationInterceptor.java:65) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.TxInterceptor.visitCommitCommand(TxInterceptor.java:153) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:182) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
                at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402)
                at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103)
                at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
                at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
                at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
                at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
                at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1172) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.CacheImpl.removeInternal(CacheImpl.java:308) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.CacheImpl.remove(CacheImpl.java:302) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.DecoratedCache.remove(DecoratedCache.java:325) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:313) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.realFileDelete(DistributedSegmentReadLocker.java:183) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.deleteOrReleaseReadLock(DistributedSegmentReadLocker.java:109) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.lucene.InfinispanDirectory.deleteFile(InfinispanDirectory.java:213) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
                at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:578) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:517) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexFileDeleter.deleteCommits(IndexFileDeleter.java:297) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:463) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexWriter.finishCommit(IndexWriter.java:3509) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3490) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3467) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3451) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.commitIndexWriter(IndexWriterHolder.java:160) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.commitIndexWriter(IndexWriterHolder.java:173) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                at org.hibernate.search.backend.impl.lucene.ExclusiveIndexWorkspaceImpl.afterTransactionApplied(ExclusiveIndexWorkspaceImpl.java:45) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:138) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_17]
                at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_17]
                at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_17]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_17]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_17]
                at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_17]
      Caused by: javax.resource.ResourceException: IJ000460: Error checking for a transaction
                at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:362)
                at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)
                at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139)
                ... 73 more
      Caused by: javax.resource.ResourceException: IJ000459: Transaction is not active: tx=TransactionImple < ac, BasicAction: 0:ffff0a402145:2215b4e0:515e34e2:6b status: ActionStatus.COMMITTED >
                at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:352)
                ... 75 more
       
      13:20:59,933 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (Hibernate Search: Index updates queue processor for index nodeinfo-1) ISPN000136: Execution error: org.infinispan.CacheException: org.infinispan.loaders.CacheLoaderException: This might be related to https://jira.jboss.org/browse/ISPN-604
                at org.infinispan.container.DefaultDataContainer$DefaultEvictionListener.onEntryRemoved(DefaultDataContainer.java:247) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.remove(BoundedConcurrentHashMap.java:1735) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.util.concurrent.BoundedConcurrentHashMap.remove(BoundedConcurrentHashMap.java:2225) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.container.DefaultDataContainer.remove(DefaultDataContainer.java:178) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:198) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.locking.ClusteringDependentLogic$AllNodesLogic.commitEntry(ClusteringDependentLogic.java:160) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:271) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:373) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:258) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.EntryWrappingInterceptor.visitCommitCommand(EntryWrappingInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitCommitCommand(AbstractTxLockingInterceptor.java:100) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.NotificationInterceptor.visitCommitCommand(NotificationInterceptor.java:65) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.TxInterceptor.visitCommitCommand(TxInterceptor.java:153) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:182) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
                at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402)
                at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103)
                at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
                at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
                at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
                at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
                at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1172) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.CacheImpl.removeInternal(CacheImpl.java:308) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.CacheImpl.remove(CacheImpl.java:302) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.DecoratedCache.remove(DecoratedCache.java:325) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:313) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.realFileDelete(DistributedSegmentReadLocker.java:183) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.deleteOrReleaseReadLock(DistributedSegmentReadLocker.java:109) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.lucene.InfinispanDirectory.deleteFile(InfinispanDirectory.java:213) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
                at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:578) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:517) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexFileDeleter.deleteCommits(IndexFileDeleter.java:297) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:463) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexWriter.finishCommit(IndexWriter.java:3509) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3490) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3467) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3451) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.commitIndexWriter(IndexWriterHolder.java:160) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.commitIndexWriter(IndexWriterHolder.java:173) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                at org.hibernate.search.backend.impl.lucene.ExclusiveIndexWorkspaceImpl.afterTransactionApplied(ExclusiveIndexWorkspaceImpl.java:45) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:138) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_17]
                at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_17]
                at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_17]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_17]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_17]
                at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_17]
      Caused by: org.infinispan.loaders.CacheLoaderException: This might be related to https://jira.jboss.org/browse/ISPN-604
                at org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory.getConnection(ManagedConnectionFactory.java:96) [infinispan-cachestore-jdbc-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.loaders.jdbc.binary.JdbcBinaryCacheStore.loadBucket(JdbcBinaryCacheStore.java:277) [infinispan-cachestore-jdbc-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.loaders.bucket.BucketBasedCacheStore.removeLockSafe(BucketBasedCacheStore.java:106) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.loaders.bucket.BucketBasedCacheStore.removeLockSafe(BucketBasedCacheStore.java:49) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.loaders.LockSupportCacheStore.remove(LockSupportCacheStore.java:230) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.container.DefaultDataContainer$DefaultEvictionListener.onEntryRemoved(DefaultDataContainer.java:245) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                ... 67 more
      Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000460: Error checking for a transaction
                at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:147)
                at org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory.getConnection(ManagedConnectionFactory.java:93) [infinispan-cachestore-jdbc-5.2.1.Final.jar:5.2.1.Final]
                ... 72 more
      Caused by: javax.resource.ResourceException: IJ000460: Error checking for a transaction
                at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:362)
                at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)
                at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139)
                ... 73 more
      Caused by: javax.resource.ResourceException: IJ000459: Transaction is not active: tx=TransactionImple < ac, BasicAction: 0:ffff0a402145:2215b4e0:515e34e2:6b status: ActionStatus.COMMITTED >
                at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:352)
                ... 75 more
       
      13:20:59,941 ERROR [org.infinispan.transaction.TransactionCoordinator] (Hibernate Search: Index updates queue processor for index nodeinfo-1) ISPN000188: Error while processing a commit in a two-phase transaction: org.infinispan.CacheException: org.infinispan.loaders.CacheLoaderException: This might be related to https://jira.jboss.org/browse/ISPN-604
                at org.infinispan.container.DefaultDataContainer$DefaultEvictionListener.onEntryRemoved(DefaultDataContainer.java:247) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.remove(BoundedConcurrentHashMap.java:1735) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.util.concurrent.BoundedConcurrentHashMap.remove(BoundedConcurrentHashMap.java:2225) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.container.DefaultDataContainer.remove(DefaultDataContainer.java:178) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:198) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.locking.ClusteringDependentLogic$AllNodesLogic.commitEntry(ClusteringDependentLogic.java:160) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:271) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:373) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:258) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.EntryWrappingInterceptor.visitCommitCommand(EntryWrappingInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitCommitCommand(AbstractTxLockingInterceptor.java:100) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.NotificationInterceptor.visitCommitCommand(NotificationInterceptor.java:65) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.TxInterceptor.visitCommitCommand(TxInterceptor.java:153) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:182) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
                at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402)
                at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103)
                at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
                at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
                at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
                at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
                at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1172) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.CacheImpl.removeInternal(CacheImpl.java:308) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.CacheImpl.remove(CacheImpl.java:302) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.DecoratedCache.remove(DecoratedCache.java:325) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:313) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.realFileDelete(DistributedSegmentReadLocker.java:183) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.deleteOrReleaseReadLock(DistributedSegmentReadLocker.java:109) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.lucene.InfinispanDirectory.deleteFile(InfinispanDirectory.java:213) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
                at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:578) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:517) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexFileDeleter.deleteCommits(IndexFileDeleter.java:297) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:463) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexWriter.finishCommit(IndexWriter.java:3509) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3490) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3467) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3451) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.commitIndexWriter(IndexWriterHolder.java:160) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.commitIndexWriter(IndexWriterHolder.java:173) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                at org.hibernate.search.backend.impl.lucene.ExclusiveIndexWorkspaceImpl.afterTransactionApplied(ExclusiveIndexWorkspaceImpl.java:45) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:138) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_17]
                at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_17]
                at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_17]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_17]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_17]
                at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_17]
      Caused by: org.infinispan.loaders.CacheLoaderException: This might be related to https://jira.jboss.org/browse/ISPN-604
                at org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory.getConnection(ManagedConnectionFactory.java:96) [infinispan-cachestore-jdbc-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.loaders.jdbc.binary.JdbcBinaryCacheStore.loadBucket(JdbcBinaryCacheStore.java:277) [infinispan-cachestore-jdbc-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.loaders.bucket.BucketBasedCacheStore.removeLockSafe(BucketBasedCacheStore.java:106) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.loaders.bucket.BucketBasedCacheStore.removeLockSafe(BucketBasedCacheStore.java:49) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.loaders.LockSupportCacheStore.remove(LockSupportCacheStore.java:230) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.container.DefaultDataContainer$DefaultEvictionListener.onEntryRemoved(DefaultDataContainer.java:245) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                ... 67 more
      Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000460: Error checking for a transaction
                at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:147)
                at org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory.getConnection(ManagedConnectionFactory.java:93) [infinispan-cachestore-jdbc-5.2.1.Final.jar:5.2.1.Final]
                ... 72 more
      Caused by: javax.resource.ResourceException: IJ000460: Error checking for a transaction
                at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:362)
                at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)
                at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139)
                ... 73 more
      Caused by: javax.resource.ResourceException: IJ000459: Transaction is not active: tx=TransactionImple < ac, BasicAction: 0:ffff0a402145:2215b4e0:515e34e2:6b status: ActionStatus.COMMITTED >
                at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:352)
                ... 75 more
       
      13:20:59,949 WARN  [com.arjuna.ats.jta] (Hibernate Search: Index updates queue processor for index nodeinfo-1) ARJUNA016029: SynchronizationImple.afterCompletion - failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=-1, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@2c} org.infinispan.transaction.synchronization.SynchronizationAdapter@4b with exception: org.infinispan.CacheException: Could not commit.
                at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:83) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
                at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402)
                at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103)
                at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
                at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
                at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
                at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
                at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1172) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.CacheImpl.removeInternal(CacheImpl.java:308) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.CacheImpl.remove(CacheImpl.java:302) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.DecoratedCache.remove(DecoratedCache.java:325) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:313) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.realFileDelete(DistributedSegmentReadLocker.java:183) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.deleteOrReleaseReadLock(DistributedSegmentReadLocker.java:109) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.lucene.InfinispanDirectory.deleteFile(InfinispanDirectory.java:213) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
                at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:578) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:517) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexFileDeleter.deleteCommits(IndexFileDeleter.java:297) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:463) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexWriter.finishCommit(IndexWriter.java:3509) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3490) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3467) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3451) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.commitIndexWriter(IndexWriterHolder.java:160) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.commitIndexWriter(IndexWriterHolder.java:173) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                at org.hibernate.search.backend.impl.lucene.ExclusiveIndexWorkspaceImpl.afterTransactionApplied(ExclusiveIndexWorkspaceImpl.java:45) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:138) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_17]
                at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_17]
                at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_17]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_17]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_17]
                at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_17]
      Caused by: javax.transaction.xa.XAException
                at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:224) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:185) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
                ... 34 more
       
      13:20:59,953 WARN  [com.arjuna.ats.arjuna] (Hibernate Search: Index updates queue processor for index nodeinfo-1) ARJUNA012127: TwoPhaseCoordinator.afterCompletion - returned failure for SynchronizationImple< 0:ffff0a402145:2215b4e0:515e34e2:6c, SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=-1, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@2c} org.infinispan.transaction.synchronization.SynchronizationAdapter@4b >
      {code}

       

      Note the mention of the ISPN-604 error in amongst all of that,

       

      Caused by: org.infinispan.loaders.CacheLoaderException: This might be related to https://jira.jboss.org/browse/ISPN-604

       

      If I set the transaction mode to NONE for the index caches then it runs ok for a while but there are no transactions on the index updates which is probably not a good thing. It does eventually fail with an odd error about lucene not being able to read a file which is strange as the index is mean't to be in the database. I'll add that error trace once I reproduce it again (takes a few hours and I didn't save the output).

       

      So, I am currently stuck. I can put the indexes into the filesystem but of course that is not transactional or clusterable. With the size of data I need to store I cannot keep indexes in memory and rebuild on startup.

       

      Not sure how to proceed from here.

      • 1045 Views
      • Categories: ModeShape 3.1
      • Tags:
        • 1. Re: Configuration for an All PostgreSQL repository storage
          bwallis42
          bwallis42 Apr 5, 2013 3:37 AM (in response to bwallis42)

          If I set the transaction mode to NONE for the index caches then it runs ok for a while but there are no transactions on the index updates which is probably not a good thing. It does eventually fail with an odd error about lucene not being able to read a file which is strange as the index is mean't to be in the database. I'll add that error trace once I reproduce it again (takes a few hours and I didn't save the output).

           

          That failed a lot quicker than I expected with SYNC indexing, last time I had ASYNC.

           

           

          {code}
          14:35:26,659 ERROR [org.hibernate.search.exception.impl.LogErrorHandler] (Lucene Merge Thread #44 for index nodeinfo) HSEARCH000058: HSEARCH000118: Exception during index Merge operation: org.apache.lucene.index.MergePolicy$MergeException: java.io.FileNotFoundException: No sub-file with id .frq found (fileName=_cd.cfs files: [.tii, .fnm, .tis, .fdt, .nrm, .fdx, .prx])
                    at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:509) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                    at org.hibernate.search.backend.impl.lucene.overrides.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:58) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
          Caused by: java.io.FileNotFoundException: No sub-file with id .frq found (fileName=_cd.cfs files: [.tii, .fnm, .tis, .fdt, .nrm, .fdx, .prx])
                    at org.apache.lucene.index.CompoundFileReader.openInput(CompoundFileReader.java:157) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                    at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:93) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                    at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                    at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4238) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3908) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                    at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
          {code}

           

          and it took about 3 hours to fail with ASYNC set. The error is slightly different, a different sub-file ID

           

           

          {code}
          17:48:30,093 ERROR [org.hibernate.search.exception.impl.LogErrorHandler] (Lucene Merge Thread #1171 for index nodeinfo) HSEARCH000058: HSEARCH000118: Exception during index Merge operation: org.apache.lucene.index.MergePolicy$MergeException: java.io.FileNotFoundException: No sub-file with id .fnm found (fileName=_9mw.cfs files: [.tii, .tis, .frq, .fdt, .nrm, .prx, .fdx])
                    at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:509) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                    at org.hibernate.search.backend.impl.lucene.overrides.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:58) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
                    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
          Caused by: java.io.FileNotFoundException: No sub-file with id .fnm found (fileName=_9mw.cfs files: [.tii, .tis, .frq, .fdt, .nrm, .prx, .fdx])
                    at org.apache.lucene.index.CompoundFileReader.openInput(CompoundFileReader.java:157) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                    at org.apache.lucene.index.CompoundFileReader.openInput(CompoundFileReader.java:146) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                    at org.apache.lucene.index.FieldInfos.<init>(FieldInfos.java:71) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                    at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:80) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                    at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:116) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                    at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                    at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4368) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                    at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3908) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                    at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
                    at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
          {code}
            • Actions
          • 2. Re: Configuration for an All PostgreSQL repository storage
            clementp
            clementp Apr 5, 2013 1:44 AM (in response to bwallis42)

            My personal understanding regarding infinispan is that the cache loader itself need not be transactional or replicatable in order for it be HA. Think of the cache loader as the the file system underneath a traditional transactional, replicated database. Infinispan itself handles the two phase commits and as long as the cache loader is reasonably written (write the necessary data to disk on prepare, fsync and then mark the journal on commit, again, fsync), there isn't a need to have the cache loader itself be part of the transaction. Of course, the disclaimer is there for cacheloaders that do not have transactional semantics (e.g. Cassandra or HBase), in those cases, there is no guarantee that the writes would have gone through. Last I checked, the BerkeleyDB store is reasonably robust in terms of handling XA transactions (although I should defer to the Infinispan forums if there are any doubts). A heuristical outcome would happen similar to when a database engine cannot fsync after a commit (leaving the system in an unknown state). The feature request in ISPN is there probably because it makes sense to have the container handle transaction recovery in one fell swoop without having the cache loader be a "side-effect" of the transaction if the cache loader happens to be a container managed JDBC datasource.

             

            As for replication, Infinispan itself is the one that handles the replication of data (and not the cache loader, although one can use a shared cache loader, with shared=true specified). In our tests, we simply have each infinispan node write to its local disk using BerkeleyDB and have Infinispan (via JGroups) replicate/distribute the data across nodes.

             

            Hope it helps.

            1 of 1 people found this helpful
              • Actions
            • 3. Re: Configuration for an All PostgreSQL repository storage
              rhauch
              rhauch Apr 5, 2013 9:41 AM (in response to clementp)

              I agree with Clement regarding the cache loader being transactional. It would be nice to have ISPN-604, but ISPN sitll works correctly when using cache loaders that use transactional backends.

               

              As for putting the lucene indexes inside Infinispan, please see Best way to configure ModeShape & Hibernate Search so that indexes are stored in Infinispan. (Please contribute index-specific questions to that post so the Hibernate Search folks can follow and help out.)

               

              You've also not said which version of ModeShape you're using. My guess is that you're using 'master' snapshots (e.g., what will be 3.2), which has a number of fixes for how the indexes are updated within user transactions.

                • Actions
              • 4. Re: Configuration for an All PostgreSQL repository storage
                bwallis42
                bwallis42 Apr 5, 2013 7:10 PM (in response to clementp)

                Clement Pang wrote:

                 

                In our tests, we simply have each infinispan node write to its local disk using BerkeleyDB and have Infinispan (via JGroups) replicate/distribute the data across nodes.

                 

                I'm not sure I understand how this works. Does each node have a complete copy of the index in this case? ie: when a change is made on a remote node and its index is updated, the update is replicated (via infinispan/JGroups) to the current node which then, seeing the update, stores that update to its local copy of the index using its cache loader?

                 

                What happens in the case when the local node is down. Does infinispan/JGroups take care of queuing the updates on the remote node and send them when the local node is back up? I realise that this is more an infinispan question but I'm still learning about that (as is probably obvious).

                 

                thanks

                  • Actions
                • 5. Re: Configuration for an All PostgreSQL repository storage
                  bwallis42
                  bwallis42 Apr 5, 2013 7:16 PM (in response to rhauch)

                  As for putting the lucene indexes inside Infinispan, please see Best way to configure ModeShape & Hibernate Search so that indexes are stored in Infinispan. (Please contribute index-specific questions to that post so the Hibernate Search folks can follow and help out.)

                   

                  You've also not said which version of ModeShape you're using. My guess is that you're using 'master' snapshots (e.g., what will be 3.2), which has a number of fixes for how the indexes are updated within user transactions.

                   

                  yep, a 3.2-Snapshot from master of about 2 days ago.

                   

                  My problem at the moment is that there is no configuration that I have found for the index caches that works as decribed above. With transactions set to either NON-XA or FULL-XA it fails immediatly on accessing the repository, with NONE, depending on SYNC or ASYNC, it fails either immediatly or a some time later with the odd file not found error. I'll post that error to the other topic as it seems to be hibernate search specific.

                   

                  thanks

                    • Actions
                  • 6. Re: Configuration for an All PostgreSQL repository storage
                    clementp
                    clementp Apr 5, 2013 10:17 PM (in response to bwallis42)

                    You can configure whether each node would have all the data by specifying whether it is a replicated or distributed cache. In the distributed configuration, you can also configure how many nodes in the cluster should have the data. As for what happens when a node rejoins, the options are (copied from the docs):

                     

                    • (state retrieval) It can "synchronize" itself with the content currently held in the cluster. This is called "state retrieval". In other words, it will fetch data from other cluster members to have exactly the same content as the others.
                    • (loader) It can reload data from its local storage. In this case, the node will revert to the state it has before it crashed. Of course, this content may be obsolete and different from what is currently in the cluster.
                    • (cluster loader) You can also ask the node to lazily fetch content from other cluster members when needed. In this case, you don't use the state retrieval feature, but let the node restart with an empty cache. When a request is made for data, since the cache is empty, the request flows to the cluster loader and is retrieved from other members. Everything looks like if the cache was preloaded at that startup.
                    1 of 1 people found this helpful
                      • Actions