Configuration for an All PostgreSQL repository storage
bwallis42 Apr 4, 2013 11:06 PMI'm having some difficulty getting this to work.
This may be related to the issues highlighted in this thread by Horia to do with indexing caches but I think there is also an underlying problem (ISPN-604) that is going to make this impossible to get working in a production environment.
I am getting concerned that the timeframe to a usable transactional modeshape/infinispan/JDBC database combination is going to be too far into the future for my project. I would really like to know if what I am trying to do is achievable and if so what configuration I should be using.
What my eventual configuration is meant to achieve is
- a small cluster of servers (2-3) running my application which is storing all document data and metadata into a JCR repository that is replicated across all the nodes.
- All data written to the repository is immediately persisted to secondary storage (ie: write through caches)
- Data written to a repository on one node is immediately available on the other nodes
- As I have other data stores used in the application (Hibernate/JPA and HornetQ at least) we need to use XA transactions. We absolutely have to maintain data consistency in the face of failures, this is a key requirement.
- Storage of 50 million documents with a size around 20TB in all (this is what our current system can do on a single server)
As far as I understand the persistence options for infinispan, the only choice for the cache loader that supports transactions is to use a relational database. The choices seem to be the various JDBC based loaders and a decent relational DB underneath (PostgreSQL is our preferred DB at the moment, oracle an option) or the BerkeleyDB one (but that has limitations in the replication area and it's datasheet does not mention XA support).
There is a comment in the infinispan documentation at the end of the page on cache loaders and stores that is a concern
When a cache is transactional and a cache loader is present, the cache loader won't be enlisted in the transaction in which the cache is part. That means that it is possible to have inconsistencies at cache loader level: the transaction to succeed applying the in-memory state but (partially) fail applying the changes to the store. Manual recovery would not work with caches stores.
I'm assuming that this is a reference to the long standing bug ISPN-604 that has been a concern to me for quite a while. Currently scheduled for a fix in infinispan 5.3.0 which is unlikely to be in an EAP release of JBoss anytime soon. (I actually find it hard to believe that this is not a critical bug rather than a feature request).
My first step is to get the persistence of the repository working on a single server to a PostgreSQL database for both the repository data and indexes using write through JDBC stores for the caches *with transactions enabled*. Thus far I have failed.
My current configuration is (using NON_XA, but similar results with FULL_XA)
{code}
<subsystem xmlns="urn:jboss:domain:infinispan:1.4">
... hibernate stuff
<cache-container name="modeshapeWorkspaceCache" default-cache="workspaceCache">
<local-cache name="workspaceCache">
<transaction mode="NON_XA"/>
<eviction strategy="LRU" max-entries="100"/>
<expiration max-idle="5000" lifespan="10000" interval="1000"/>
</local-cache>
<local-cache name="testRepository/default">
<transaction mode="NON_XA"/>
<eviction strategy="LRU" max-entries="100"/>
<expiration max-idle="5000" lifespan="10000" interval="1000"/>
</local-cache>
</cache-container>
<cache-container name="modeshapeCache">
<local-cache name="testRepositoryCache" batching="false">
<transaction mode="NON_XA"/>
<eviction strategy="LIRS" max-entries="512"/>
<string-keyed-jdbc-store datasource="java:jboss/datasources/JcrDataDS" passivation="false" purge="false">
<property name="databaseType">
postgres
</property>
<property name="createTableOnStart">
true
</property>
<string-keyed-table prefix="stringbased">
<id-column name="id" type="VARCHAR(200)"/>
<data-column name="datum" type="BYTEA"/>
<timestamp-column name="version" type="BIGINT"/>
</string-keyed-table>
</string-keyed-jdbc-store>
</local-cache>
</cache-container>
<cache-container name="modeshapeBinaryCache">
<local-cache name="testRepository_binary_data" batching="false">
<transaction mode="NON_XA"/>
<eviction strategy="LIRS" max-entries="5"/>
<binary-keyed-jdbc-store datasource="java:jboss/datasources/JcrDataDS" passivation="false" purge="false">
<property name="databaseType">
postgres
</property>
<property name="createTableOnStart">
true
</property>
<binary-keyed-table prefix="binarybased">
<id-column name="id" type="VARCHAR(200)"/>
<data-column name="datum" type="BYTEA"/>
<timestamp-column name="version" type="BIGINT"/>
</binary-keyed-table>
</binary-keyed-jdbc-store>
</local-cache>
<local-cache name="testRepository_binary_metadata" batching="false">
<transaction mode="NON_XA"/>
<eviction strategy="LIRS" max-entries="2000"/>
<binary-keyed-jdbc-store datasource="java:jboss/datasources/JcrDataDS" passivation="false" purge="false">
<property name="databaseType">
postgres
</property>
<property name="createTableOnStart">
true
</property>
<binary-keyed-table prefix="binarybased">
<id-column name="id" type="VARCHAR(200)"/>
<data-column name="datum" type="BYTEA"/>
<timestamp-column name="version" type="BIGINT"/>
</binary-keyed-table>
</binary-keyed-jdbc-store>
</local-cache>
</cache-container>
<cache-container name="modeshapeIndexCache">
<local-cache name="testRepository_index_locks" batching="false">
<transaction mode="NON_XA"/>
<binary-keyed-jdbc-store datasource="java:jboss/datasources/JcrDataDS" passivation="false" purge="false">
<property name="databaseType">
postgres
</property>
<property name="createTableOnStart">
true
</property>
<binary-keyed-table prefix="binarybased">
<id-column name="id" type="VARCHAR(200)"/>
<data-column name="datum" type="BYTEA"/>
<timestamp-column name="version" type="BIGINT"/>
</binary-keyed-table>
</binary-keyed-jdbc-store>
</local-cache>
<local-cache name="testRepository_index_data" batching="false">
<transaction mode="NON_XA"/>
<eviction strategy="LIRS" max-entries="2000"/>
<binary-keyed-jdbc-store datasource="java:jboss/datasources/JcrDataDS" passivation="false" purge="false">
<property name="databaseType">
postgres
</property>
<property name="createTableOnStart">
true
</property>
<binary-keyed-table prefix="binarybased">
<id-column name="id" type="VARCHAR(200)"/>
<data-column name="datum" type="BYTEA"/>
<timestamp-column name="version" type="BIGINT"/>
</binary-keyed-table>
</binary-keyed-jdbc-store>
</local-cache>
<local-cache name="testRepository_index_metadata" batching="false">
<transaction mode="NON_XA"/>
<eviction strategy="LIRS" max-entries="2000"/>
<binary-keyed-jdbc-store datasource="java:jboss/datasources/JcrDataDS" passivation="false" purge="false">
<property name="databaseType">
postgres
</property>
<property name="createTableOnStart">
true
</property>
<binary-keyed-table prefix="binarybased">
<id-column name="id" type="VARCHAR(200)"/>
<data-column name="datum" type="BYTEA"/>
<timestamp-column name="version" type="BIGINT"/>
</binary-keyed-table>
</binary-keyed-jdbc-store>
</local-cache>
</cache-container>
</subsystem>
<subsystem xmlns="urn:jboss:domain:modeshape:1.0">
<repository name="testRepository" cache-name="testRepositoryCache" cache-container="modeshapeCache">
<workspaces cache-container="modeshapeWorkspaceCache">
<workspace name="default"/>
</workspaces>
<indexing systemContentMode="SYNC"/>
<cache-index-storage data-cache-name="testRepository_index_data" metadata-cache-name="testRepository_index_metadata" lock-cache-name="testRepository_index_locks" cache-container="modeshapeIndexCache"/>
<cache-binary-storage min-value-size="0" min-string-size="256" data-cache-name="testRepository_binary_data" metadata-cache-name="testRepository_binary_metadata" cache-container="modeshapeBinaryCache"/>
</repository>
<webapp name="modeshape-rest.war"/>
<webapp name="modeshape-cmis.war"/>
<webapp name="modeshape-webdav.war"/>
</subsystem>
{code}
This just does not work at all. As soon as I access the repository via the REST api (http://localhost:8080/modeshape-rest/testRepository) I start getting errors like the following
{code}
13:20:59,923 ERROR [org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory] (Hibernate Search: Index updates queue processor for index nodeinfo-1) ISPN008018: Sql failure retrieving connection from datasource: java.sql.SQLException: javax.resource.ResourceException: IJ000460: Error checking for a transaction
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:147)
at org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory.getConnection(ManagedConnectionFactory.java:93) [infinispan-cachestore-jdbc-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.loaders.jdbc.binary.JdbcBinaryCacheStore.loadBucket(JdbcBinaryCacheStore.java:277) [infinispan-cachestore-jdbc-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.loaders.bucket.BucketBasedCacheStore.removeLockSafe(BucketBasedCacheStore.java:106) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.loaders.bucket.BucketBasedCacheStore.removeLockSafe(BucketBasedCacheStore.java:49) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.loaders.LockSupportCacheStore.remove(LockSupportCacheStore.java:230) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.container.DefaultDataContainer$DefaultEvictionListener.onEntryRemoved(DefaultDataContainer.java:245) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.remove(BoundedConcurrentHashMap.java:1735) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.util.concurrent.BoundedConcurrentHashMap.remove(BoundedConcurrentHashMap.java:2225) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.container.DefaultDataContainer.remove(DefaultDataContainer.java:178) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:198) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.locking.ClusteringDependentLogic$AllNodesLogic.commitEntry(ClusteringDependentLogic.java:160) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:271) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:373) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:258) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.EntryWrappingInterceptor.visitCommitCommand(EntryWrappingInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitCommitCommand(AbstractTxLockingInterceptor.java:100) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.NotificationInterceptor.visitCommitCommand(NotificationInterceptor.java:65) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.TxInterceptor.visitCommitCommand(TxInterceptor.java:153) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:182) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1172) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.CacheImpl.removeInternal(CacheImpl.java:308) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.CacheImpl.remove(CacheImpl.java:302) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.DecoratedCache.remove(DecoratedCache.java:325) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:313) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.realFileDelete(DistributedSegmentReadLocker.java:183) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.deleteOrReleaseReadLock(DistributedSegmentReadLocker.java:109) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.lucene.InfinispanDirectory.deleteFile(InfinispanDirectory.java:213) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:578) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:517) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexFileDeleter.deleteCommits(IndexFileDeleter.java:297) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:463) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexWriter.finishCommit(IndexWriter.java:3509) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3490) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3467) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3451) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.commitIndexWriter(IndexWriterHolder.java:160) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.commitIndexWriter(IndexWriterHolder.java:173) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
at org.hibernate.search.backend.impl.lucene.ExclusiveIndexWorkspaceImpl.afterTransactionApplied(ExclusiveIndexWorkspaceImpl.java:45) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:138) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_17]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_17]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_17]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_17]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_17]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_17]
Caused by: javax.resource.ResourceException: IJ000460: Error checking for a transaction
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:362)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139)
... 73 more
Caused by: javax.resource.ResourceException: IJ000459: Transaction is not active: tx=TransactionImple < ac, BasicAction: 0:ffff0a402145:2215b4e0:515e34e2:6b status: ActionStatus.COMMITTED >
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:352)
... 75 more
13:20:59,933 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (Hibernate Search: Index updates queue processor for index nodeinfo-1) ISPN000136: Execution error: org.infinispan.CacheException: org.infinispan.loaders.CacheLoaderException: This might be related to https://jira.jboss.org/browse/ISPN-604
at org.infinispan.container.DefaultDataContainer$DefaultEvictionListener.onEntryRemoved(DefaultDataContainer.java:247) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.remove(BoundedConcurrentHashMap.java:1735) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.util.concurrent.BoundedConcurrentHashMap.remove(BoundedConcurrentHashMap.java:2225) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.container.DefaultDataContainer.remove(DefaultDataContainer.java:178) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:198) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.locking.ClusteringDependentLogic$AllNodesLogic.commitEntry(ClusteringDependentLogic.java:160) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:271) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:373) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:258) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.EntryWrappingInterceptor.visitCommitCommand(EntryWrappingInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitCommitCommand(AbstractTxLockingInterceptor.java:100) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.NotificationInterceptor.visitCommitCommand(NotificationInterceptor.java:65) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.TxInterceptor.visitCommitCommand(TxInterceptor.java:153) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:182) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1172) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.CacheImpl.removeInternal(CacheImpl.java:308) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.CacheImpl.remove(CacheImpl.java:302) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.DecoratedCache.remove(DecoratedCache.java:325) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:313) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.realFileDelete(DistributedSegmentReadLocker.java:183) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.deleteOrReleaseReadLock(DistributedSegmentReadLocker.java:109) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.lucene.InfinispanDirectory.deleteFile(InfinispanDirectory.java:213) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:578) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:517) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexFileDeleter.deleteCommits(IndexFileDeleter.java:297) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:463) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexWriter.finishCommit(IndexWriter.java:3509) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3490) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3467) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3451) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.commitIndexWriter(IndexWriterHolder.java:160) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.commitIndexWriter(IndexWriterHolder.java:173) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
at org.hibernate.search.backend.impl.lucene.ExclusiveIndexWorkspaceImpl.afterTransactionApplied(ExclusiveIndexWorkspaceImpl.java:45) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:138) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_17]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_17]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_17]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_17]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_17]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_17]
Caused by: org.infinispan.loaders.CacheLoaderException: This might be related to https://jira.jboss.org/browse/ISPN-604
at org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory.getConnection(ManagedConnectionFactory.java:96) [infinispan-cachestore-jdbc-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.loaders.jdbc.binary.JdbcBinaryCacheStore.loadBucket(JdbcBinaryCacheStore.java:277) [infinispan-cachestore-jdbc-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.loaders.bucket.BucketBasedCacheStore.removeLockSafe(BucketBasedCacheStore.java:106) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.loaders.bucket.BucketBasedCacheStore.removeLockSafe(BucketBasedCacheStore.java:49) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.loaders.LockSupportCacheStore.remove(LockSupportCacheStore.java:230) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.container.DefaultDataContainer$DefaultEvictionListener.onEntryRemoved(DefaultDataContainer.java:245) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
... 67 more
Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000460: Error checking for a transaction
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:147)
at org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory.getConnection(ManagedConnectionFactory.java:93) [infinispan-cachestore-jdbc-5.2.1.Final.jar:5.2.1.Final]
... 72 more
Caused by: javax.resource.ResourceException: IJ000460: Error checking for a transaction
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:362)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139)
... 73 more
Caused by: javax.resource.ResourceException: IJ000459: Transaction is not active: tx=TransactionImple < ac, BasicAction: 0:ffff0a402145:2215b4e0:515e34e2:6b status: ActionStatus.COMMITTED >
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:352)
... 75 more
13:20:59,941 ERROR [org.infinispan.transaction.TransactionCoordinator] (Hibernate Search: Index updates queue processor for index nodeinfo-1) ISPN000188: Error while processing a commit in a two-phase transaction: org.infinispan.CacheException: org.infinispan.loaders.CacheLoaderException: This might be related to https://jira.jboss.org/browse/ISPN-604
at org.infinispan.container.DefaultDataContainer$DefaultEvictionListener.onEntryRemoved(DefaultDataContainer.java:247) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.util.concurrent.BoundedConcurrentHashMap$Segment.remove(BoundedConcurrentHashMap.java:1735) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.util.concurrent.BoundedConcurrentHashMap.remove(BoundedConcurrentHashMap.java:2225) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.container.DefaultDataContainer.remove(DefaultDataContainer.java:178) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.container.entries.ReadCommittedEntry.commit(ReadCommittedEntry.java:198) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.locking.ClusteringDependentLogic$AllNodesLogic.commitEntry(ClusteringDependentLogic.java:160) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:271) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:373) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:258) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.EntryWrappingInterceptor.visitCommitCommand(EntryWrappingInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitCommitCommand(AbstractTxLockingInterceptor.java:100) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.NotificationInterceptor.visitCommitCommand(NotificationInterceptor.java:65) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.TxInterceptor.visitCommitCommand(TxInterceptor.java:153) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.AbstractVisitor.visitCommitCommand(AbstractVisitor.java:136) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.commands.tx.CommitCommand.acceptVisitor(CommitCommand.java:60) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:182) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1172) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.CacheImpl.removeInternal(CacheImpl.java:308) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.CacheImpl.remove(CacheImpl.java:302) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.DecoratedCache.remove(DecoratedCache.java:325) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:313) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.realFileDelete(DistributedSegmentReadLocker.java:183) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.deleteOrReleaseReadLock(DistributedSegmentReadLocker.java:109) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.lucene.InfinispanDirectory.deleteFile(InfinispanDirectory.java:213) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:578) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:517) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexFileDeleter.deleteCommits(IndexFileDeleter.java:297) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:463) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexWriter.finishCommit(IndexWriter.java:3509) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3490) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3467) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3451) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.commitIndexWriter(IndexWriterHolder.java:160) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.commitIndexWriter(IndexWriterHolder.java:173) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
at org.hibernate.search.backend.impl.lucene.ExclusiveIndexWorkspaceImpl.afterTransactionApplied(ExclusiveIndexWorkspaceImpl.java:45) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:138) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_17]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_17]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_17]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_17]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_17]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_17]
Caused by: org.infinispan.loaders.CacheLoaderException: This might be related to https://jira.jboss.org/browse/ISPN-604
at org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory.getConnection(ManagedConnectionFactory.java:96) [infinispan-cachestore-jdbc-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.loaders.jdbc.binary.JdbcBinaryCacheStore.loadBucket(JdbcBinaryCacheStore.java:277) [infinispan-cachestore-jdbc-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.loaders.bucket.BucketBasedCacheStore.removeLockSafe(BucketBasedCacheStore.java:106) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.loaders.bucket.BucketBasedCacheStore.removeLockSafe(BucketBasedCacheStore.java:49) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.loaders.LockSupportCacheStore.remove(LockSupportCacheStore.java:230) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.container.DefaultDataContainer$DefaultEvictionListener.onEntryRemoved(DefaultDataContainer.java:245) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
... 67 more
Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000460: Error checking for a transaction
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:147)
at org.infinispan.loaders.jdbc.connectionfactory.ManagedConnectionFactory.getConnection(ManagedConnectionFactory.java:93) [infinispan-cachestore-jdbc-5.2.1.Final.jar:5.2.1.Final]
... 72 more
Caused by: javax.resource.ResourceException: IJ000460: Error checking for a transaction
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:362)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139)
... 73 more
Caused by: javax.resource.ResourceException: IJ000459: Transaction is not active: tx=TransactionImple < ac, BasicAction: 0:ffff0a402145:2215b4e0:515e34e2:6b status: ActionStatus.COMMITTED >
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:352)
... 75 more
13:20:59,949 WARN [com.arjuna.ats.jta] (Hibernate Search: Index updates queue processor for index nodeinfo-1) ARJUNA016029: SynchronizationImple.afterCompletion - failed for SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=-1, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@2c} org.infinispan.transaction.synchronization.SynchronizationAdapter@4b with exception: org.infinispan.CacheException: Could not commit.
at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:83) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1172) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.CacheImpl.removeInternal(CacheImpl.java:308) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.CacheImpl.remove(CacheImpl.java:302) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.DecoratedCache.remove(DecoratedCache.java:325) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.AbstractDelegatingCache.remove(AbstractDelegatingCache.java:313) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.realFileDelete(DistributedSegmentReadLocker.java:183) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.lucene.readlocks.DistributedSegmentReadLocker.deleteOrReleaseReadLock(DistributedSegmentReadLocker.java:109) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.lucene.InfinispanDirectory.deleteFile(InfinispanDirectory.java:213) [infinispan-lucene-directory-5.2.1.Final.jar:5.2.1.Final]
at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:578) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:517) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexFileDeleter.deleteCommits(IndexFileDeleter.java:297) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexFileDeleter.checkpoint(IndexFileDeleter.java:463) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexWriter.finishCommit(IndexWriter.java:3509) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3490) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3467) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3451) [lucene-core-3.6.2.jar:3.6.2 1423725 - rmuir - 2012-12-18 19:45:40]
at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.commitIndexWriter(IndexWriterHolder.java:160) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
at org.hibernate.search.backend.impl.lucene.IndexWriterHolder.commitIndexWriter(IndexWriterHolder.java:173) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
at org.hibernate.search.backend.impl.lucene.ExclusiveIndexWorkspaceImpl.afterTransactionApplied(ExclusiveIndexWorkspaceImpl.java:45) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.applyUpdates(LuceneBackendQueueTask.java:138) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueTask.run(LuceneBackendQueueTask.java:67) [hibernate-search-engine-4.2.0.Final.jar:4.2.0.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_17]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_17]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_17]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_17]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_17]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_17]
Caused by: javax.transaction.xa.XAException
at org.infinispan.transaction.TransactionCoordinator.handleCommitFailure(TransactionCoordinator.java:224) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.transaction.TransactionCoordinator.commit(TransactionCoordinator.java:185) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
at org.infinispan.transaction.synchronization.SynchronizationAdapter.afterCompletion(SynchronizationAdapter.java:81) [infinispan-core-5.2.1.Final.jar:5.2.1.Final]
... 34 more
13:20:59,953 WARN [com.arjuna.ats.arjuna] (Hibernate Search: Index updates queue processor for index nodeinfo-1) ARJUNA012127: TwoPhaseCoordinator.afterCompletion - returned failure for SynchronizationImple< 0:ffff0a402145:2215b4e0:515e34e2:6c, SynchronizationAdapter{localTransaction=LocalTransaction{remoteLockedNodes=null, isMarkedForRollback=false, lockedKeys=null, backupKeyLocks=null, topologyId=-1, isFromStateTransfer=false} org.infinispan.transaction.synchronization.SyncLocalTransaction@2c} org.infinispan.transaction.synchronization.SynchronizationAdapter@4b >
{code}
Note the mention of the ISPN-604 error in amongst all of that,
Caused by: org.infinispan.loaders.CacheLoaderException: This might be related to https://jira.jboss.org/browse/ISPN-604
If I set the transaction mode to NONE for the index caches then it runs ok for a while but there are no transactions on the index updates which is probably not a good thing. It does eventually fail with an odd error about lucene not being able to read a file which is strange as the index is mean't to be in the database. I'll add that error trace once I reproduce it again (takes a few hours and I didn't save the output).
So, I am currently stuck. I can put the indexes into the filesystem but of course that is not transactional or clusterable. With the size of data I need to store I cannot keep indexes in memory and rebuild on startup.
Not sure how to proceed from here.