1 Reply Latest reply on Oct 20, 2017 1:42 PM by galder.zamarreno

Infinispan 7.0.3Final Cluster ISPN000214: Unable to remove entry under WARNs

caro82 Oct 13, 2017 7:32 AM

We are running on Solaris Sparc, Java 8, a cluster of two Infinispan 7.0.3 Final nodes, deployed on 2 separate zones, on the same global machine. I attached the Infinispan configuration.

The Infinispan cluster is accessed (for reads/writes) from another application (a cluster of four Jboss 9.1.0 nodes) using a hotrod client.

The hotrod client uses a custom FailoverRequestBalancingStrategy which forwards the requests to the first node within the infinispan cluster. If a change in cluster is detected, e.g. one node goes down, it will reevaluate the target node.

Starting approximately 1 day ago the second node of the cluster (requests are posted to the node other than this one) started to experience the WARNs, ERRORs listed at the end of my comment.

The WARNs started to occur over-night, without any application restart in between.

We isolated the caches that are accessed when this error occurs: these are the caches set with Database persistence, like the one below.

<replicated-cache name="cache11" mode="ASYNC" start="EAGER">

<binary-keyed-jdbc-store datasource="java:jboss/datasources/JdbcDS" passivation="true" preload="true" purge="true">

<binary-keyed-table prefix="IS7">

<id-column name="id" type="VARCHAR2(500)"/>

<data-column name="datum" type="BLOB"/>

<timestamp-column name="version" type="NUMBER(19)"/>

</binary-keyed-table>

</binary-keyed-jdbc-store>

</replicated-cache>

We turned down the problematic Infinispan node, and the warns started to occur on the remaining infinispan node.

Please let me know if there is a fix or workaround available for this issue.

I would also appreciate any recommendations on how to troubleshoot this.

2017-10-12 14:40:26,072 WARN [org.infinispan.eviction.impl.ActivationManagerImpl] (remote-thread--p3-t4) ISPN000214: Unable to remove entry under [B@1b06673d from cache store after activation: org.infinispan.persistence.spi.PersistenceException: This might be related to https://jira.jboss.org/browse/ISPN-604

at org.infinispan.persistence.jdbc.connectionfactory.ManagedConnectionFactory.getConnection(ManagedConnectionFactory.java:85) [infinispan-cachestore-jdbc.jar:7.0.3.Final]

at org.infinispan.persistence.jdbc.binary.JdbcBinaryStore.loadBucket(JdbcBinaryStore.java:534) [infinispan-cachestore-jdbc.jar:7.0.3.Final]

...

at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_92]

Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000460: Error checking for a transaction

at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:154)

at org.infinispan.persistence.jdbc.connectionfactory.ManagedConnectionFactory.getConnection(ManagedConnectionFactory.java:82) [infinispan-cachestore-jdbc.jar:7.0.3.Final]

... 71 more

Caused by: javax.resource.ResourceException: IJ000460: Error checking for a transaction

at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:416)

at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:513)

at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:146)

... 72 more

Caused by: javax.resource.ResourceException: IJ000459: Transaction is not active: tx=TransactionImple < ac, BasicAction: 0:ffff0af540e0:4af75786:59dfc4f9:8705 status: ActionStatus.COMMITTED >

at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:400)

... 74 more

2017-10-12 14:40:26,092 ERROR [org.infinispan.persistence.jdbc.connectionfactory.ManagedConnectionFactory] (remote-thread--p3-t4) ISPN008018: Sql failure retrieving connection from datasource: java.sql.SQLException: javax.resource.ResourceException: IJ000460: Error checking for a transaction