org.infinispan.util.concurrent.TimeoutException: Replication timeout for Sakthi-PC-15984

[sakthivelmurugasamy]

Hi All,

I am using infinispan as L2 cache running in GlassFish V2.1.1 Server.

In sometimes      I am facing a problem "org.hibernate.cache.CacheException: org.infinispan.util.concurrent.TimeoutException: Replication timeout for Sakthi-PC-15984"

Caused by: org.hibernate.cache.CacheException: org.infinispan.util.concurrent.TimeoutException: Replication timeout for Sakthi-PC-15984

at org.hibernate.cache.infinispan.util.CacheAdapterImpl.remove(CacheAdapterImpl.java:125)

at org.hibernate.cache.infinispan.access.TransactionalAccessDelegate.remove(TransactionalAccessDelegate.java:144)

at org.hibernate.cache.infinispan.collection.TransactionalAccess.remove(TransactionalAccess.java:48)

at org.hibernate.action.CollectionAction.evict(CollectionAction.java:138)

at org.hibernate.action.CollectionUpdateAction.execute(CollectionUpdateAction.java:93)

at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:268)

at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:260)

at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:182)

at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)

at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51)

at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1206)

at com.orangescape.entity.DbMediatorService.commit(DbMediatorService.java:164)

... 131 more

Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for Sakthi-PC-15984

at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:421)

at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:100)

at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:124)

at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)

at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:216)

at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:199)

at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:192)

at org.infinispan.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:114)

at org.infinispan.interceptors.ReplicationInterceptor.visitRemoveCommand(ReplicationInterceptor.java:88)

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:70)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

at org.infinispan.interceptors.LockingInterceptor.visitRemoveCommand(LockingInterceptor.java:221)

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:70)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)

at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:61)

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:70)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

at org.infinispan.interceptors.MarshalledValueInterceptor.visitRemoveCommand(MarshalledValueInterceptor.java:106)

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:70)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:183)

at org.infinispan.interceptors.TxInterceptor.visitRemoveCommand(TxInterceptor.java:137)

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:70)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

at org.infinispan.interceptors.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:122)

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:70)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)

at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)

at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:61)

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:70)

at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:269)

at org.infinispan.CacheDelegate.remove(CacheDelegate.java:212)

at org.hibernate.cache.infinispan.util.CacheAdapterImpl.remove(CacheAdapterImpl.java:123)

... 142 more

Caused by: org.hibernate.cache.CacheException: org.infinispan.util.concurrent.TimeoutException: Replication timeout for Sakthi-PC-15984

at org.hibernate.cache.infinispan.util.CacheAdapterImpl.remove(CacheAdapterImpl.java:125)

at org.hibernate.cache.infinispan.access.TransactionalAccessDelegate.remove(TransactionalAccessDelegate.java:144)

at org.hibernate.cache.infinispan.collection.TransactionalAccess.remove(TransactionalAccess.java:48)

at org.hibernate.action.CollectionAction.evict(CollectionAction.java:138)

at org.hibernate.action.CollectionUpdateAction.execute(CollectionUpdateAction.java:93)

at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:268)

at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:260)

at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:182)

at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)

at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:51)

at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1206)

at com.orangescape.entity.DbMediatorService.commit(DbMediatorService.java:164)

... 131 more

Caused by: org.infinispan.util.concurrent.TimeoutException: Replication timeout for Sakthi-PC-15984

at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:421)

at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:100)

at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:124)

at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:229)

at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:216)

at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:199)

at org.infinispan.remoting.rpc.RpcManagerImpl.broadcastRpcCommand(RpcManagerImpl.java:192)

at org.infinispan.interceptors.ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:114)

at org.infinispan.interceptors.ReplicationInterceptor.visitRemoveCommand(ReplicationInterceptor.java:88)

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:70)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

at org.infinispan.interceptors.LockingInterceptor.visitRemoveCommand(LockingInterceptor.java:221)

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:70)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)

at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:61)

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:70)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

at org.infinispan.interceptors.MarshalledValueInterceptor.visitRemoveCommand(MarshalledValueInterceptor.java:106)

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:70)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:183)

at org.infinispan.interceptors.TxInterceptor.visitRemoveCommand(TxInterceptor.java:137)

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:70)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

at org.infinispan.interceptors.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:122)

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:70)

at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)

at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)

at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)

at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:61)

at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:70)

at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:269)

at org.infinispan.CacheDelegate.remove(CacheDelegate.java:212)

at org.hibernate.cache.infinispan.util.CacheAdapterImpl.remove(CacheAdapterImpl.java:123)

... 142 more

Server and configuration details are,

I am running an Application in 3 instance Cluster with Load Balancer.

Configurations	Details
Server:	GlassFish V2.1.1
Infinispan:	infinispan-core-4.0.0.FINAL.jar
Hibernate:	hibernate3.5.0_CR_2.jar
MySQL:	mysql-connector-java-5.1.13-bin.jar
JDK	1.6.0_21

Please find the attached Error log(server.log), Infinispan & Jgroups Config files.

I will provide any additional info if needed. Please suggest the solution ASAP.

Thanks in advance!

[galder.zamarreno]

First of all, I see that you're using a non default Infinispan 2LC config. Any particular reason for doing so? You should try reverting to the default configuration and see how it goes. If anything, I would only suggest changing clustering settings if default ones don't suit your env.

Secondly, I'd suggest using latest 4.1.0.Final and lates Hibernate 3.5.x release. Definitely not a CR release, specially since 3.5.x has been out for a while.