Issue with Longer Transaction with Huge data in Cluster with Infinispan / JGroups
Hi All,
After a length transaction completes in one of the node in the clustered environment, another node is not responding for certain period of time.
We migrated from JBoss Cache 3.x and JGroups 2.6 to Infinispan 5.3 and JGroups 3.3 in our application which consisted of 2 nodes. We had Job scheduled in our application using Quartz schedulers to have batch processing of particular data separately in 2 nodes. We had constraint to have lengthy transaction (1hr) in our Job class for processing huge number of data and update in to DB. After this operation happens in one node, another node is not responding to the user request for certain time.
In our Infinispan configuration, we have set entity & collection namedCache to INVALIDATION_ASYNC mode with useRelpQueue and asyncMarshalling enabled. We are caching standard queries as well in LOCAL_SYNC and Timestamp namedCache is in REPLICATION_ASYNC mode.
Default JGroups configuration which was shipped with Infinispan 5.3 was utilized for JGroups configuration.
After the lengthy transaction completes say 30 minutes in Node 1, it sends invalidation request to Node 2. Node 2 tries to invalidate this but not responding to the user requests. Node 1 is responding fine! Node 2 seems to not respond to any user request for next 20 mins or so.
The below exception is repeatedly thrown for all requests in node 2,
ERROR infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:148) - ISPN000136: Execution error
org.infinispan.loaders.CacheLoaderException: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
at org.infinispan.loaders.cluster.ClusterCacheLoader.doRemoteCall(ClusterCacheLoader.java:150)
at org.infinispan.loaders.cluster.ClusterCacheLoader.load(ClusterCacheLoader.java:84)
at org.infinispan.interceptors.CacheLoaderInterceptor.loadIfNeeded(CacheLoaderInterceptor.java:189)
at org.infinispan.interceptors.CacheLoaderInterceptor.loadIfNeededAndUpdateStats(CacheLoaderInterceptor.java:263)
at org.infinispan.interceptors.CacheLoaderInterceptor.visitGetKeyValueCommand(CacheLoaderInterceptor.java:122)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:62)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
at org.infinispan.interceptors.EntryWrappingInterceptor.visitGetKeyValueCommand(EntryWrappingInterceptor.java:119)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:62)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:96)
at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.visitGetKeyValueCommand(AbstractTxLockingInterceptor.java:98)
at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitGetKeyValueCommand(OptimisticLockingInterceptor.java:130)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:62)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:96)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:62)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
at org.infinispan.interceptors.MarshalledValueInterceptor.visitGetKeyValueCommand(MarshalledValueInterceptor.java:203)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:62)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:234)
at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:229)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:62)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:96)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:62)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
at org.infinispan.statetransfer.StateTransferInterceptor.handleDefault(StateTransferInterceptor.java:200)
at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:96)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:62)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
at org.infinispan.interceptors.CacheMgmtInterceptor.visitGetKeyValueCommand(CacheMgmtInterceptor.java:113)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:62)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:96)
at org.infinispan.interceptors.IsMarshallableInterceptor.visitGetKeyValueCommand(IsMarshallableInterceptor.java:97)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:62)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:96)
at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:62)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
at org.infinispan.CacheImpl.get(CacheImpl.java:391)
at org.infinispan.CacheImpl.get(CacheImpl.java:383)
at org.hibernate.cache.infinispan.util.CacheAdapterImpl.get(CacheAdapterImpl.java:77)
at org.hibernate.cache.infinispan.impl.BaseRegion.get(BaseRegion.java:209)
at org.hibernate.cache.infinispan.timestamp.TimestampsRegionImpl.get(TimestampsRegionImpl.java:61)
at org.hibernate.cache.UpdateTimestampsCache.isUpToDate(UpdateTimestampsCache.java:93)
at org.hibernate.cache.StandardQueryCache.isUpToDate(StandardQueryCache.java:187)
at org.hibernate.cache.StandardQueryCache.get(StandardQueryCache.java:137)
at org.hibernate.loader.Loader.getResultFromQueryCache(Loader.java:2245)
at org.hibernate.loader.Loader.listUsingQueryCache(Loader.java:2194)
at org.hibernate.loader.Loader.list(Loader.java:2164)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:448)
at org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:363)
at org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:196)
at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1258)
at org.hibernate.impl.QueryImpl.list(QueryImpl.java:102)
....
....
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
Caused by: org.infinispan.util.concurrent.TimeoutException: Timed out waiting for valid responses!
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:562)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:303)
at org.infinispan.loaders.cluster.ClusterCacheLoader.doRemoteCall(ClusterCacheLoader.java:147)
... 142 more
Can we invoke this entity invalidation with some specified batch size during the lengthy transaction itself? At least enabling useReplQueue has provided us some performance gain but the impact of another node not responding to the user requests still prevails...
Any quick suggestion / help would be highly appreciated!!!