1 Reply Latest reply on Mar 3, 2015 11:54 PM by muthukumaran.kothandaraman

    Infinispan : Removing a non-tx, sync-replicated caches clusterwide

    muthukumaran.kothandaraman
    muthukumaran.kothandaraman Mar 3, 2015 4:27 AM

      Requirement

      My app has a requirement of dynamically creating and removing caches in a sync-replicated cluster using EmbeddedCacheManager.

       

      Approach taken was

      - create a master cache which holds the cache-names where apps put or remove entries

      - a cache-listener was attached to this cache and on @CacheEntryCreated and @cacheEntryModified caches are being created and removed cluster-wide

       

      Observed Results

      This approach works without any issues. Now I have a strange issue when one node of cluster is crashed or rebooted to simulate topology-changes with Timeout/Suspect exceptions.

      Cache removal across cluster fails. This failure is not observed in case of 2-node cluster (as I understand, when one node in 2-node cluster crashes, no significant consistency requirements exist for any operations). But when the cluster-size increases beyond 2 nodes, I see issues in consistently removing the caches.

       

      I wrote a small simulator with above approach and I see following exceptions on Node 1, Node 2 and Node 3 respectively. Colored portions represent my sample program's debug messages.

       

      Further details

      - Infinispan version 6.0.2-Final

      - Sync-replicated cluster

       

      Possible Solutions and queries around the same

      - Retry cache-removal based on TimeoutException/SuspectException. Would this be correct approach ?

      - Retry cache-removal till cacheExists returns false - but I suspect that this may not still guarantee that the cache-removal has happened clusterwide

      - Should cache-removal be happening only from one node of cluster ? Because embeddedCacheManager.removeCache appears to be clusterwide command from code

       

      Any other approaches to perform cluster-wide cache-removal in a more robust manner even under nodal failure scenarios ?

       

       

      Exception during cache-removal on Node 1

       

      2015-03-01 20:00:33,993 ERROR [InvocationContextInterceptor] (Thread-1) ISPN000136: Execution error

      org.infinispan.util.concurrent.TimeoutException: Node IN00106404-21233 timed out

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:174)

          at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:521)

          at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:281)

          at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:233)

          at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitRemoveCommand(NonTxDistributionInterceptor.java:110)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:321)

          at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:402)

          at org.infinispan.interceptors.EntryWrappingInterceptor.visitRemoveCommand(EntryWrappingInterceptor.java:216)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitRemoveCommand(NonTransactionalLockingInterceptor.java:81)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

          at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

          at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:219)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitRemoveCommand(StateTransferInterceptor.java:156)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:166)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)

          at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)

          at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1306)

          at org.infinispan.CacheImpl.removeInternal(CacheImpl.java:407)

          at org.infinispan.CacheImpl.remove(CacheImpl.java:400)

          at org.infinispan.CacheImpl.remove(CacheImpl.java:395)

          at infinispan.dynacache.DataLoader$1.run(DataLoader.java:87)

      Caused by: org.jgroups.TimeoutException: timeout sending message to IN00106404-21233

          at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:353)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:167)

          ... 39 more

      org.infinispan.util.concurrent.TimeoutException: Node IN00106404-21233 timed out

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:174)

          at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:521)

          at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:281)

          at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:233)

          at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitRemoveCommand(NonTxDistributionInterceptor.java:110)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:321)

          at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:402)

          at org.infinispan.interceptors.EntryWrappingInterceptor.visitRemoveCommand(EntryWrappingInterceptor.java:216)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitRemoveCommand(NonTransactionalLockingInterceptor.java:81)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

          at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

          at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:219)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitRemoveCommand(StateTransferInterceptor.java:156)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:166)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)

          at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)

          at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1306)

          at org.infinispan.CacheImpl.removeInternal(CacheImpl.java:407)

          at org.infinispan.CacheImpl.remove(CacheImpl.java:400)

          at org.infinispan.CacheImpl.remove(CacheImpl.java:395)

          at infinispan.dynacache.DataLoader$1.run(DataLoader.java:87)

      Caused by: org.jgroups.TimeoutException: timeout sending message to IN00106404-21233

          at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:419)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:353)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:167)

          ... 39 more

       

       

      Exception during cache-removal on Node 2

       

      org.infinispan.commons.CacheException: Error removing cache

          at org.infinispan.manager.DefaultCacheManager.removeCache(DefaultCacheManager.java:471)

          at infinispan.dynacache.DynamicCacheListener.removeCache(DynamicCacheListener.java:91)

          at infinispan.dynacache.DynamicCacheListener.observeRemove(DynamicCacheListener.java:64)

          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

          at java.lang.reflect.Method.invoke(Method.java:606)

          at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation$1.run(AbstractListenerImpl.java:207)

          at org.infinispan.util.concurrent.WithinThreadExecutor.execute(WithinThreadExecutor.java:22)

          at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation.invoke(AbstractListenerImpl.java:229)

          at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation.invoke(AbstractListenerImpl.java:192)

          at org.infinispan.notifications.cachelistener.CacheNotifierImpl.notifyCacheEntryRemoved(CacheNotifierImpl.java:230)

          at org.infinispan.interceptors.locking.ClusteringDependentLogic$AbstractClusteringDependentLogic.notifyCommitEntry(ClusteringDependentLogic.java:114)

          at org.infinispan.interceptors.locking.ClusteringDependentLogic$InvalidationLogic.commitEntry(ClusteringDependentLogic.java:347)

          at org.infinispan.interceptors.locking.ClusteringDependentLogic$ReplicationLogic.commitEntry(ClusteringDependentLogic.java:387)

          at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntry(EntryWrappingInterceptor.java:317)

          at org.infinispan.interceptors.EntryWrappingInterceptor.commitEntryIfNeeded(EntryWrappingInterceptor.java:542)

          at org.infinispan.interceptors.EntryWrappingInterceptor.commitContextEntries(EntryWrappingInterceptor.java:303)

          at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:348)

          at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:402)

          at org.infinispan.interceptors.EntryWrappingInterceptor.visitRemoveCommand(EntryWrappingInterceptor.java:216)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitRemoveCommand(NonTransactionalLockingInterceptor.java:81)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

          at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

          at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:213)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitRemoveCommand(StateTransferInterceptor.java:156)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:166)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)

          at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)

          at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39)

          at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48)

          at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:95)

          at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:50)

          at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:172)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

          at java.lang.Thread.run(Thread.java:745)

      Caused by: org.infinispan.remoting.RemoteException: ISPN000217: Received exception from IN00106404-21233, see cause for remote stack trace

          at org.infinispan.remoting.transport.AbstractTransport.checkResponse(AbstractTransport.java:41)

          at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:66)

          at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:547)

          at org.infinispan.manager.DefaultCacheManager.removeCache(DefaultCacheManager.java:463)

          ... 53 more

      Caused by: org.infinispan.commons.CacheException: Problems invoking command.

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:221)

          at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)

          at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)

          at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)

          at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:665)

          at org.jgroups.JChannel.up(JChannel.java:708)

          at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1015)

          at org.jgroups.protocols.RSVP.up(RSVP.java:187)

          at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)

          at org.jgroups.protocols.FlowControl.up(FlowControl.java:370)

          at org.jgroups.protocols.FlowControl.up(FlowControl.java:381)

          at org.jgroups.protocols.tom.TOA.up(TOA.java:121)

          at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010)

          at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)

          at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:390)

          at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:774)

          at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:570)

          at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)

          at org.jgroups.protocols.FD.up(FD.java:255)

          at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)

          at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)

          at org.jgroups.protocols.Discovery.up(Discovery.java:379)

          at org.jgroups.protocols.MPING.up(MPING.java:181)

          at org.jgroups.protocols.TP.passMessageUp(TP.java:1399)

          at org.jgroups.protocols.TP$MyHandler.run(TP.java:1585)

          ... 3 more

      Caused by: java.lang.NullPointerException

          at org.infinispan.commands.RemoteCommandsFactory.fromStream(RemoteCommandsFactory.java:195)

          at org.infinispan.marshall.exts.ReplicableCommandExternalizer.fromStream(ReplicableCommandExternalizer.java:106)

          at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:147)

          at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:59)

          at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:389)

          at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:205)

          at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:152)

          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

          at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)

          at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:136)

          at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)

          at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)

          at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:206)

          ... 27 more

       

       

      Exception during cache-removal on Node 3


      2015-03-01 19:58:48,074 WARN  [CommandAwareRpcDispatcher] (OOB-1,IN00106404-21233) ISPN000220: Problems un-marshalling remote command from byte buffer

      java.lang.NullPointerException

          at org.infinispan.commands.RemoteCommandsFactory.fromStream(RemoteCommandsFactory.java:195)

          at org.infinispan.marshall.exts.ReplicableCommandExternalizer.fromStream(ReplicableCommandExternalizer.java:106)

          at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:147)

          at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:59)

          at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:389)

          at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:205)

          at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:152)

          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

          at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)

          at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:136)

          at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)

          at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)

          at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:206)

          at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)

          at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)

          at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)

          at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:665)

          at org.jgroups.JChannel.up(JChannel.java:708)

          at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1015)

          at org.jgroups.protocols.RSVP.up(RSVP.java:187)

          at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)

          at org.jgroups.protocols.FlowControl.up(FlowControl.java:370)

          at org.jgroups.protocols.FlowControl.up(FlowControl.java:381)

          at org.jgroups.protocols.tom.TOA.up(TOA.java:121)

          at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010)

          at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)

          at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:390)

          at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:774)

          at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:570)

          at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)

          at org.jgroups.protocols.FD.up(FD.java:255)

          at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)

          at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)

          at org.jgroups.protocols.Discovery.up(Discovery.java:379)

          at org.jgroups.protocols.MPING.up(MPING.java:181)

          at org.jgroups.protocols.TP.passMessageUp(TP.java:1399)

          at org.jgroups.protocols.TP$MyHandler.run(TP.java:1585)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

          at java.lang.Thread.run(Thread.java:745)

      >>>>> Removing the dynamiccache key-2 and printing current list of caches

      withReplicationQueue

       

       

       

       

      2015-03-01 19:58:48,261 INFO  [CacheJmxRegistration] (remote-thread-0) ISPN000031: MBeans were successfully registered to the platform MBean server.

      Cache Started : 1425220128292Name : key-3

      >>>>> Creating the dynamiccache key-3 and printing current list of caches

      withReplicationQueue

      key-3

       

       

       

       

      2015-03-01 19:58:48,292 INFO  [DynamicCacheListener] (remote-thread-0) Cache entry key-3 added in cache Cache 'withReplicationQueue'@IN00106404-21233

      2015-03-01 19:59:18,362 WARN  [CommandAwareRpcDispatcher] (OOB-1,IN00106404-21233) ISPN000220: Problems un-marshalling remote command from byte buffer

      java.lang.NullPointerException

          at org.infinispan.commands.RemoteCommandsFactory.fromStream(RemoteCommandsFactory.java:195)

          at org.infinispan.marshall.exts.ReplicableCommandExternalizer.fromStream(ReplicableCommandExternalizer.java:106)

          at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:147)

          at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:59)

          at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:389)

          at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:205)

          at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:152)

          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

          at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)

          at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:136)

          at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)

          at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)

          at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:206)

          at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)

          at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)

          at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)

          at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:665)

          at org.jgroups.JChannel.up(JChannel.java:708)

          at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1015)

          at org.jgroups.protocols.RSVP.up(RSVP.java:187)

          at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)

          at org.jgroups.protocols.FlowControl.up(FlowControl.java:370)

          at org.jgroups.protocols.FlowControl.up(FlowControl.java:381)

          at org.jgroups.protocols.tom.TOA.up(TOA.java:121)

          at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010)

          at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)

          at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:390)

          at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:774)

          at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:570)

          at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)

          at org.jgroups.protocols.FD.up(FD.java:255)

          at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)

          at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)

          at org.jgroups.protocols.Discovery.up(Discovery.java:379)

          at org.jgroups.protocols.MPING.up(MPING.java:181)

          at org.jgroups.protocols.TP.passMessageUp(TP.java:1399)

          at org.jgroups.protocols.TP$MyHandler.run(TP.java:1585)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

          at java.lang.Thread.run(Thread.java:745)

      Cache Stopped : 1425220158378 Name : key-3

      >>>>> Removing the dynamiccache key-3 and printing current list of caches

      withReplicationQueue

       

       

       

       

      2015-03-01 19:59:18,596 INFO  [CacheJmxRegistration] (remote-thread-0) ISPN000031: MBeans were successfully registered to the platform MBean server.

      Cache Started : 1425220158674Name : key-4

      >>>>> Creating the dynamiccache key-4 and printing current list of caches

      withReplicationQueue

      key-4

       

       

       

       

      2015-03-01 19:59:18,674 INFO  [DynamicCacheListener] (remote-thread-0) Cache entry key-4 added in cache Cache 'withReplicationQueue'@IN00106404-21233

      2015-03-01 19:59:48,711 WARN  [CommandAwareRpcDispatcher] (OOB-1,IN00106404-21233) ISPN000220: Problems un-marshalling remote command from byte buffer

      java.lang.NullPointerException

          at org.infinispan.commands.RemoteCommandsFactory.fromStream(RemoteCommandsFactory.java:195)

          at org.infinispan.marshall.exts.ReplicableCommandExternalizer.fromStream(ReplicableCommandExternalizer.java:106)

          at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:147)

          at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:59)

          at org.infinispan.marshall.core.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:389)

          at org.infinispan.marshall.core.ExternalizerTable.readObject(ExternalizerTable.java:205)

          at org.infinispan.marshall.core.JBossMarshaller$ExternalizerTableProxy.readObject(JBossMarshaller.java:152)

          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

          at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

          at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)

          at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:136)

          at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101)

          at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80)

          at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:28)

          at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:206)

          at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)

          at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)

          at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)

          at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:665)

          at org.jgroups.JChannel.up(JChannel.java:708)

          at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1015)

          at org.jgroups.protocols.RSVP.up(RSVP.java:187)

          at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)

          at org.jgroups.protocols.FlowControl.up(FlowControl.java:370)

          at org.jgroups.protocols.FlowControl.up(FlowControl.java:381)

          at org.jgroups.protocols.tom.TOA.up(TOA.java:121)

          at org.jgroups.protocols.pbcast.GMS.up(GMS.java:1010)

          at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:234)

          at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:390)

          at org.jgroups.protocols.pbcast.NAKACK2.handleMessage(NAKACK2.java:774)

          at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:570)

          at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)

          at org.jgroups.protocols.FD.up(FD.java:255)

          at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)

          at org.jgroups.protocols.MERGE2.up(MERGE2.java:209)

          at org.jgroups.protocols.Discovery.up(Discovery.java:379)

          at org.jgroups.protocols.MPING.up(MPING.java:181)

          at org.jgroups.protocols.TP.passMessageUp(TP.java:1399)

          at org.jgroups.protocols.TP$MyHandler.run(TP.java:1585)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

          at java.lang.Thread.run(Thread.java:745)

      Cache Stopped : 1425220188743 Name : key-4

      >>>>> Removing the dynamiccache key-4 and printing current list of caches

      withReplicationQueue

       

       

       

       

      2015-03-01 19:59:48,883 INFO  [CacheJmxRegistration] (remote-thread-0) ISPN000031: MBeans were successfully registered to the platform MBean server.

      Cache Started : 1425220188914Name : key-5

      >>>>> Creating the dynamiccache key-5 and printing current list of caches

      withReplicationQueue

      key-5

       

       

       

       

      2015-03-01 19:59:48,914 INFO  [DynamicCacheListener] (remote-thread-0) Cache entry key-5 added in cache Cache 'withReplicationQueue'@IN00106404-21233

      Cache Stopped : 1425220218959 Name : key-5

      2015-03-01 20:00:33,946 ERROR [InvocationContextInterceptor] (remote-thread-0) ISPN000136: Execution error

      org.infinispan.util.concurrent.TimeoutException: Replication timeout for IN00106404-12788

          at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:77)

          at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:547)

          at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:281)

          at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:198)

          at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitRemoveCommand(NonTxDistributionInterceptor.java:110)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:321)

          at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:402)

          at org.infinispan.interceptors.EntryWrappingInterceptor.visitRemoveCommand(EntryWrappingInterceptor.java:216)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitRemoveCommand(NonTransactionalLockingInterceptor.java:83)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

          at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

          at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:213)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitRemoveCommand(StateTransferInterceptor.java:156)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:166)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)

          at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)

          at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39)

          at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48)

          at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:95)

          at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:50)

          at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:172)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

          at java.lang.Thread.run(Thread.java:745)

      2015-03-01 20:00:33,946 ERROR [InboundInvocationHandlerImpl] (remote-thread-0) ISPN000260: Exception executing command

      org.infinispan.util.concurrent.TimeoutException: Replication timeout for IN00106404-12788

          at org.infinispan.remoting.transport.AbstractTransport.parseResponseAndAddToResponseList(AbstractTransport.java:77)

          at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:547)

          at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:281)

          at org.infinispan.interceptors.distribution.BaseDistributionInterceptor.handleNonTxWriteCommand(BaseDistributionInterceptor.java:198)

          at org.infinispan.interceptors.distribution.NonTxDistributionInterceptor.visitRemoveCommand(NonTxDistributionInterceptor.java:110)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:321)

          at org.infinispan.interceptors.EntryWrappingInterceptor.setSkipRemoteGetsAndInvokeNextForDataCommand(EntryWrappingInterceptor.java:402)

          at org.infinispan.interceptors.EntryWrappingInterceptor.visitRemoveCommand(EntryWrappingInterceptor.java:216)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitRemoveCommand(NonTransactionalLockingInterceptor.java:83)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

          at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112)

          at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.statetransfer.StateTransferInterceptor.handleNonTxWriteCommand(StateTransferInterceptor.java:213)

          at org.infinispan.statetransfer.StateTransferInterceptor.visitRemoveCommand(StateTransferInterceptor.java:156)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.CacheMgmtInterceptor.visitRemoveCommand(CacheMgmtInterceptor.java:166)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)

          at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:73)

          at org.infinispan.commands.AbstractVisitor.visitRemoveCommand(AbstractVisitor.java:37)

          at org.infinispan.commands.write.RemoveCommand.acceptVisitor(RemoveCommand.java:57)

          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:333)

          at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:39)

          at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:48)

          at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:95)

          at org.infinispan.remoting.InboundInvocationHandlerImpl.access$000(InboundInvocationHandlerImpl.java:50)

          at org.infinispan.remoting.InboundInvocationHandlerImpl$2.run(InboundInvocationHandlerImpl.java:172)

          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

          at java.lang.Thread.run(Thread.java:745)



      Regards

      Muthu