Eviction Not happening on 5.1.4.Final

[bramalingam81]

This is my Eviction Configuration.

<eviction strategy="LIRS" maxEntries="1800000" />

<expiration maxIdle="240000" wakeUpInterval="10000"

lifespan="360000" reaperEnabled="true" />

JAVA_OPTS="$JAVA_OPTS -Xms2048m -Xmx2048m -Xss1024k -XX:PermSize=512m -XX:MaxPermSize=512m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:ParallelGCThreads=23 -XX:MaxTenuringThreshold=10 -XX:+HeapDumpOnOutOfMemoryError -Xloggc:gc_server_log.log -XX:-UseGCOverheadLimit -Djava.endorsed.dirs='$JAVA_ENDORSED_DIRS' -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+PrintGCTaskTimeStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -verbose:gc -Xloggc:gc_server_log.log"

I have set my tomcat to max of 2GB of memory. I simulated a load test for 12 lakh keys each unique entry (key+value) being 120 character, load test ran out of memory after 2.7 lakhs entries.

When I looked in to the heap histogram, I found that all the entries are created as Immortal. As these are immortal, I believe the eviction is not happening !

8:              209370  5024880 org.infinispan.container.entries.ImmortalCacheEntry

13:             209370  3349920 org.infinispan.container.entries.ImmortalCacheValue

45:             1142    73088   org.infinispan.commons.hash.MurmurHash3$State

69:             281     33720   org.infinispan.marshall.jboss.ExtendedRiverMarshaller

75:             1113    26712   org.infinispan.io.ExposedByteArrayOutputStream

76:             1107    26568   org.infinispan.io.ByteBuffer

While I was debugging, I looked into the class InternalCacheEntry :

@Override

   public InternalCacheEntry create(Object key, Object value, EntryVersion ignored, long created, long lifespan, long lastUsed, long maxIdle) {

      if (lifespan < 0 && maxIdle < 0) return new ImmortalCacheEntry(key, value);

      if (lifespan > -1 && maxIdle < 0) return new MortalCacheEntry(key, value, lifespan, created);

      if (lifespan < 0 && maxIdle > -1) return new TransientCacheEntry(key, value, maxIdle, lastUsed);

      return new TransientMortalCacheEntry(key, value, maxIdle, lifespan, lastUsed, created);

   }

As per my configuration, both lifespan and maxIdle are > 0, so Ideally it should have created TransientMortalCacheEntry. But as per the heap, its creating ImmortalCacheEntry/Value.

Can someone help me on this issue ?

[nadirx]

Balaji, after looking up what a lakh is (100000 one hundred thousand), it is obvious what the problem is: you set maxEntries to be 1800000 (one million eight hundred thousand) which is much higher than 2.7 lakh (two hundred and seventy thousand)

Eviction in Infinispan is based on number of entries, not memory occupation.

[bramalingam81]

Hi Tristan,

Thank you on the response. I reduced the entry (key+value) to be of 120 characters and I was able to have the load test succeeded for 5 lakh entries, and total heap occupied was around 150MB.

My concern still is ON why ImmortalCacheEntry/Value is getting created. Based on my configuration, I was expecting it to Create TransientMortalCacheEntry/Value. Can yo please clarify on this ? As I was looking into the source of ImmortalCacheEntry to find that these objects will NEVER be evicted from memory. As I have the lifespan and maxidle time > 0 , I believe TransientMortalCacheEntry/Value should be created.

Thank you !

[nadirx]

Eviction and expiration are not the same and the behaviour is correct.

Expiration only affects TransientMortalCacheEntries since they have a lifetime explicitly set on them by the user.

Eviction doesn't care about mortality or immortality of an entry (it works on InternalCacheEntries). Eviction does not necessarily mean "death" of an entry: if passivation is on, it will simply be put in the cachestore.

[bramalingam81]

Hi Tristan,

I am still struggling with OOM Issue ! Can you help me out with the issue ?

This is the servlet by which data is being populated to cache !

public class CacheServlet extends HttpServlet

{

    @Override

    @SuppressWarnings("unchecked")

    protected void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException

    {

        @SuppressWarnings("rawtypes")

        Cache cache = ContextFactory.getInstance().getCache("application-cache");

        @SuppressWarnings("rawtypes")

        TransactionManager tm = ((CacheImpl) cache).getAdvancedCache().getTransactionManager();

        try

        {

            String value =

                    "The sharp fall in fourth quarter GDP growth to 5.3 percent against market expectations of 6.1 percent places an additional burden on the RBI Governor D Subbarao. The central bank is fighting inflation, the rupee’s fall, liquidity and government borrowing. The below-trend GDP growth has all and sundry clamouring for the RBI to ease monetary policy. The fact is that the RBI cannot just focus on easing policy to raise GDP growth expectations. Every action of the RBI will have an effect on one macro factor or the other and the RBI has to do a fine balancing act.Liquidity that continues to be in deficit mode with banks borrowing around Rs 1,00,000 crore from the RBI on a daily average basis despite RBI bond buying of around Rs 50,000 crore in April 2012 to date.Government borrowing with weekly bond auctions of Rs 15,000 crore in an environment of tight liquidity and a weakening rupee.";

            String key = request.getParameter("key");

            tm.begin();

            System.out.println("Txn within TM = " + tm.getTransaction().toString());

            cache.put(key, value + "  " + key + "  " + System.currentTimeMillis());

            logger.info("added to cache = " + key);

            response.getWriter().write("SUCCESS");

            response.getWriter().close();

            // Validate the entry is now in the cache

            tm.commit();

        }

        catch (Exception e)

        {

            logger.error(e, e);

            try

            {

                if (tm != null)

                {

                    tm.rollback();

                }

            }

            catch (Exception ex)

            {

            }

        }

    }

}

//Singleton Factory

public class ContextFactory

{

    private EmbeddedCacheManager cacheManager = null;

    @SuppressWarnings("rawtypes")

    private Cache cache = null;

    private ContextFactory()

    {

        try

        {

            cacheManager = new DefaultCacheManager("infinispan-cluster.xml");

        }

        catch (IOException e)

        {

            logger.error("Error while creating singleton instance =", e);

        }

    }

    public static ContextFactory getInstance()

    {

        try

        {

            if (g_self == null)

            {

                g_self = new ContextFactory();

            }

            return g_self;

        }

        catch (Exception e)

        {

            logger.error("Error while creating singleton getInstance =", e);

            throw new RuntimeException("Error in creation new instance of ContextFactory", e);

        }

    }

    @SuppressWarnings("rawtypes")

    public Cache getCache(String cacheName)

    {

        if (cache != null)

        {

            return cache;

        }

        cache = cacheManager.getCache();

        if (cache.getStatus() != ComponentStatus.RUNNING)

        {

            cache.start();

        }

        return cache;

    }

}

Total Characters = Key + Value = would approximate equal to 925 characters.  This will be the size of 1 entry.

public class CacheReaderServlet extends HttpServlet {

@Override

    protected void doGet(HttpServletRequest request,

                              HttpServletResponse response) throws IOException {

                    String key = request.getParameter("key");

                    Cache cache = ContextFactory.getInstance()

                                        .getCache("application-cache");

                    logger.info("getting from cache = " + key);

                    response.getWriter().write("getting from cache = " + cache.get(key));

        response.getWriter().close();

            }

  }

			<eviction strategy="LIRS" maxEntries="1800000" />
		<expiration maxIdle="240000" wakeUpInterval="10000"
			lifespan="360000" reaperEnabled="true" />

	<transaction useSynchronization="true" transactionMode="TRANSACTIONAL"
			useEagerLocking="false"
			transactionManagerLookupClass="org.infinispan.transaction.lookup.JBossStandaloneJTAManagerLookup"
			use1PcForAutoCommitTransactions="false" syncRollbackPhase="false"
			syncCommitPhase="true" lockingMode="OPTIMISTIC" eagerLockSingleNode="true"
			cacheStopTimeout="30000" autoCommit="false">

JMeter load script is being setup to insert 5,000,000 of Entries. As each entry is of around 925 characters, maxEntries configured is 1,800,000*925 = 1590MB. This time my tomcat memory was capped @ 3GB. So, total heap available was 3*1024 - 256MB(NewSize) =2816MB. When I looked into the heap, all was containing only ImmortalCacheEntry/Values objects. As the size of cache is configured to 1,800,000 entries, I was expecting the cache to start evicting the entries after it reached 1,800,000. I have NOT configured any loader, So that entry should Ideally be removed from cache.  Though the full GC runs, the ImmortalCacheEntry/Values are not getting garbage collected.

Please let me know if I am missing something. I am continuously simulating various loads to ensure that it doesn't runs out of memory.

Thanks !

[nadirx]

Hi Balji, could you please attach your configuration XML so that I can see it properly.

Thanks

Tristan

[bramalingam81]

Hi Tristan,

I resolved the issue ! There was a bug in my code and the configuration was right ! My Code was NEVER using the configuration.  After I fixed the below code, Expiration/Evication/Passivation  seems to be happening the way I expected. I too verified the same by enabling Trace logs on org.infinispan.eviction package !

@SuppressWarnings("rawtypes")

    public Cache getCache(String cacheName)

    {

        if (cache != null)

        {

            return cache;

        }

        cache = cacheManager.getCache(); --> cacheName passed in the parameter is NEVER getting used

        if (cache.getStatus() != ComponentStatus.RUNNING)

        {

            cache.start();

        }

        return cache;

    }

Thank you on the support !

However, I feel that the response time has become slow. Earlier, because the cache configuration was not used, it was taking the DummyTransactionManager with DummyTransaction. I believe it has slowed down because of JBossTransactionManager.

These are our requirements for the Distributed Cache !

1. Total 3 nodes are configured

2. As soon as the key gets added on to cache on one node, we want that key to be immediately (within 1sec) available in any of the other 2 nodes. So, we have configured the <sync> replication with numOwners = 2.

3.  We do not have a case of the same key being updated my concurrent request on to different nodes, Unless there is a collision between the replication and the request landing at the same time for the same key !

If the overall throughput will get affected based on the transaction, I am just curious to understand on whether Non-Transactional Cache would solve my purpose.  I have attached the configuration file.

Please let me know on this.

Thanks !

Balaji

[nadirx]

It was as I suspected then. You were using a default cache, but the eviction configuration was on a named cache.

Don't use a TM if you don't need transactions.

Also, when you say "As soon as the key gets added on to cache on one node, we want that key to be immediately (within 1sec) available in any of the other 2 nodes.", I don't understand if this is a resilience requirement (i.e. you can accept a data loss of max 1 second) or merely that you want the other nodes to be able to see the data as soon as possible. In the latter case: the data is available to all nodes as soon as it is "put". You don't need sync: if a node doesn't "own" a key it will do an RPC to get it from the owner.

Tristan

[bramalingam81]

Hi Tristan,

Okay. Now that I understand that if a node does not "own" a key, it would do an RPC to get it from the owner. However, this does not guarantee fail-safe cluster. There are totally 3 nodes in the cluster. In order to ensure a fail-safety of the cache data, I have enabled the sync to replicate the same data atleast in any one of the other 2 nodes. So, in total the same key would be available in atleast 2 nodes.

As cache store does the RPC to pull the key from the owner, I believe I can make the replication async instead of sync.

I am not too clear on when to use Transactions/Locks and when NOT to use them.  Could you please clarify me on that ?

Let me brief you a scenario !

1. First request lands on Node 1. Key K1 already existed on Nodes N1 , N2.This request updates key K1 on to Node 1.

2. As the cache is configured to be sync, it replicates the same key to Node N2. During replication would it update the key K1 on Node N2.

3.  Meantime, when a new request lands on Node N2, it would also try to update the key K1.

Now comes the collision. How does Infinispan handle this scenario ? Can this scenario be accomplished without Transaction/Locks ?

Apologize me If my question is silly !

Thanks !

Balaji

[nadirx]

I am not too clear on when to use Transactions/Locks and when NOT to use them.  Could you please clarify me on that ?

Let me brief you a scenario !

1. First request lands on Node 1. Key K1 already existed on Nodes N1 , N2.This request updates key K1 on to Node 1.

2. As the cache is configured to be sync, it replicates the same key to Node N2. During replication would it update the key K1 on Node N2.

3.  Meantime, when a new request lands on Node N2, it would also try to update the key K1.

Now comes the collision. How does Infinispan handle this scenario ? Can this scenario be accomplished without Transaction/Locks ?

A transaction is needed when you want to make multiple operations appear as a single atomic operation, i.e. changing two keys at once. In your case you can probably use the putIfAbsent() methods to avoid Node 2 from overwriting K1 if it already exists.

Tristan

[bramalingam81]

Hi Tristan,

I have made the cache as NON_TRANSACTIONAL. After that I have started getting issues. I believe its because of the collision scenario that I have mentioned above.

ERROR [OOB-14,localhost-40374] [2012-06-05 07:56:32,190] (InvocationContextInterceptor.java:handleAll:146) - ISPN000136: Execution error

org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [60 seconds] on key [6594469] for requestor [Thread[OOB-14,localhost-40374,5,Thread Pools]]! Lock held by [(another thread)]

        at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:216)

        at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLockNoCheck(LockManagerImpl.java:199)

        at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:114)

        at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutKeyValueCommand(NonTransactionalLockingInterceptor.java:67)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)

        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

        at org.infinispan.interceptors.StateTransferLockInterceptor.handleWithRetries(StateTransferLockInterceptor.java:213)

        at org.infinispan.interceptors.StateTransferLockInterceptor.handleWriteCommand(StateTransferLockInterceptor.java:181)

        at org.infinispan.interceptors.StateTransferLockInterceptor.visitPutKeyValueCommand(StateTransferLockInterceptor.java:152)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

        at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:131)

        at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:90)

        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

        at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)

        at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:61)

        at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:70)

        at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:127)

        at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:136)

        at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithRetry(InboundInvocationHandlerImpl.java:162)

        at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:114)

        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommand(CommandAwareRpcDispatcher.java:221)

        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:201)

        at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:447)

        at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:354)

        at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:230)

        at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:542)

        at org.jgroups.JChannel.up(JChannel.java:715)

        at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1015)

        at org.jgroups.protocols.pbcast.StreamingStateTransfer.up(StreamingStateTransfer.java:262)

        at org.jgroups.protocols.RSVP.up(RSVP.java:189)

        at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)

        at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)

        at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)

        at org.jgroups.protocols.pbcast.GMS.up(GMS.java:884)

        at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)

        at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:747)

        at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:417)

        at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:601)

        at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140)

        at org.jgroups.protocols.FD.up(FD.java:273)

        at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282)

        at org.jgroups.protocols.MERGE3.up(MERGE3.java:290)

        at org.jgroups.protocols.Discovery.up(Discovery.java:355)

        at org.jgroups.protocols.TP.passMessageUp(TP.java:1194)

        at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1749)

        at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1731)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

        at java.lang.Thread.run(Thread.java:662)

Also started getting timeout exception :

ERROR [ajp-bio-22009-exec-15] [2012-06-05 07:16:04,599] (InvocationContextInterceptor.java:handleAll:146) - ISPN000136: Execution error

org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to localhost-25440

        at org.infinispan.util.Util.rewrapAsCacheException(Util.java:525)

        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:172)

        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:489)

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:161)

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:183)

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:240)

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:227)

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:222)

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:217)

        at org.infinispan.interceptors.DistributionInterceptor.handleWriteCommand(DistributionInterceptor.java:506)

        at org.infinispan.interceptors.DistributionInterceptor.visitPutKeyValueCommand(DistributionInterceptor.java:275)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

        at org.infinispan.interceptors.DeadlockDetectingInterceptor.handleDataCommand(DeadlockDetectingInterceptor.java:117)

        at org.infinispan.interceptors.DeadlockDetectingInterceptor.visitPutKeyValueCommand(DeadlockDetectingInterceptor.java:74)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)

        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

        at org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:89)

        at org.infinispan.interceptors.ActivationInterceptor.visitPutKeyValueCommand(ActivationInterceptor.java:70)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

        at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:212)

        at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:147)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

        at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutKeyValueCommand(NonTransactionalLockingInterceptor.java:68)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

        at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)

        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

        at org.infinispan.interceptors.StateTransferLockInterceptor.handleWithRetries(StateTransferLockInterceptor.java:213)

        at org.infinispan.interceptors.StateTransferLockInterceptor.handleWriteCommand(StateTransferLockInterceptor.java:181)

        at org.infinispan.interceptors.StateTransferLockInterceptor.visitPutKeyValueCommand(StateTransferLockInterceptor.java:152)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

        at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

        at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:131)

        at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:90)

        at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)

        at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)

        at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:345)

        at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1006)

        at org.infinispan.CacheImpl.put(CacheImpl.java:702)

        at org.infinispan.CacheImpl.put(CacheImpl.java:694)

        at org.infinispan.CacheSupport.put(CacheSupport.java:53)

        at in.verse.cache.distributed.CacheServlet.doGet(CacheServlet.java:36)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225)

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)

        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)

        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)

        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:200)

        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:565)

        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:307)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

        at java.lang.Thread.run(Thread.java:662)

Caused by: org.jgroups.TimeoutException: timeout sending message to localhost-25440

        at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:359)

        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:270)

        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:165)

        ... 67 more

Can you help me with appropriate configuration changes ?

Thanks !

Balaji

[nadirx]

Balaji, you are not using cache.putIfAbsent(). Could you try that ?

Tristan