1 Reply Latest reply on Oct 8, 2018 2:51 AM by galder.zamarreno

    Infinispan v9.0.3 performance degradation




      We are using Infinispan v9.0.3 in standalone mode on a 4 CPU Linux machine with 32 GB physical memory with Java Heap Size of 25GB (both ms/mx set to 25GB).


      The entries in cache have all fixed key size of 16 bytes and value object size varying from 1KB - 100KB.


      The client is a multi-threaded process (24 concurrent threads) running on a remote Linux server connecting to Infinispan using Hotrod client. We also are using Single File Store for persistence. 


      These client threads are all performing a query first to check if a cache entry exists and then insert if entry absent or update if entry is present. There are  indexes defined on 3 fields in the value object that are used for searching purposes.


      The average number of requests being processed per day is between 5-6 Million. What we have been observing is that the performance degrades drastically after 2-3 days of continuous operation. The only way we are  able to restore the performance is by stopping the client process and then stop/restart the Infinispan Server. This then performs at reasonable level for 2-3 days the starts to degrade again. 


      Please provide any insight if anyone has experienced similar issue or have a suggestion of how to resolve the problem. This seem to be purely due to the high volume of requests as we have an exact replica of this setup in a lower environment (stage) where the volumes are much lower (100K - 200K perday)  and we don't have to recycle the processes at all.


      Thanks in advance.