Hi,
We are using Infinispan v9.0.3 in standalone mode on a 4 CPU Linux machine with 32 GB physical memory with Java Heap Size of 25GB (both ms/mx set to 25GB).
The entries in cache have all fixed key size of 16 bytes and value object size varying from 1KB - 100KB.
The client is a multi-threaded process (24 concurrent threads) running on a remote Linux server connecting to Infinispan using Hotrod client. We also are using Single File Store for persistence.
These client threads are all performing a query first to check if a cache entry exists and then insert if entry absent or update if entry is present. There are indexes defined on 3 fields in the value object that are used for searching purposes.
The average number of requests being processed per day is between 5-6 Million. What we have been observing is that the performance degrades drastically after 2-3 days of continuous operation. The only way we are able to restore the performance is by stopping the client process and then stop/restart the Infinispan Server. This then performs at reasonable level for 2-3 days the starts to degrade again.
Please provide any insight if anyone has experienced similar issue or have a suggestion of how to resolve the problem. This seem to be purely due to the high volume of requests as we have an exact replica of this setup in a lower environment (stage) where the volumes are much lower (100K - 200K perday) and we don't have to recycle the processes at all.
Thanks in advance.
