We are using WildFly 8.2 with Java HotSpot 7 in our prod environment.
The web service application runs smooth for 5 days and after that the performance starts degrading slowly - by 30-40 milliseconds every day. During first 5 days the avg response time of the web service is 200 milliseconds. At the end of 10th day it goes to 400-500 milliseconds and we are forced to restart since performance is critical.
We use Infinispan as the distributed cache.
We store both mortal and immortal data in the memory.
We don't use the Infinispan sub-system that comes with WildFly but we use it as a library.
Mortal data lives for 2 days and on a given day we add close to 2 GB of mortal data to the memory.
Since we store lot of data in memory we have the following setup for heap:
Heap Size : 15GB
GC Policy : CMS
Min and Max heap both set to 15GB
New size and max new size both set to 3 GB.
Full GC initiation ratio : 70%
Server Env and Threads:
Sun Solaris with 256 Core
4096 default task threads
512 default IO threads
512 MSO threads
A full GC happens every day and half. After a full GC the heap size reduces to 2 GB.
We analysed both heap and thread dump and we didn't find anything. There are no blocked threads nor there is any memory leak. Also the CPU utilisation never goes beyond 5%.
Only thing we found is, the calls to external systems (database and API calls) are slowing down considerably.
We use Spring DAO template to connect to Oracle DB.
Spring web service template for connecting to external web service.
Apache HTTP client commons to connect to an API.
Once we restart everything is back to normal for the next 5 or so days.