Hi
We are using WildFly 8.2 with Java HotSpot 7 in our prod environment.
- The web service application runs smooth for 5 days and after that the performance starts degrading slowly - by 30-40 milliseconds every day. During first 5 days the avg response time of the web service is 200 milliseconds. At the end of 10th day it goes to 400-500 milliseconds and we are forced to restart since performance is critical.
- We use Infinispan as the distributed cache.
- We store both mortal and immortal data in the memory.
- We don't use the Infinispan sub-system that comes with WildFly but we use it as a library.
- Mortal data lives for 2 days and on a given day we add close to 2 GB of mortal data to the memory.
- Since we store lot of data in memory we have the following setup for heap:
- Heap Size : 15GB
- GC Policy : CMS
- Min and Max heap both set to 15GB
- New size and max new size both set to 3 GB.
- Full GC initiation ratio : 70%
- Server Env and Threads:
- Sun Solaris with 256 Core
- 4096 default task threads
- 512 default IO threads
- 512 MSO threads
- A full GC happens every day and half. After a full GC the heap size reduces to 2 GB.
- We analysed both heap and thread dump and we didn't find anything. There are no blocked threads nor there is any memory leak. Also the CPU utilisation never goes beyond 5%.
- Only thing we found is, the calls to external systems (database and API calls) are slowing down considerably.
- We use Spring DAO template to connect to Oracle DB.
- Spring web service template for connecting to external web service.
- Apache HTTP client commons to connect to an API.
- Once we restart everything is back to normal for the next 5 or so days.
Any pointers to resolve this issue?
Rgds
Srini
