0 Replies Latest reply on Dec 12, 2016 11:10 PM by Srinivasan Duraiswamy

    WildFly 8.2 Slowing Down Over Time

    Srinivasan Duraiswamy Newbie

      Hi

       

      We are using WildFly 8.2 with Java HotSpot 7 in our prod environment.

      • The web service application runs smooth for 5 days and after that the performance starts degrading slowly - by 30-40 milliseconds every day. During first 5 days the avg response time of the web service is 200 milliseconds. At the end of 10th day it goes to 400-500 milliseconds and we are forced to restart since performance is critical.
      • We use Infinispan as the distributed cache.
        • We store both mortal and immortal data in the memory.
        • We don't use the Infinispan sub-system that comes with WildFly but we use it as a library.
        • Mortal data lives for 2 days and on a given day we add close to 2 GB of mortal data to the memory.
      • Since we store lot of data in memory we have the following setup for heap:
        • Heap Size : 15GB
        • GC Policy : CMS
        • Min and Max heap both set to 15GB
        • New size and max new size both set to 3 GB.
        • Full GC initiation ratio : 70%
      • Server Env and Threads:
        • Sun Solaris with 256 Core
        • 4096 default task threads
        • 512 default IO threads
        • 512 MSO threads
      • A full GC happens every day and half. After a full GC the heap size reduces to 2 GB.
      • We analysed both heap and thread dump and we didn't find anything. There are no blocked threads nor there is any memory leak. Also the CPU utilisation never goes beyond 5%.
      • Only thing we found is, the calls to external systems (database and API calls) are slowing down considerably.
        • We use Spring DAO template to connect to Oracle DB.
        • Spring web service template for connecting to external web service.
        • Apache HTTP client commons to connect to an API.
      • Once we restart everything is back to normal for the next 5 or so days.

       

      Any pointers to resolve this issue?

       

      Rgds

      Srini