We are trying to run a load test with 1000 concurrent users that involves database interaction and publishing JMS messages.

The test runs fine for 1-1.5 hours and start to degrade in terms of response time. By instrumenting our application code we were able to determine that the network IO(both for JMS publish as well as database interaction) takes time as the test progresses.

Following is our environment details :

- Wildfly is 8.2.0 final

- JDK 1.8 u77.

- Windows 2012, 4 cores CPU and 8 GB RAM on Application server

- Database  and JMS (HornetQ embedded with Wildlfy) running on separate machine than Wildfly application server where load test is running.

- JMS configuration file attached[standalone-jms.xml]

- Wildfly/Undertow configuration file attached[standalone-app.xml]

Observations from the load test runs :

- CPU utilization is not more that 50%

- GC pauses (CMS collector) are not more than 0.5 sec.

- SQL trace on MSSQL database does not indicate queries taking long time

- Thread dumps does not indicate any threads in blocked state.

- The application which we are load testing is basically a chat application and has long poll requests that are held on server up to maximum of 30 seconds.

Any insight on what might be causing IO to take time over the time ? Is the IO thread configuration correct/optimal ?