We run Wildfly 9 and we've got a reverse proxy in front of it, on a different machine.
After about 1.8 million requests (which is 8 hours of use on our application) we're seeing that some connections are not established which results in some requests not being handled.
Our reverse proxy reports a http 503 back to the client when the request is not possible to be served. Most request still go through, but lets say about 1 in 50 or 1 in 100 requests fail.
Both machines run Ubuntu. Analyzing tcp traffic shows that the SYN package arrives at the machine running Wildfly, but there's no response from this machine.
Of course these requests are not being logged in the Wildfly access log, nor are there any error messages in the server log.
Things we've already checked are; Thread states (threads are not being stuck in wait), memory & cpu and open file descriptors. We're not seeing anything out of the ordinary here.
When the problem occurs, we offer Wildfly some rest to recover (no requests for about 30 min - 1 hour), after that, we start sending requests again and we get 503 errors within half an hour.
If we restart Wildfly completely it takes about 1.8 million requests again for the 503 to show up.
Any ideas what we could try?
What would be the next logical place to investigate this issue?
The problem appeared to be around our reverse proxy apache2 and the firewall.
After introducing keepalive the problem seems to be resolved.