I am trying to identify if behaviour that I am observing is correct or if WildFly is leaking file handle descriptors.
During our standard performance testing after upgrading from WildFly 11 to 14 we ran into an issue regarding too many open files. After digging into things a bit more it looks like it is actually the number of pipes that WildFly has open that is increasing.
To help reproduce the problem I have created a simple JSF 2.2 application that contains a large image (100mb to simplify testing). I can make this available if required.
I am retrieving the image using the standard JSF resource URL: /contextroot/javax.faces.resource/css/images/big-image.png.xhtml
And have also tried adding omnifaces and using the unmapped resource handler URL: /contextroot/javax.faces.resource/css/images/big-image.png
Adding Omnifaces did not change the behaviour I am seeing, and I have only included it as we first thought that it might have been a contributing factor.
Behaviour I am seeing
- WildFly starts and jstack reports that it has two threads matching default task-* which is the default value for task-core-threads
- If I send in 5 concurrent requests for my large image, 3 new default task-* threads are spawned to serve the requests. 3 new Linux pipes will also be created.
- If I stop my requests and wait for 2 minutes (the default value for task-keepalive) 3 of the threads will be removed. The pipes remain open.
- Periodically - I believe about every 4.5 minutes some kind of clean up occurs and the pipes that were left over from the step above are removed.
- However... If one of the original 2 worker threads is removed, e.g. task-1, task-3 and task-4 are removed, leaving task-2 and task-5, the pipe associated with task-1 is never cleaned up.
Over time these pipes add up and as far as I can tell they are never removed. Is this a leak somewhere, and if so where? JSF? WildFly? Undertow?
Things I have tried
- I have used this tool to try and identify the file leaks: http://file-leak-detector.kohsuke.org/
- WildFly 14, 17 and 18 - All exhibit the same behaviour
- With and without Omnifaces (2.7 and 3.3) - Changes the stack trace from the file leak tool, but effectively the same behaviour
- Changing the min and max threads to be the same - this prevents handles building up, but I'd rather not go down this route
I am running WildFly on CentOS 7.7, we have seen the same behaviour in RHEL 7.7.
Java version is openjdk version "1.8.0_222"
I have attached
- lsof output of a freshly restarted WildFly 18 with my sample application (no omnifaces) deployed
- lsof output after a few cycles of 5 concurrent requests pause for minutes, 5 concurrent requests. I waited 5 minutes after the test to let any clean up happen.
- A sample stacktrace from the file-leak-detector tool
- jstack output showing which threads are still active after the cycles of requests