We recently started using this file leak detector tool File Leak Detector - on our standalone WildFly 9.0.2.Final deployment.
It has been quite helpful in helping us find resource leaks in applications and reducing "too many open files" issues on JBoss AS/WildFly servers we have seen over the years and had difficulty identifying the exact cause(s) of.
While I have a pretty good understanding of what most of the sockets opened correspond to, something that I'm less certain about would be the sockets opened by threads named "default I/O-###"
It is usually pretty easy to understand what application/process opened based on the stack trace and/or thread name; however, in this case the only stack trace information is this:
Trace: socket channel by thread:default I/O-### on [Date/Time] at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:135) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:266) at org.xnio.nio.NioTcpServer.accept(NioTcpServer.java:385) at org.xnio.nio.NioTcpServer.accept(NioTcpServer.java:52) at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:289) at org.xnio.ChannelListeners$10.handleEvent(ChannelListeners.java:286) at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) at org.xnio.nio.NioTcpServerHandle.handleReady(NioTcpServerHandle.java:53) at org.xnio.nio.WorkerThread.run(WorkerThread.java:539)
I do have a guess about this:
This corresponds to an I/O thread for a worker named "default" that is associated with one or more subsystems. In our case, I think most of this would be related to Undertow servicing requests [remote EJB, remote JMS, and web services]. I believe client/request volume may affect the number of sockets opened.
The thing that seems strange and concerns me is the combination of age and number of sockets opened by these threads. In particular, I find it odd when I see a lot of sockets that have remained open for an extended period of time [hours or days in some cases], especially when the client/request volume is low.
I'm wondering if there is something either in the IO subsystem or Undertow subsystem configuration that might explain what I am seeing, as I have used the default IO subsystem configuration that ships with WildFly and not done much configuration on Undertow. IO subsystem configuration is an area I still don't fully understand and did ask about previously in this post Tuning IO Subsystem in WildFly 9.0.2.Final but haven't gotten a response.
Any information that might help explain what I am seeing or recommendations about tuning the IO or Undertow subsystems to prevent open files issues would be very much appreciated.