I migrated recently from JBossAS 7.1.1 (Java 7) to Wildfly 13 (Java 8).
Almost everything was fine, but, before 2 days running, the server stopped working with a lot of errors of "Too many open files".
I've never received that before (that I remember).
Restarting the server solved the issue for the moment, but I wanted to know what caused that.
I digged around a bit and found that the docker container in which it was running allowed until 4096 open files, so, for now, I increased this limit in the wildfly container.
This error is not happening anymore, because I increased the limit a lot, but I created a cronjob to verify the top 10 processes that has more opened file descriptors.
Right after restarting, Wildfly have about 600, but, as time goes on, it increases, and sometimes decreases, but when comparing the results of a day with the day before, it increased by about 1000 opened file descriptors.
This is an example of the results after Wildfly was running for 5 days:
pid = 9570 with 6035 fds: java
pid = 8766 with 241 fds: node
pid = 8568 with 235 fds: mysqld
pid = 9058 with 89 fds: java
pid = 2033 with 59 fds: dockerd
pid = 9100 with 34 fds: docker-proxy
pid = 1953 with 22 fds: fail2ban-server
pid = 5303 with 21 fds: httpd
pid = 5270 with 21 fds: httpd
pid = 5212 with 21 fds: httpd
The java process related to wildfly has 6035 opened file descriptors, and this was a moment with very few online users (the other java process is logstash).
I've run the command ls -l /proc/9570/fd to see the opened file descriptor and saw 6045 lines, of which 5312 are similar to the following:
lr-x------ 1 1000 1000 64 Sep 27 03:20 999 -> /opt/jboss/wildfly/standalone/tmp/app.war/undertow6144806476267093537upload (deleted)
So I think there is some leak in the file upload process (although I don't receive errors).
Maybe this is related to this bug in Undertow [UNDERTOW-961] File descriptors leak in MultiPartParserDefinition - JBoss Issue Tracker (created at 19/Jan/17 7:36 AM and resolved at 16/Feb/17 6:02 PM), but I'm not sure (and the issue was solved, although I don't know which version of Undertow Wildfly 13 uses).
I don't remember receiving such issues in JBossAS 7.1.1 (or maybe there was, but in less intensity, in such a way that the service was always restarted before the number of opened file descriptors reached the limit).
I haven't changed the file upload process after the migration, and like I said, I don't receive error during file upload.
Actually, I started receiving about 5 or 10 errors per day like:
java.io.IOException: UT000128: Remote peer closed connection before all data could be read
after migrating, but it's very few of those errors, while the number of opened file descriptors increase by about 1000 per day. The error above happens when I call request.getParameter("some_param") in a servlet filter, but very rarely (and I couldn't simulate). I think this is not related to the problem of too many open files.