Is WildFly leaking sockets when behind an HTTP proxy like nginx or lighttpd?
bnigmann Sep 26, 2013 11:06 PMI have tried the following steps with both lighttpd and nginx, using both wildfly-8.0.0.Alpha4 and the latest nightly build (wildfly-8.0.0.Beta1-SNAPSHOT). The OS is Ubuntu 12.04, using OpenJDK 7u25-2.3.10-1ubuntu0.12.04.2. The same thing happens on a random, ancient Gentoo box I have, though.
For simplicity of this use case, I have the following scenario: nginx, listening on port 80, forwards all requests to WildFly on port 8080 on the same host. I was trying to post the relevant configuration piece in nginx here as well, but that breaks the forum editor and I can no longer post it. So I put it on pastebin: http://pastebin.com/7jyh8zzb
Steps to reproduce:
Unpack one of the vanilla tarballs mentioned above:
$ cd /opt/test
$ tar xvzf /tmp/wildfly-8.0.0.Alpha4.tar.gz
Then, without making any changes to the configuration files, start WildFly (I used a non-root user; "jboss"):
$ cd wildfly-8.0.0.Alpha4/
$ ./bin/standalone.sh
Open a second console window, find the PID of the WildFly process with jps and watch the output of lsof in a separate console window:
$ watch 'lsof -p <wildfly-pid> | grep protocol'
This only shows two lines with some random protocol JARs at first.
Now point a browser to http://localhost/ (on port 80, where nginx is listening). You should get the WildFly default welcome page, as expected, and the connection closes on the browser side. However, in the lsof watcher window, I now see entries like the following appear:
java 22416 jboss 309u sock 0,7 0t0 236117 can't identify protocol
java 22416 jboss 310u sock 0,7 0t0 236120 can't identify protocol
java 22416 jboss 311u sock 0,7 0t0 236124 can't identify protocol
Reload the page in the browser several times, and with every reload of the WildFly default page, more of these "socket zombies" appear. Googling for this output leads to several terms, like "half open TCP connection" but I will call them "zombie sockets" for now. I have found out the following so far:
- These zombie sockets never go away once they are created. Only a WildFly restart clears them.
- You don't even have to deploy your own WAR or EAR file, the default context will do.
- Every new HTTP request adds a new zombie socket, until the user you are running WildFly as runs out of allowed file handles and Java throws "Too many open files" errors left and right.
- I had an internal system running on WildFly in this setup for a few hours, with people accessing it, and it accumulated over 10,000 of these zombie sockets. So increasing the user's file limits is not an option, as this grows forever.
- There are no active sockets showing in netstat.
- Restarting nginx does not clear those up; these zombies are owned by the java process and my jboss user.
- Pointing the browser directly to WildFly on http://localhost:8080/ does not show this problem and there are no zombie sockets.(However the goal is to have WildFly running as non-provileged user, so requests to port 80 need to be forwarded from the proxy.)
Can anyone reproduce this or suggest anything else I could try to trouble-shoot this?