Is WildFly leaking sockets when behind an HTTP proxy like nginx or lighttpd?

[bnigmann]

I have tried the following steps with both lighttpd and nginx, using both wildfly-8.0.0.Alpha4 and the latest nightly build (wildfly-8.0.0.Beta1-SNAPSHOT). The OS is Ubuntu 12.04, using OpenJDK 7u25-2.3.10-1ubuntu0.12.04.2. The same thing happens on a random, ancient Gentoo box I have, though.

For simplicity of this use case, I have the following scenario: nginx, listening on port 80, forwards all requests to WildFly on port 8080 on the same host. I was trying to post the relevant configuration piece in nginx here as well, but that breaks the forum editor and I can no longer post it. So I put it on pastebin: http://pastebin.com/7jyh8zzb

Steps to reproduce:

Unpack one of the vanilla tarballs mentioned above:

$ cd /opt/test

$ tar xvzf /tmp/wildfly-8.0.0.Alpha4.tar.gz

Then, without making any changes to the configuration files, start WildFly (I used a non-root user; "jboss"):

$ cd wildfly-8.0.0.Alpha4/

$ ./bin/standalone.sh

Open a second console window, find the PID of the WildFly process with jps and watch the output of lsof in a separate console window:

$ watch 'lsof -p <wildfly-pid> | grep protocol'

This only shows two lines with some random protocol JARs at first.

Now point a browser to http://localhost/ (on port 80, where nginx is listening). You should get the WildFly default welcome page, as expected, and the connection closes on the browser side. However, in the lsof watcher window, I now see entries like the following appear:

java 22416 jboss 309u sock 0,7 0t0 236117 can't identify protocol

java 22416 jboss 310u sock 0,7 0t0 236120 can't identify protocol

java 22416 jboss 311u sock 0,7 0t0 236124 can't identify protocol

Reload the page in the browser several times, and with every reload of the WildFly default page, more of these "socket zombies" appear. Googling for this output leads to several terms, like "half open TCP connection" but I will call them "zombie sockets" for now. I have found out the following so far:

1. These zombie sockets never go away once they are created. Only a WildFly restart clears them.
2. You don't even have to deploy your own WAR or EAR file, the default context will do.
3. Every new HTTP request adds a new zombie socket, until the user you are running WildFly as runs out of allowed file handles and Java throws "Too many open files" errors left and right.
4. I had an internal system running on WildFly in this setup for a few hours, with people accessing it, and it accumulated over 10,000 of these zombie sockets. So increasing the user's file limits is not an option, as this grows forever.
5. There are no active sockets showing in netstat.
6. Restarting nginx does not clear those up; these zombies are owned by the java process and my jboss user.
7. Pointing the browser directly to WildFly on http://localhost:8080/ does not show this problem and there are no zombie sockets.(However the goal is to have WildFly running as non-provileged user, so requests to port 80 need to be forwarded from the proxy.)

Can anyone reproduce this or suggest anything else I could try to trouble-shoot this?

[jaikiran]

Can you post the output of

netstat -a

on the server as well as the client

[ctomc]

Hi,

can you try with latest nightly build of wildfly that included Undertow beta 13 with many improvements in areas you have problems.

you can find nightly builds here WildFly nightly builds available

---

tomaz

[bnigmann]

Hi Tomaz,

I already tried it with a nightly snapshot from a couple of days ago, that contains Undertow beta 13:

2013-09-25 09:47:05,791 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 47) JBAS017502: Undertow 1.0.0.Beta13 starting

2013-09-25 09:47:05,790 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-3) JBAS017502: Undertow 1.0.0.Beta13 starting

2013-09-25 09:47:06,032 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-8) JBAS017525: Started server default-server.

This is still happening in beta 13.

[bnigmann]

I actually ran the client and the server on the same host (but the problem also happens when running on different hosts), using wget for the connections. After I had called wget a few times, the "zombie processes" were piling up:

# lsof -p 4976 | grep identify

java    4976 root  522u  sock                0,7      0t0   314839 can't identify protocol

java    4976 root  524u  sock                0,7      0t0   312937 can't identify protocol

java    4976 root  525u  sock                0,7      0t0   314886 can't identify protocol

java    4976 root  526u  sock                0,7      0t0   312951 can't identify protocol

java    4976 root  527u  sock                0,7      0t0   312955 can't identify protocol

java    4976 root  528u  sock                0,7      0t0   312960 can't identify protocol

java    4976 root  529u  sock                0,7      0t0   314912 can't identify protocol

java    4976 root  530u  sock                0,7      0t0   314921 can't identify protocol

java    4976 root  531u  sock                0,7      0t0   312971 can't identify protocol

java    4976 root  532u  sock                0,7      0t0   314939 can't identify protocol

I pasted the netstat output here: http://pastebin.com/3wCCj2hY

[jeryma]

We also encountered this problem, I hope someone will help us solve this problem. mark

[jeryma]

After I have used nightly builds edition,this problem also exists.

[dmlloyd]

Thanks for the netstat output Bernd, this will give us something to start with.

[bnigmann]

David,

Here is a netstat diff of the "netstat -an" output before I started WildFly and the output after WildFly was started and 4 HTTP requests to the root context were made:

# diff before.log after_req.log

6a7

> tcp        0      0 127.0.0.1:9990          0.0.0.0:*               LISTEN    

8a10,11

> tcp        0      0 127.0.0.1:9999          0.0.0.0:*               LISTEN    

> tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN    

66a69,70

> unix  2      [ ]         STREAM     CONNECTED     1246953 

> unix  2      [ ]         STREAM     CONNECTED     1247253

The lsof output for that WildFly process shows (among other things) four "zombie sockets", one per request made:

# lsof -p 15045 | grep identify

java    15045 root  324u  sock    0,7      0t0  1246985 can't identify protocol

java    15045 root  325u  sock    0,7      0t0  1246995 can't identify protocol

java    15045 root  326u  sock    0,7      0t0  1247277 can't identify protocol

java    15045 root  327u  sock    0,7      0t0  1247006 can't identify protocol

For what it's worth, the "zombie socket" file descriptors appear to always be starting with 324u, 325u, 326u and so on (in my experiments at least).

I'm not sure how helpful the netstat output is, as all that changed are the added listener ports for HTTP and the admin console, plus two unix sockets. All of these disappear again after I stop Wildfly and compare netstat again:

# diff before.log after_jboss.log

(no output)

If there is any other info I can provide, please let me know.

[swd847]

This should be fixed in Undertow upstream, which should make it into Wildfly in a few hours

[shuminli]

we have been seeing this behavior in the HTTPS connector as well.  should the upstream fix you mentioned resolve the socket leak on that connector too?

Red Hat Enterprise Linux Server release 6.4 (Santiago)

jdk1.7.0_40

[bnigmann]

I can confirm this. In today's nightly build, Undertow was upgraded from 1.0.0.Beta13 to 1.0.0.Beta15. The zombie socket problem is no longer happening in my test environment. Thank you guys!