Open connections with Apache 2.2+mod_proxy_ajp+JBoss/Tomcat

[skeumurian]

Hello,

I've been trying to track down a problem on our server for a while now and thought I'd check in here to see if anyone has a solution. I have Apache 2.2.2 setup as a front end for JBoss 4.0.2 on the same server. I have a rather long list of named virtual hosts being served on this one system. I'm using mod_proxy_ajp to route traffic to multiple web applications on the server. I'm running into a situation where the number of connections from Apache to the Tomcat AJP connector (localhost:8009) is forever growing and eventually people are blocked from accessing the server. The connections are left in the "ESTABLISHED" state as viewed by netstat.

tcp4 0 0 localhost.8009 localhost.49163 ESTABLISHED
tcp4 0 0 localhost.49163 localhost.8009 ESTABLISHED

I've tried using Apache with the worker MPM as well as the pre-fork MPM and the problem seems to occur with both, but becomes a problem a lot sooner with the pre-fork MPM.

I tried editing jboss-4.0.2/server/default/deploy/jbossweb-tomcat55.sar/server.xml to include a connectionTimeout value of 20000 milliseconds. This timeout works, but then instead of the connections being left in the "ESTABLISHED" state they are now left as "CLOSE_WAIT." I've tried changing various values on the Apache side, such as specifying smax, ttl, etc. to no avail.

I'm defining the proxy to my application like this:

<Location /cdf>
 ProxyPass ajp://localhost:8009/cdf smax=30 ttl=30 retry=1
 ProxyPassReverse ajp://localhost:8009/cdf
</Location>

This is included in each named virtual host that needs access to the application. Other virtual hosts have their own Location entries that proxy pass to other applications.

My http-mpm.conf is setup like this for the worker MPM:

<IfModule mpm_worker_module>
 StartServers 2
 MaxClients 300
 MinSpareThreads 25
 MaxSpareThreads 300
 ThreadsPerChild 25
 MaxRequestsPerChild 0
</IfModule>

and like this for the prefork:

<IfModule mpm_prefork_module>
 StartServers 15
 MinSpareServers 51
 MaxSpareServers 65
 MaxClients 500
 MaxRequestsPerChild 0
</IfModule>

I have a test server that I has a simpler version of the same setup on (no named virtual hosts) and I'm seeing this behavior there as well. I've tried creating a test application that only includes a basic index.html and it appears to behave the same way. Am I missing something? Am I using mod_proxy_ajp incorrectly?

Thanks,
Stephen

[gustavosoares]

Have you try using mod_proxy_http to foward connection to the http connector on tomcat instead of using mod_proxy_ajp? I have faced some performance issues using mod_proxy_ajp and when I switch to mod_proxy_http I had no problem at all.

Gustavo

[skeumurian]

Thanks for the suggestion. I haven't tried that yet, but I can test that on my other server. I'll post the rests here when I have them.

For the most part the performance of proxy_ajp has been fine for what we need, but these dangling connections are annoying.

-= Stephen =-

[venus2]

Any luck? one of our user is having same problem. Thanks for sharing.

[yositunel]

Same issue. Basically, I have to restart apache often since it seems the firewall (SonicWall) kills the idle connections. Any thoughts?

[yositunel]

HI -
I'm not sure if this is totally related to your problem, but I have similar issues. If there's a firewall in between your apache and application server, then the solution that worked for me was keepalive On.

The default Linux keepalive settings will need to be modified to match those of the allowable idle connection time out on the firewall.

LINUX DEFAULTS:
# cat /proc/sys/net/ipv4/tcp_keepalive_time
7200
# cat /proc/sys/net/ipv4/tcp_keepalive_intvl
75
# cat /proc/sys/net/ipv4/tcp_keepalive_probes
9

Apache settings:
ProxyPass .... keepalive=On

For additional info, see http://libkeepalive.sourceforge.net/docs/TCP-Keepalive-HOWTO

[shileiofchina]

I have met the same problem . I use mod_jk and After some times ,Os has too many 8009 connections being left in the "ESTABLISHED".

Please help me!