After server restart AJP connections start increasing and staying in ESTABLISHED mod_cluster/1.3.2.Final

[wolberc]

Hi,

in our current setup we have an Apache LoadBalancer 2.4 in front of 2 Wildfly 8.2.0 Final Servers (standalone cluster).

On another server we are using a JBPM Process to trigger certain methods on the wildfly servers. This works perfectly when all servers are cleanly started. We have a test that triggers around 600.000 calls to the loadbalancer.

lsof -i shows around 60 connections all the time on the loadbalancer

Whenever we restart one of the wildfly servers  as soon as it gets balanced calls the AJP Connections are neither reused or closed any longer. It goes on until the server has too many open files which crashes the wildfly. The wildfly server that is not restarted does not have the problem ans still runs totally correct.

Does anybody have an idea on what to check and what to look out for to get the RootCause?

The infrastructure is running on AWS.

Those are the configs we currently use:

Virtual Host Config on Apache:

<VirtualHost ip:80>

  ServerName ip

  ## Vhost docroot

  DocumentRoot "/var/www/html"

  ## Directories, there should at least be a declaration for /var/www/html

  <Location "/">

    Require ip x.x.x.x

  </Location>

  <Location "/loc1">

    Require all granted

  </Location>

  <Location "/loc2">

    Require all granted

  </Location>

  <Location "/loc3">

    Require all granted

  </Location>

  <Location "/loc4">

    Require all granted

  </Location>

  <Location "/mod-cluster-manager">

    Require ip x.x y.y

    SetHandler mod_cluster-manager

  </Location>

  ## Logging

  ErrorLog "/var/log/apache2/error.log"

  LogLevel info

  ServerSignature Off

  CustomLog "/var/log/apache2/access.log" "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %D"

  ## Rewrite rules

  RewriteEngine On

  #redirect to real url

  RewriteCond %{REQUEST_FILENAME} -d

  RewriteRule .*[^/]$ /$1/

  ## Custom fragment

  KeepAliveTimeout 60

  MaxKeepAliveRequests 0

  ManagerBalancerName qa

  ServerAdvertise On

  EnableMCPMReceive

  RemoteIPHeader X-Forwarded-For

</VirtualHost>

The mod cluster config for wildfly:

        <subsystem xmlns="urn:jboss:domain:modcluster:1.2">

            <mod-cluster-config advertise-socket="modcluster" proxy-list="lb:80" balancer="qa" connector="ajp">

                <dynamic-load-provider>

                    <load-metric type="cpu"/>

                </dynamic-load-provider>

            </mod-cluster-config>

        </subsystem>

Thx in advance

Constantin

[hostalp]

Hello, I was recently dealing with pretty much the same issue - also with ModCluster 1.3.2 (isn't there a bug in this version?) but with JBoss EAP (e.g. no Undertow).

In order to work around this I had to dig out one old system property (that I used quite a long time ago before this case) - org.apache.coyote.ajp.DEFAULT_CONNECTION_TIMEOUT (value is in ms) that has to be set to some short time period after which the old AJP connections will be closed. Too short time would lead to (ineffective) frequent openings/closures of such connections, so you may try to experiment a little to find out some decent value that isn't too short yet it prevents the constant increase of established connections. I usually use values between 1-10 minutes (60000 - 600000 ms) here.

This won't obviously work with Undertow based WildFly, however check out the no-request-timeout property (Wildfly Model Reference) - that should hopefully have the same effect.

You may also try to tune other timeouts such as read-timeout as it's been reported to help in such cases as well ([WFLY-3536] Wildfly 8.1.0 Final keeps established connections forever - JBoss Issue Tracker)