Mod Cluster load balancer sends a 2nd, empty, request after ping timeout

[dshifrin]

Thank you in advance for reading this.

I have a Wildfly 9.0.2 domain with 2 mod cluster load balancers in front of my 4 application servers. We have sticky sessions set to true. I have noticed the following behavior:

User logs into application and is "stuck" to server X

All future requests go to server X.

Then on a page that has a lot of details to save user clicks the submit

This specific click also lands on server X.  Server X makes a request to a webservice to do the work. This work takes almost 11 seconds to accomplish, I have my Ping set to 10. I notice that a second request is sent to my application cluster (landing on a random server, lets say server Y, where Y is not X) that has an empty body at a little bit after 10 seconds. Since this request is empty, it fails very quickly and an error is returned to my application from server Y. Both of these requests have the same jsession id and both requests have cookies and request headers. As I changed my Ping timeout setting from 10 to something else I would notice the second request appear just a little bit after that time (ie. when I set it to 1 I saw the 2nd request land about 1.02 seconds later).

let me know if there is more information that could be useful to solve this.

Any help would be appreciated and again, thank you for taking the time to read this.

[dshifrin]

I found something that is similar to what I am experiencing.  Ping timeout acting as worker timeout

[rhusar]

Hey. Is this with httpd-based or undertow-based proxy? If the former, have you tried     LogLevel debug ?

[dshifrin]

Hi Radoslav,  We are using httpd.  I have the log level set to debug.

After 10 seconds (or whatever the ping value is set to) from when a request comes in I see this in my error.log

[proxy:error] [pid 2244] AH00940: https: disabled connection for (10.105.152.35)

This coincides with when a 2nd request is sent. I found that the 2nd request is empty because I have stickySessionForce set to false. But my question is how to stop the 2nd request from being launched. At the moment I have set Ping to the longest amount of time I expect my requests to take.

I don't see anything obvious in ssl_error_log (we are strictly https) but perhaps I don't know what to look for.

[rhusar]

I am trying to refresh my memory here... but do you have  EnableOptions in your config?  This thing is that the ping should never really fail. It's a simple request just to verify the connection to the worker is still up.

[dshifrin]

I do not know where to check for EnableOptions, but I do not see it set in any .conf file in my httpd directory.

[rhusar]

IIRC if you are not using AJP you need to set EnableOptions (needs no parameters) in your vhost. Fulfils the same role as the CPING/CPONG used by AJP but for HTTP/HTTPS connections. The endpoint needs to implement at least HTTP/1.1.

[dshifrin]

We are using AJP for multiCast.  As far as I know we do not use it for anything else. 

[rhusar]

David, you seem to be confusing the terms. AJP is binary protocol for communication proxy<->backend server. Multicast is used for discovery of proxy servers by the backend servers.

So in your httpd configuration just add EnableOptions to the virtual host.

[dshifrin]

You are correct, I am totally confusing things.  I did as you suggested and added EnableOptions to my config.  It now looks like this. Unfortunately,  the ping timeout setting is till controlling when a failover happens for a request.

# UseAlias break mod cluster in rhel 7
# UseAlias On
MemManagerFile /var/cache/httpd
Maxnode 500
Maxhost 500
### MOD CLUSTER NODE UPDATE SITE
Listen XX.XXX.XXX.XX:PORT
<VirtualHost XX.XXX.XXX.XX:PORT>
  <Directory />
  Require ip 10.
  </Directory>
  ServerAdvertise on https://XX.XXX.XXX.XX:PORT
  AdvertiseBindAddress XXX.XXX.XXX.XX:PORT
  AdvertiseFrequency 2
  AdvertiseGroup XXX.XXX.XXX.XXX:PORT
  EnableMCPMReceive
  ManagerBalancerName MOD_CLUSTER_LB
  SSLEngine on
  SSLProtocol -all +TLSv1.2
  SSLCertificateFile /<path to .crt file>
  SSLCertificateKeyFile  <path to server key>
  SSLCACertificateFile <path to ca cert>
  EnableOptions
</VirtualHost>

### MOD CLUSTER MGMT SITE
Listen XX.XXX.XXX.XX:PORT
<VirtualHost XX.XXX.XXX.XX:PORT>
  <Location /mod_cluster-manager>
   SetHandler mod_cluster-manager
   Require ip 10.
  </Location>
  SSLCertificateFile <path to server crt file>
  SSLCertificateKeyFile  <path to server key file>
  SSLCACertificateFile <path to ca cert>
  SSLProxyCACertificateFile <path to ca cert>
  SSLProxyEngine on
  ProxyPreserveHost on
  SSLProtocol -all +TLSv1.2
  SSLProxyVerify none
  SSLProxyCheckPeerCN off
  SSLProxyCheckPeerName off
  SSLProxyCheckPeerExpire off
</VirtualHost>
Listen XX.XXX.XXX.XX:PORT
<VirtualHost XX.XXX.XXX.XX:PORT>
  SSLCertificateFile <path to server crt file>
  SSLCertificateKeyFile  <path to server key file>
  SSLCACertificateFile <path to ca cert>
  ErrorLog logs/ssl_error_log
  TransferLog logs/ssl_access_log
  LogLevel debug

</VirtualHost>
### Adds reverse proxy tls support to default reverse proxy
  SSLProxyEngine on
  ProxyPreserveHost on
  SSLProxyCACertificateFile <path to ca cert>
  SSLProtocol -all +TLSv1.2
  SSLProxyVerify none
  SSLProxyCheckPeerCN off
  SSLProxyCheckPeerName off
  SSLProxyCheckPeerExpire off