mod_cluster forwarding request to jboss without requested alias deployed
I am running a mod_cluster httpd server which farms requests out amongst 3 Jboss AS back-end servers. Each Jboss server has multiple applications deployed on it, some sharing the same context root (but with different aliases), some sharing the same alias with different context roots. The same applications are generally deployed on more than one Jboss server to provide failover.
I am seeing a problem where a request for a certain application will be sent to one of the 3 Jboss servers which does not have that application deployed, whereas the other 2 Jboss servers have the application deployed and available. The requested application Alias and Context show up on the mod_cluster_manager page on the 2 servers where it is deployed, and not on the 3rd server where it is not deployed - as expected. Why then does mod_cluster send the request to the only server without the application deployed? The response as seen by the users browser is to return a blank page...
I have reproduced this bug by restarting the httpd process on the mod_cluster server whilst the 3 Jboss servers are left up and running, waiting for all the deployed applications to be registered by mod_cluster, and then requesting the page in question of the application that is deployed on Jboss servers 1 & 2 and not deployed on server 3.
In the logs i can see mod_proxy_http declining the requested URL, followed by mod_proxy_ajp trying to serve the URL.
Error log from httpd shows mod_cluster deciding to send the request to the server without the application (10.50.50.100):
[Wed Oct 27 16:18:35 2010] [debug] mod_proxy_cluster.c(1176): get_balancer_by_node found context /appContextRoot
[Wed Oct 27 16:18:35 2010] [debug] mod_proxy_cluster.c(1176): get_balancer_by_node found context /appContextRoot
[Wed Oct 27 16:18:35 2010] [debug] mod_proxy_cluster.c(1176): get_balancer_by_node found context /appContextRoot
[Wed Oct 27 16:18:35 2010] [debug] mod_proxy_cluster.c(1176): get_balancer_by_node found context /appContextRoot
[Wed Oct 27 16:18:35 2010] [debug] mod_proxy_cluster.c(1394): proxy: byrequests balancer DONE (ajp://10.50.50.100:8009)
[Wed Oct 27 16:18:35 2010] [debug] mod_proxy.c(993): Running scheme balancer handler (attempt 0)
[Wed Oct 27 16:18:35 2010] [debug] mod_proxy_http.c(1930): proxy: HTTP: declining URL ajp://10.50.50.100:8009/appContextRoot/
[Wed Oct 27 16:18:35 2010] [debug] mod_proxy_ajp.c(644): proxy: AJP: serving URL ajp://10.50.50.100:8009/appContextRoot/
[Wed Oct 27 16:18:35 2010] [debug] proxy_util.c(1999): proxy: AJP: has acquired connection for (10.50.50.100)
[Wed Oct 27 16:18:35 2010] [debug] proxy_util.c(2055): proxy: connecting ajp://10.50.50.100:8009/appContextRoot/ to 10.50.50.100:8009
[Wed Oct 27 16:18:35 2010] [debug] proxy_util.c(2153): proxy: connected /appContextRoot/ to 10.50.50.100:8009
[Wed Oct 27 16:18:35 2010] [debug] ajp_utils.c(31): Into ajp_handle_cping_cpong
[Wed Oct 27 16:18:35 2010] [debug] ajp_utils.c(102): ajp_handle_cping_cpong: Done
[Wed Oct 27 16:18:35 2010] [debug] ajp_header.c(224): Into ajp_marshal_into_msgb
[Wed Oct 27 16:18:35 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[0] [host] = [application1.domain.com] [Wed Oct 27 16:18:35 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[1] [Accept] = [text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0
.8]
[Wed Oct 27 16:18:35 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[2] [Accept-Charset] = [ISO-8859-1,utf-8;q=0.7,*;q=0.7]
[Wed Oct 27 16:18:35 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[3] [Accept-Encoding] = [gzip,deflate]
[Wed Oct 27 16:18:35 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[4] [Accept-Language] = [en-gb,en;q=0.5]
[Wed Oct 27 16:18:35 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[5] [Cache-Control] = [max-age=0] [Wed Oct 27 16:18:35 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[6] [User-Agent] = [Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.2.
8) Gecko/20100722 Firefox/3.6.8 ( .NET CLR 3.5.30729; .NET4.0C)]
[Wed Oct 27 16:18:35 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[7] [X-FORWARDED-FOR] = [193.131.98.93]
[Wed Oct 27 16:18:35 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[8] [Connection] = [keep-alive]
[Wed Oct 27 16:18:35 2010] [debug] ajp_header.c(450): ajp_marshal_into_msgb: Done
[Wed Oct 27 16:18:35 2010] [debug] mod_proxy_ajp.c(265): proxy: APR_BUCKET_IS_EOS
[Wed Oct 27 16:18:35 2010] [debug] mod_proxy_ajp.c(270): proxy: data to read (max 8186 at 4)
[Wed Oct 27 16:18:35 2010] [debug] mod_proxy_ajp.c(285): proxy: got 0 bytes of data
[Wed Oct 27 16:18:35 2010] [debug] ajp_header.c(687): ajp_read_header: ajp_ilink_received 04
[Wed Oct 27 16:18:35 2010] [debug] ajp_header.c(697): ajp_parse_type: got 04
[Wed Oct 27 16:18:35 2010] [debug] ajp_header.c(516): ajp_unmarshal_response: status = 404
[Wed Oct 27 16:18:35 2010] [debug] ajp_header.c(537): ajp_unmarshal_response: Number of headers is = 1
[Wed Oct 27 16:18:35 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[0] [Content-Length] = [0]
[Wed Oct 27 16:18:35 2010] [debug] ajp_header.c(687): ajp_read_header: ajp_ilink_received 05
[Wed Oct 27 16:18:35 2010] [debug] ajp_header.c(697): ajp_parse_type: got 05
[Wed Oct 27 16:18:35 2010] [debug] mod_proxy_ajp.c(562): proxy: got response from 10.50.50.100:8009 (10.50.50.100)
[Wed Oct 27 16:18:35 2010] [debug] proxy_util.c(2017): proxy: AJP: has released connection for (10.50.50.100)
Any thoughts on what the cause of this error is, and if it's the mod_proxy_http and mod_proxy_ajp components conflicting, what config needs modifying to fix this?
Thanks,
