1 2 Previous Next 15 Replies Latest reply on Jul 24, 2017 4:30 PM by gsandeepry

Error 503 using mod_cluster

nichele Sep 5, 2011 12:50 PM

Hi All,

i'm using mod_cluster in production but i'm having some 503.

I tried replacing mod_cluster with mod_proxy_balancer and in such case i didn't have any 503 so i'm thinking to a misconfiguration of mod_cluster.

My environemnt is composed by an instance of apache httpd ver. 2.2.19, and of 4 instances of tomcat 6.0.13 with mod_cluster 1.1.2.

In any server.xml file i added:

<Listener className="org.jboss.modcluster.catalina.ModClusterListener"

advertise="false"

proxyList="my-proxy:6666"

maxAttempts="3"

nodeTimeout="600"

ping="30"

loadMetricClass="org.jboss.modcluster.load.metric.impl.BusyConnectorsLoadMetric" />

In my httpd conf file i have:

Listen *:6666

Order deny,allow

#Deny from all

Allow from all

</Directory>

KeepAliveTimeout 60

MaxKeepAliveRequests 0

ManagerBalancerName mycluster

ServerAdvertise Off

</VirtualHost>

SetHandler mod_cluster-manager

Order deny,allow

Deny from all

Allow from xx.xx.xx.xx

</Location>

ProxyStatus On

In the apache error log i have:

Sep 5 06:54:10 httpd[26073]: [debug] mod_proxy_cluster.c(622): update_workers_node starting

Sep 5 06:54:10 httpd[26073]: [debug] mod_proxy_cluster.c(637): update_workers_node done

Sep 5 06:54:20 httpd[26073]: [error] (70007)The timeout specified has expired: proxy: AJP: attempt to connect to 10.214.14.209:8001 (10.214.14.209) failed

Sep 5 06:54:20 httpd[26073]: [error] ap_proxy_connect_backend disabling worker for (10.214.14.209)

Sep 5 06:54:20 httpd[26073]: [error] proxy: AJP: failed to make connection to backend: 10.214.14.209

Sep 5 06:54:20 httpd[26073]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE02)

Sep 5 06:54:21 httpd[26073]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE02)

Sep 5 06:54:21 httpd[25976]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE02)

Sep 5 06:54:21 httpd[26073]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE02)

Sep 5 06:54:21 httpd[25976]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE02)

Sep 5 06:54:22 httpd[25976]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE02)

For completeness, this is the mod_proxy_balancer configuration i tested:

BalancerMember ajp://node01:8001/ route=NODE01

BalancerMember ajp://node02:8001/ route=NODE02

BalancerMember ajp://node03:8001/ route=NODE03

BalancerMember ajp://node04:8001/ route=NODE04

ProxySet lbmethod=bybusyness

ProxySet stickysession=JSESSIONID|jsessionid

</Proxy>

ProxyPass / balancer://mycluster/

ProxyPassReverse / balancer://mycluster/

ProxyStatus On

Do you have any suggestion ? I tried to play with the timeouts but i was not able to find any working values.

As usual thanks in advance

Stef

1. Re: Error 503 using mod_cluster

jfclere Sep 6, 2011 2:45 AM (in response to nichele)

It looks like to me that the node is not sending the right address to httpd, to say it another way: 10.214.14.20 is not the address of node1 (or any of the nodes in the BalancerMember list). You have to provide the address either while starting the node whith -b node1 for example or adding address=node1 in the ajp connector.
Actions
2. Re: Error 503 using mod_cluster

nichele Sep 6, 2011 3:49 AM (in response to jfclere)

Hi Jean-Frederic,
i already have the address in the ajp connector:

<Connector port="8001" protocol="AJP/1.3" redirectPort="8441" address="10.214.14.209" maxThreads="200" />

and usually the system works fine.
This is how the log continues if it can be useful (as you can see after some seconds apache is able to created worker to 10.214.14.209

Sep 5 06:54:29 httpd[26261]: [debug] proxy_util.c(1806): proxy: grabbed scoreboard slot 0 in child 26261 for worker proxy:reverse
Sep 5 06:54:29 httpd[26261]: [debug] proxy_util.c(1825): proxy: worker proxy:reverse already initialized
Sep 5 06:54:29 httpd[26261]: [debug] proxy_util.c(1902): proxy: initialized worker 0 in child 26261 for (*) min=0 max=30 smax=30
Sep 5 06:54:30 httpd[26261]: [debug] mod_proxy_cluster.c(416): add_balancer_node: Create balancer balancer://mycluster
Sep 5 06:54:30 httpd[26261]: [debug] mod_proxy_cluster.c(228): Created: worker for ajp://10.214.14.209:8001
Sep 5 06:54:30 httpd[26261]: [debug] mod_proxy_cluster.c(334): proxy: initialized worker 1 in child 26261 for (10.214.14.209) min=0 max=30 smax=30
Sep 5 06:54:30 httpd[26261]: [debug] mod_proxy_cluster.c(392): Created: worker for ajp://10.214.14.209:8001 1 (status): 1
Sep 5 06:54:30 httpd[26261]: [debug] mod_proxy_cluster.c(228): Created: worker for ajp://10.194.187.15:8001
.....

But sometime there is a 503 (not always on the same host) with a timeout in the apache server log.

Ste
Actions
3. Re: Error 503 using mod_cluster

rhusar Sep 6, 2011 7:39 AM (in response to nichele)

I remember in some case CreateBalancers 0/1/2 helped, I don't remember why and how unfortunately. Here are the docs for that:

http://docs.jboss.org/mod_cluster/1.1.0/html/native.config.html#d0e490
Actions
4. Re: Error 503 using mod_cluster

jfclere Sep 6, 2011 10:14 AM (in response to rhusar)

"But sometime there is a 503 (not always on the same host) with a timeout in the apache server log."

That could be a problem of gc in the JVM too.

Try to increase the ping and the nodeTimeout values or check the application behaviour.
Actions
5. Re: Error 503 using mod_cluster

rhusar Sep 6, 2011 10:17 AM (in response to jfclere)

+1 In situations like that, for undestanding what is wrong make sure to check the servers behind the load balancer by accessing the standard HTTP too.
Actions
6. Re: Error 503 using mod_cluster

nichele Sep 6, 2011 4:05 PM (in response to rhusar)

Hi,
i'm trying increasing timeout but just now i have found another error in my apache log that causes at the end a 503:

Sep 6 17:35:07 httpd[29260]: [error] ajp_marshal_into_msgb - No such method
Sep 6 17:35:07 httpd[29260]: [error] ajp_send_header: ajp_marshal_into_msgb failed
Sep 6 17:35:07 httpd[29260]: [error] (120010)APR does not understand this error code: proxy: AJP: request failed to 10.194.187.15:8001 (10.194.187.15)
Sep 6 17:35:07 httpd[25446]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE03)
Sep 6 17:35:08 httpd[25446]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE03)
Sep 6 17:35:08 httpd[29260]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE03)
Sep 6 17:35:08 httpd[29260]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE03)
Sep 6 17:35:09 httpd[29260]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE03)

Thoughts ?

Thanks
ste
Actions
7. Re: Error 503 using mod_cluster

jfclere Sep 7, 2011 5:39 AM (in response to nichele)

weird... do you have a little more trace? - before the "ajp_marshal_into_msgb - No such method"
It looks like you have a weird request being processed... Unfornatunatly it marks the node in error instead just reject the request.
Actions
8. Re: Error 503 using mod_cluster

rhusar Sep 7, 2011 6:30 AM (in response to jfclere)

"No such method" sounds like messed up versions :-)
Actions
9. Re: Error 503 using mod_cluster

nichele Sep 7, 2011 11:17 AM (in response to jfclere)

I was able to reproduce this last issue:

1. open a shell
2. type "telnet host post"
3. type <SPACE><SPACE><SPACE><ENTER>

that's all. The request is malformed and in the apache log there is:

Sep 7 15:07:35 httpd[15164]: [error] ajp_marshal_into_msgb - No such method
Sep 7 15:07:35 httpd[15164]: [error] ajp_send_header: ajp_marshal_into_msgb failed
Sep 7 15:07:35 httpd[15164]: [error] (120010)APR does not understand this error code: proxy: AJP: request failed to 10.254.234.112:8001 (my-node04)
Sep 7 15:07:38 httpd[15164]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE04)
Sep 7 15:07:39 httpd[15164]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE04)
Sep 7 15:07:45 httpd[18651]: [debug] proxy_util.c(1806): proxy: grabbed scoreboard slot 0 in child 18651 for worker proxy:reverse

and some 503 are returned (i think 2 in this case since we had 2 workers).

Im supposing this is bug, doesn't it ?
The same scenario doesn't cause issues with mod_proxy (just the request is rejected)

(still testing 503 caused by timeout)

Thanks
ste
Actions
10. Re: Error 503 using mod_cluster

jfclere Sep 12, 2011 9:45 AM (in response to nichele)

"I'm supposing this is bug, doesn't it ?"

Yes. In fact the bug is in mod_proxy_ajp the request should be rejected with not such method (bad request). I have patch httpd-trunk and proposed the patch for back port.
Actions
11. Re: Error 503 using mod_cluster

nichele Sep 10, 2011 6:06 AM (in response to jfclere)

Coud you provide me the patch for 2.2.15 (the version bundled with mod_cluster) ? or for any other version if available....

But i have not undestood why with mod_proxy_balancer (and mod_proxy_apj since i'm using ajp as protocol) the issue doesn't occur.

thanks
ste
Actions
12. Re: Error 503 using mod_cluster

jfclere Sep 13, 2011 2:14 AM (in response to nichele)

You should upgrade to 2..2.21 see http://www.apache.org/dist/httpd/CHANGES_2.2
Actions
13. Re: Error 503 using mod_cluster

nichele Sep 13, 2011 3:37 AM (in response to jfclere)

Thanks Jean-Frederic,
one things i do not understand yet...

Why mod_cluster releases modules for apache ? I mean, i can use apache 2.2.21 but then i have to overwrite for instance mod_proxy.so with the one coming from mod_cluster...is this correct ? am I not loosing some fixes done in the apache code in this way ?

I'm asking this since using official httpd modules without to overwrite them with the mod_cluster ones (of course i copied the ones not available in apache) i have some issues (in particular it seems apache doesn't do stickysession).

thanks
ste
Actions
14. Re: Error 503 using mod_cluster

jfclere Sep 13, 2011 11:02 AM (in response to nichele)

mod_cluster delivers bundles to make it more easy to test mod_cluster.
we deliver so files too in that case you will have to copy only:
mod_slotmem.so
mod_proxy_cluster.so
mod_manager.so
mod_advertise.so

Or wait for the next release of mod_cluster
Actions

1 2 Previous Next

Go to original post