-
1. Re: mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED
mbabacek May 29, 2013 4:59 PM (in response to thinksteep)Hi user, this is your community speaking:
Could you post your configs here please? mod_cluster.conf and the modcluster subsystem element from NODE and NODE2?
Furthermore, paste the whole error_log, or at least more of it please. I wonder whether all the workers are not in the error state, judging by the internal_find_best_byrequests function.
-
2. Re: mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED
thinksteep May 29, 2013 5:14 PM (in response to mbabacek)Hi Michal,
Thanks for quick reply!
Here is web sub-system configuration:
<server name="newnode2" xmlns="urn:jboss:domain:1.2">
.......................................
<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">
<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
<connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp"/>
<virtual-server name="default-host" enable-welcome-root="true">
<alias name="localhost"/>
<alias name="example.com"/>
</virtual-server>
</subsystem>
And mod cluster sub system:
<subsystem xmlns="urn:jboss:domain:modcluster:1.0">
<mod-cluster-config advertise-socket="modcluster" proxy-list="apacheIP:8080" balancer="mycluster" advertise-security-key="secret">
<dynamic-load-provider>
<load-metric type="busyness"/>
</dynamic-load-provider>
</mod-cluster-config>
</subsystem>
Apache httpd.conf entries:
LoadModule slotmem_module modules/mod_slotmem.so
LoadModule manager_module modules/mod_manager.so
LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
LoadModule advertise_module modules/mod_advertise.so
Listen apacheMachineIP:8080
<VirtualHost apacheMachineIP:8080>
<Location />
Order deny,allow
Allow from all
</Location>
<Location /mod_cluster-manager>
SetHandler mod_cluster-manager
Order deny,allow
Allow from all
</Location>
KeepAliveTimeout 60
#KeepAlive Off
MaxKeepAliveRequests 0
ManagerBalancerName mycluster
ServerAdvertise On
AdvertiseFrequency 5
EnableMCPMReceive
</VirtualHost>
Attached error_log from apache (it is trimmed down version which has entries for initial start---> shutdown --->Start again (Which is when I am seeing above entries and all requests are resulting in 503 error).
Even though I have two nodes, I am just playing with Node2 only because this is the one causing issue.
Please let me know if any information required.
Thank you,
-
error_log.zip 2.6 KB
-
-
3. Re: mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED
mbabacek May 29, 2013 5:41 PM (in response to thinksteep)Hey Steep, it looks like one of our nodes uses an IPv6 and the other one IPv4 interface, is this intentional?
First, make everything IPv4 only and test it.
Furthermore, try to bind both balancer and workers to IPv6 interfaces and test again.
If you need to work with IPv6, it would be best for you to upgrade to mod_cluster 1.2.3+ as there were some IPv6 related bugs in 1.2.0 if my memory doesn't fail me.
Are you up to any exotic network setup, like an IPv6/IPv4 hybrid cluster? Why?
-
4. Re: mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED
thinksteep May 29, 2013 5:51 PM (in response to mbabacek)Hi Michal,
Thanks for your time on this.
1) AFAIK, We are using IPv4 only. How were you able to figure out IPv4 & IPv6 combination? Could you provide traces? That helps as backup while discussing with network admin.
2) Log attached in previous reply is from only one node (Which is NODE2). Other node is down and I am not using it at this moment because issue it is working fine on restarts.
Thank you for your time and help.
-
5. Re: mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED
thinksteep May 29, 2013 6:18 PM (in response to thinksteep)One more observation is, on restart of APACHE (Don't need to restart NODE2 after above steps), further requests are being processed succesfully (No more 503 errors).
Thank you,
-
6. Re: mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED
mbabacek May 30, 2013 11:53 AM (in response to thinksteep)Well, it look like your worker "secondnodeIP" - if it is not just something you searched/replaced in the log, is coming from an IPv6:
[Wed May 29 16:33:57 2013] [debug] mod_proxy_ajp.c(616): proxy: got response from [foo:yada:yada:yada:yada:yada:yada:dbbc]:8009 (secondnodeIP)
I can't tell from
Listen apacheMachineIP:8080
what address are you listening on...yet, this looks like your client (browser) address: 192.168.1.79,
suggesting that the balancer is listening on IPv4 as well, since it tried to serve you a non existing favicon.
Your workers might pick up balancer's advertising (ou don't have advertise="false" configures) or they use what you configured in proxy-list.
What I see as a problem in this setup is that balancer can't talk back to IPv6 addresses and contact workers, hence:
[Wed May 29 16:35:42 2013] [debug] proxy_util.c(2471): (70007)The timeout specified has expired: proxy: ajp: attempt to connect to [foo:yada:yada:yada:yada:yada:yada:dbbc]:8009 (secondnodeIP) failed
Suggestion: Try everything on IPv4 first and if it works, move to IPv6.
EDIT: I've seen your latest remark regarding Apache's restart: What address it bound to after it had started? What the debug log looks like now? Is the balancer able to talk back to the workers now?
-
7. Re: mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED
thinksteep May 30, 2013 11:35 AM (in response to mbabacek)Hi Michal,
Thanks for your time on this.
I have a request, can you please mask the IP in your answer? I don't want to make it public.
Yes, you are correct, I did search & replace.
Today we disabled IPv6 on both NODE2 & Apache server. Now they have only IPv4 enabled on both machines. Still same issue.
Attached full log. Sequence of operations are:
1) Started Node2
2) Browser request to application (Browser/Client is also on Node2 machine)--->Sucess
3) Stopped Node2
4) Started Node2
5) Browser request to application ---> 503 error
6) Restart the Node2 again
7) Browser request to application ----> 503 error
8) Restart Apache server
9) Browser request to application -----> Sucess
This would be the sequence you see in attached log. Please suggest your thoughts.
Thanks again for your time and help.
-
error_log.zip 19.3 KB
-
-
8. Re: mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED
mbabacek May 31, 2013 1:00 PM (in response to thinksteep)I had tried to reproduce it with the same httpd version and I failed. I'll try it again... stay tuned.
-
9. Re: mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED
thinksteep Jun 6, 2013 10:02 AM (in response to mbabacek)Hi Michal,
Thanks for your time and help.
Any luck on this?
Thank you,