Stopping and restarting JBoss nodes cause 503 error in mod_cluster

[antlia]

Hi to all,

I'm using mod_cluster as load balancer for JBoss cluster with the following configuration:

- Domain manager

- Two slaves running on different machines

- Apache server + mod_cluster extension running on a separate machine

I'm experimenting this issue: while I simulate a system failure, by killing all jboss processes in unix console on all the machines, and then restarting the nodes, i see that mod_cluster-manager is seeing correctly the cluster (3 nodes, correct machine names, correct applications deployed and so on), but when trying to access to an application deployed i got:

503 error Service Temporary Unavailable

In mod_cluster log apache side, it's reported the following error:

All worker in error state

Can anyone help me?

Regards,

Antlia

[jfclere]

You need to wait a few seconds (~ 10) until the node are back in the OK state. (load !=-1).

Did you wait enough?

[antlia]

Hi Jean-Frederic,

I waited for ~ 60 secs, but the message still appears when refreshing the page, the only way for restore the correct state of execution is to restart apache service in Load Balancer machine.

Edit:

While seeing 503 error, mod_cluster-manager correctly shows all the nodes in the cluster, each of them having load  factor != -1

[rhusar]

What version are you using?

Is there any more relevant info if you turn on the debug log in httpd? (LogLevel debug)

[antlia]

Hi Radolslav, I'm using the following versions of mod_cluster and JBoss AS:

- mod_cluster 1.2.0 Final

- JBoss AS 7.1.1 Final

I've replied the issue, after changing Apache LogLevel to debug as you suggested.

The error.log attached to this message, shows the following test:

- Startup JBoss in domain mode (single node)

- Startup Apache (wait some time for cluster configuration completion in mod_cluster)

- Load mod_cluster-manager page: everything is fine (node listed, application correctly deployed, load != -1)

- Web application page loaded succesfully

- Stop JBoss node from JBoss admin console (server-three stopped correctly)

- Refresh mod_cluster-manager view: cluster correctly refreshed, node-list empty

- Restart JBoss node from JBoss admin console: node succesfully restarted

- Refresh mod_cluster-manager view: cluster correctly refreshed, one element present in node list

- Reload web applicatoin page: 503 Error

In error.log, before "All workers in error state error", now I can see: [debug] mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED

Another interesting thing: during the first request, the log also reports this warn: [warn] manager_handler STATUS error: MEM: Can't read node, but I've configured JBoss with instance-id="${jboss.node.name" attribute, which is (as the JBoss documentation says), the default workaround for avoiding such message...so I'm not able to explain this warn...

Can you help me?

Best Regards

[jfclere]

The [warn] manager_handler STATUS error: MEM: Can't read node are normal when you start or restart AS.

I don't find anything obvious in the error_log file.

Are you using ProxyPass in the httpd.conf?

[antlia]

Hi Jean Frederick,

i tried both with CreateBalancer 1 + ProxyPass directives, and without these instructions...but none of these soluctions do the trick...here I've attached the configuration files I'm currently using in Apache (apache2 on Ubuntu).

[jfclere]

you shouldn't mix ProxyPass and mod_cluster (at least with 1.2.0.Final). What the log with ProxyPasss or without?

[antlia]

Sorry Jean Frederick i didn't understand, should I remove all ProxyPass and ProxyPassReverse directives from my apache's config files? Can I keep ProxyPreserveHost?

Why ProxyPass directives can cause problems? Is there a bug in mod_cluster that prevents its normal behavior?

However, I made the test again, removing CreateBanlancers, ProxyPass, ProxyPassReverse and ProxyPreserveHost directives, the test is the same I made yesterday, described by the previous post, and this is apache's error.log

Thanks for the support.

[jfclere]

should I remove all ProxyPass and ProxyPassReverse directives from my apache's config files?

Yes

Can I keep ProxyPreserveHost?

Yes

There were bugs in the old version of mod_cluster that affected the mod_proxy ProxyPass directives.

I can't find any error in the error.log. Could you compile a never version .1.2.x of mod_cluster and retry?

[antlia]

I forgot to mention the fact that I'm using x64 version of mod_cluster libraries, downloaded directly from mod_cluster official site.

I'll try to download the binaries and compile them, directly from my machine.

Thanks for your support.

[jfclere]

I suggest you compile the source directly from the git repos (use the 1.2.x) branch.

[antlia]

Hi Jean-Frederic,

i tried to compile the source directly from the git repos with no luck...I also tried to install apache on a x86-based system (maybe x64 libraries are affected by some bug), nothing to do, the problem remains the same.

Maybe it can be an issue on my JBoss configuration?

I've disabled auto-announce, listing the proxies manually in domain.xml and also ajp protocol for mod_cluster messages (anyway I'm having the error even using http protocol)...is there any other option to enable/tune for mod_cluster's to work correctly?

[jfclere]

"i tried to compile the source directly from the git repos with no luck.."

Does it mean that 1.2.x from the git repo has the same problem?

[antlia]

Yes, sorry for the poor explanation...yes, I mean that I have compiled succesfully the sources from the git repository, obtaining the same issue when starting/stopping JBoss nodes.