JBoss cluster node has -1 load

[dmoses]

This cluster has been running without issue for several months but all of a sudden after restarting the jboss servers one of the nodes has a -1 load in the mod_cluster manager and is not sent any requests.  When this happens you can manually hit this server and it responds and serves up the application but if you try and access the cluster via apache/mod_cluster it will never send you to this server.  We have tried stopping both nodes in the cluster and noticed that the problem will bounce over to the other node and the server that previously had the -1 load is now running ok.  Mod_cluster manager sees both nodes with the application deployed it just is not distributing the load properly.

Has anyone ever experienced an issue like this?

we are running jboss 6.0 final with mod_cluster 1.1.0.

[jfclere]

1.1.0. is a old version. Could you upgrade to 1.1.2 and retry?

[rhusar]

BTW doesn't load equal to -1 mean the node is in error state? Really, -1 cant be the value advertized by the server (unless there would be an exception).

[jfclere]

-1 means the cping/cpong failed.

[dmoses]

I see cping/cpong error messages only when I restart the server but then they go away.

[Wed May 25 07:18:35 2011] [error] ajp_cping_cpong: apr_socket_recv failed

[Wed May 25 07:18:40 2011] [error] ajp_cping_cpong: apr_socket_recv failed

[Wed May 25 07:18:45 2011] [error] ajp_cping_cpong: apr_socket_recv failed

[Wed May 25 07:18:50 2011] [error] ajp_cping_cpong: apr_socket_recv failed

[Wed May 25 07:18:55 2011] [error] ajp_cping_cpong: apr_socket_recv failed

[Wed May 25 07:19:00 2011] [error] ajp_cping_cpong: apr_socket_recv failed

[pferraro]

Those cping/cpong failures suggest that the JBoss Web connectors are getting shutdown prior to the AS mod_cluster service issuing the appopriate STOP-APP/REMOVE-APP messages.  At the moment, I can't think of any circumstances that would cause this to happen (mod_cluster shutdown is triggered by a JMX notification prior to stopping the JBoss Web connectors) - unless you're also seeing errors in your server log that indicate these messages were never sent.

[dmoses]

I figured out what the issue was.  In the apache config someone made a change and set CreateBalancers=0.  Once this was changed to 1 and apache restarted the second node started receiving requests again.

[rhusar]

Thanks for sharing Derric.

(here is a quick link to docs for anyone else popping to this thread http://docs.jboss.org/mod_cluster/1.1.0/html/native.config.html#d0e490 )

[okelet]

Hi

I have the same issue. With CreateBalancers 1, and defined "ProxyPass / balancer://mycluster/ stickysession=JSESSIONID|jsessionid" inside a VirtualHost, when I restart Apache, all nodes get load -1 and applications are inaccesible for new sessions (no matter how much time I wait, load -1 al always shown, previous sessions works fine). I have to restart JBoss and then the load shows fine and applications work and new sessions works again.

Any solution or workaround?

Regards.