mod_cluster 1.2.0 + JBoss AS 7.1.1 - second node always has load of "-1"

[benashmead]

Hi,

I'm trying to get mod_cluster and JBoss AS 7.1.1 working nicely in an EC2 environment (i.e. no multicast). I think it's almost there - I've got my two application nodes up and running, they're talking to Apache/mod_cluster (sort of), but the load balancing seems to be a bit wonky.

Here's my bare-bones configuration:

httpd:

     ManagerBalancerName mycluster

     AllowDisplay on

     <VirtualHost *:80>

          KeepAliveTimeout 300

          MaxKeepAliveRequests 0

          AdvertiseFrequency 5

          AdvertiseSecurityKey cluster

          EnableMCPMReceive

          ServerAdvertise Off

     <Location /mod_cluster_manager>

          SetHandler mod_cluster-manager

     </Location>

     </VirtualHost>

cluster nodes:

<subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="tcp">

            <stack name="udp">

                <transport type="UDP" socket-binding="jgroups-udp" diagnostics-socket-binding="jgroups-diagnostics"/>

                <protocol type="PING"/>

                <protocol type="MERGE2"/>

                <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>

                <protocol type="FD"/>

                <protocol type="VERIFY_SUSPECT"/>

                <protocol type="BARRIER"/>

                <protocol type="pbcast.NAKACK"/>

                <protocol type="UNICAST2"/>

                <protocol type="pbcast.STABLE"/>

                <protocol type="pbcast.GMS"/>

                <protocol type="UFC"/>

                <protocol type="MFC"/>

                <protocol type="FRAG2"/>

            </stack>

            <stack name="tcp">

                <transport type="TCP" socket-binding="jgroups-tcp" diagnostics-socket-binding="jgroups-diagnostics"/>

                <protocol type="TCPPING">

                    <property name="initial_hosts">

                        10.0.0.10[7600],10.0.0.20[7600]

                    </property>

                    <property name="num_initial_members">

                        2

                    </property>

                    <property name="port_range">

                        0

                    </property>

                    <property name="timeout">

                        2000

                    </property>

                </protocol>

                <protocol type="MERGE2"/>

                <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>

                <protocol type="FD"/>

                <protocol type="VERIFY_SUSPECT"/>

                <protocol type="BARRIER"/>

                <protocol type="pbcast.NAKACK"/>

                <protocol type="UNICAST2"/>

                <protocol type="pbcast.STABLE"/>

                <protocol type="pbcast.GMS"/>

                <protocol type="UFC"/>

                <protocol type="MFC"/>

                <protocol type="FRAG2"/>

            </stack>

        </subsystem>

        <subsystem xmlns="urn:jboss:domain:modcluster:1.0">

            <mod-cluster-config advertise-socket="modcluster" proxy-list="10.0.0.11:80" advertise-security-key="cluster">

                <dynamic-load-provider>

                    <load-metric type="cpu" capacity="1"/>

                </dynamic-load-provider>

            </mod-cluster-config>

        </subsystem>

        <subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">

            <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>

            <connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp"/>

            <virtual-server name="default-host" enable-welcome-root="true">

                <alias name="localhost"/>

                <alias name="example.com"/>

            </virtual-server>

        </subsystem>

Now, when I start both nodes, the first node that speaks to mod_cluster first (can be either one, depending on the order in which I start them) will start at a load of "0", gradually climb towards "100" then just sit there. The second node to join the cluster will stay stuck at "-1" forever. I initially tried the "busyness" load balancing factor with much the same results - the first node starts at a load of "99" and stays there, the second sticks at "-1".

Killing off the first node results in a 503 error from Apache, and "[error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state" in the error.log.

Screenshot attached, can anyone tell me what I'm doing wrong?

Thanks.

[mbabacek]

Hi Ben, please, set LogLevel debug and paste your error_log here...    -1 is an error code, it means the node is not fit.

[benashmead]

Ah, ok - I'm seeing the following in the logs:

[Mon Jun 24 12:28:07 2013] [debug] mod_manager.c(2323): manager_handler STATUS (/) processing: "JVMRoute=cluster2&Load=28"

[Mon Jun 24 12:28:07 2013] [debug] mod_manager.c(1404): Processing STATUS

[Mon Jun 24 12:28:07 2013] [debug] mod_proxy_cluster.c(1608): proxy_cluster_isup: Can't find worker for 2

So it looks as though the node is reporting the right load metric, but mod_cluster doesn't know what to do with it, maybe? Not sure where to go from here..

[mbabacek]

...and paste your error_log here...

It is unnecessarily difficult to try to help you without seeing the whole context.

If you are concerned for privacy of your systems, search&replace contexts, hostnames and addresses with foo-bar-whatever string.

Furthermore, as I stated in my previous post, -1 is not an actual Load, it is an error code.

[benashmead]

No problem, here you go:

[Mon Jun 24 12:25:48 2013] [notice] caught SIGTERM, shutting down

[Mon Jun 24 12:26:15 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)

[Mon Jun 24 12:26:15 2013] [notice] Digest: generating secret for digest authentication ...

[Mon Jun 24 12:26:15 2013] [notice] Digest: done

[Mon Jun 24 12:26:15 2013] [notice] Advertise initialized for process 1853

[Mon Jun 24 12:26:15 2013] [notice] Apache/2.2.3 (CentOS) configured -- resuming normal operations

[Mon Jun 24 12:26:16 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:26:16 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:26:16 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:26:16 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:26:16 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:26:16 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:26:16 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:26:16 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:26:16 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:26:16 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:26:16 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:26:16 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:26:16 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:26:16 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:26:16 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:26:16 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:26:48 2013] [debug] mod_manager.c(1667): manager_trans INFO (/)

[Mon Jun 24 12:26:48 2013] [debug] mod_manager.c(2323): manager_handler INFO (/) processing: ""

[Mon Jun 24 12:26:48 2013] [debug] mod_manager.c(2366): manager_handler INFO  OK

[Mon Jun 24 12:26:48 2013] [debug] mod_manager.c(1667): manager_trans CONFIG (/)

[Mon Jun 24 12:26:48 2013] [debug] mod_manager.c(2323): manager_handler CONFIG (/) processing: "JVMRoute=cluster1&Host=cluster1&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp&ping=10"

[Mon Jun 24 12:26:48 2013] [debug] mod_manager.c(2366): manager_handler CONFIG  OK

[Mon Jun 24 12:26:48 2013] [debug] mod_manager.c(1667): manager_trans ENABLE-APP (/)

[Mon Jun 24 12:26:48 2013] [debug] mod_manager.c(2323): manager_handler ENABLE-APP (/) processing: "JVMRoute=cluster1&Alias=default-host%2Clocalhost%2Cexample.com&Context=%2FClusterWebApp"

[Mon Jun 24 12:26:48 2013] [debug] mod_manager.c(2366): manager_handler ENABLE-APP  OK

[Mon Jun 24 12:26:48 2013] [debug] mod_manager.c(1667): manager_trans STATUS (/)

[Mon Jun 24 12:26:48 2013] [debug] mod_manager.c(2323): manager_handler STATUS (/) processing: "JVMRoute=cluster1&Load=19"

[Mon Jun 24 12:26:48 2013] [debug] mod_manager.c(1404): Processing STATUS

[Mon Jun 24 12:26:48 2013] [debug] proxy_util.c(2044): proxy: ajp: has acquired connection for (cluster1)

[Mon Jun 24 12:26:48 2013] [debug] proxy_util.c(2102): proxy: connecting ajp://cluster1:8009/ to cluster1:8009

[Mon Jun 24 12:26:48 2013] [debug] proxy_util.c(2195): proxy: connected ajp://cluster1:8009/ to cluster1:8009

[Mon Jun 24 12:26:48 2013] [debug] proxy_util.c(2347): proxy: ajp: fam 2 socket created to connect to cluster1

[Mon Jun 24 12:26:48 2013] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done

[Mon Jun 24 12:26:48 2013] [debug] proxy_util.c(2062): proxy: ajp: has released connection for (cluster1)

[Mon Jun 24 12:26:48 2013] [debug] mod_manager.c(2366): manager_handler STATUS  OK

[Mon Jun 24 12:26:49 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:26:49 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:26:49 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:26:49 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:26:49 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:26:49 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:26:49 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:26:49 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:26:49 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:26:49 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:26:49 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:26:49 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:26:49 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:26:49 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:26:49 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:26:49 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:26:58 2013] [debug] mod_manager.c(1667): manager_trans STATUS (/)

[Mon Jun 24 12:26:58 2013] [debug] mod_manager.c(2323): manager_handler STATUS (/) processing: "JVMRoute=cluster1&Load=28"

[Mon Jun 24 12:26:58 2013] [debug] mod_manager.c(1404): Processing STATUS

[Mon Jun 24 12:26:58 2013] [debug] proxy_util.c(2044): proxy: ajp: has acquired connection for (cluster1)

[Mon Jun 24 12:26:58 2013] [debug] proxy_util.c(2102): proxy: connecting ajp://cluster1:8009/ to cluster1:8009

[Mon Jun 24 12:26:58 2013] [debug] proxy_util.c(2195): proxy: connected ajp://cluster1:8009/ to cluster1:8009

[Mon Jun 24 12:26:58 2013] [debug] proxy_util.c(2347): proxy: ajp: fam 2 socket created to connect to cluster1

[Mon Jun 24 12:26:58 2013] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done

[Mon Jun 24 12:26:58 2013] [debug] proxy_util.c(2062): proxy: ajp: has released connection for (cluster1)

[Mon Jun 24 12:26:58 2013] [debug] mod_manager.c(2366): manager_handler STATUS  OK

[Mon Jun 24 12:27:08 2013] [debug] mod_manager.c(1667): manager_trans STATUS (/)

[Mon Jun 24 12:27:08 2013] [debug] mod_manager.c(2323): manager_handler STATUS (/) processing: "JVMRoute=cluster1&Load=36"

[Mon Jun 24 12:27:08 2013] [debug] mod_manager.c(1404): Processing STATUS

[Mon Jun 24 12:27:08 2013] [debug] proxy_util.c(2044): proxy: ajp: has acquired connection for (cluster1)

[Mon Jun 24 12:27:08 2013] [debug] proxy_util.c(2102): proxy: connecting ajp://cluster1:8009/ to cluster1:8009

[Mon Jun 24 12:27:08 2013] [debug] proxy_util.c(2195): proxy: connected ajp://cluster1:8009/ to cluster1:8009

[Mon Jun 24 12:27:08 2013] [debug] proxy_util.c(2347): proxy: ajp: fam 2 socket created to connect to cluster1

[Mon Jun 24 12:27:08 2013] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done

[Mon Jun 24 12:27:08 2013] [debug] proxy_util.c(2062): proxy: ajp: has released connection for (cluster1)

[Mon Jun 24 12:27:08 2013] [debug] mod_manager.c(2366): manager_handler STATUS  OK

[Mon Jun 24 12:27:18 2013] [debug] mod_manager.c(1667): manager_trans STATUS (/)

[Mon Jun 24 12:27:18 2013] [debug] mod_manager.c(2323): manager_handler STATUS (/) processing: "JVMRoute=cluster1&Load=44"

[Mon Jun 24 12:27:18 2013] [debug] mod_manager.c(1404): Processing STATUS

[Mon Jun 24 12:27:18 2013] [debug] proxy_util.c(2044): proxy: ajp: has acquired connection for (cluster1)

[Mon Jun 24 12:27:18 2013] [debug] proxy_util.c(2102): proxy: connecting ajp://cluster1:8009/ to cluster1:8009

[Mon Jun 24 12:27:18 2013] [debug] proxy_util.c(2195): proxy: connected ajp://cluster1:8009/ to cluster1:8009

[Mon Jun 24 12:27:18 2013] [debug] proxy_util.c(2347): proxy: ajp: fam 2 socket created to connect to cluster1

[Mon Jun 24 12:27:18 2013] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done

[Mon Jun 24 12:27:18 2013] [debug] proxy_util.c(2062): proxy: ajp: has released connection for (cluster1)

[Mon Jun 24 12:27:18 2013] [debug] mod_manager.c(2366): manager_handler STATUS  OK

[Mon Jun 24 12:27:27 2013] [debug] mod_manager.c(1667): manager_trans INFO (/)

[Mon Jun 24 12:27:27 2013] [debug] mod_manager.c(2323): manager_handler INFO (/) processing: ""

[Mon Jun 24 12:27:27 2013] [debug] mod_manager.c(2366): manager_handler INFO  OK

[Mon Jun 24 12:27:27 2013] [debug] mod_manager.c(1667): manager_trans CONFIG (/)

[Mon Jun 24 12:27:27 2013] [debug] mod_manager.c(2323): manager_handler CONFIG (/) processing: "JVMRoute=cluster2&Host=cluster2&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp&ping=10"

[Mon Jun 24 12:27:27 2013] [debug] mod_manager.c(2366): manager_handler CONFIG  OK

[Mon Jun 24 12:27:27 2013] [debug] mod_manager.c(1667): manager_trans ENABLE-APP (/)

[Mon Jun 24 12:27:27 2013] [debug] mod_manager.c(2323): manager_handler ENABLE-APP (/) processing: "JVMRoute=cluster2&Alias=default-host%2Clocalhost%2Cexample.com&Context=%2F"

[Mon Jun 24 12:27:27 2013] [debug] mod_manager.c(2366): manager_handler ENABLE-APP  OK

[Mon Jun 24 12:27:27 2013] [debug] mod_manager.c(1667): manager_trans ENABLE-APP (/)

[Mon Jun 24 12:27:27 2013] [debug] mod_manager.c(2323): manager_handler ENABLE-APP (/) processing: "JVMRoute=cluster2&Alias=default-host%2Clocalhost%2Cexample.com&Context=%2FClusterWebApp"

[Mon Jun 24 12:27:27 2013] [debug] mod_manager.c(2366): manager_handler ENABLE-APP  OK

[Mon Jun 24 12:27:27 2013] [debug] mod_manager.c(1667): manager_trans STATUS (/)

[Mon Jun 24 12:27:27 2013] [debug] mod_manager.c(2323): manager_handler STATUS (/) processing: "JVMRoute=cluster2&Load=0"

[Mon Jun 24 12:27:27 2013] [debug] mod_manager.c(1404): Processing STATUS

[Mon Jun 24 12:27:27 2013] [debug] mod_proxy_cluster.c(1608): proxy_cluster_isup: Can't find worker for 2

[Mon Jun 24 12:27:27 2013] [debug] mod_manager.c(2366): manager_handler STATUS  OK

[Mon Jun 24 12:27:28 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:27:28 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:27:28 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:27:28 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:27:28 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:27:28 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:27:28 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:27:28 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:27:28 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:27:28 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:27:28 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:27:28 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:27:28 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:27:28 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:27:28 2013] [debug] mod_proxy_cluster.c(678): update_workers_node starting

[Mon Jun 24 12:27:28 2013] [debug] mod_proxy_cluster.c(693): update_workers_node done

[Mon Jun 24 12:27:28 2013] [debug] mod_manager.c(1667): manager_trans STATUS (/)

[Mon Jun 24 12:27:28 2013] [debug] mod_manager.c(2323): manager_handler STATUS (/) processing: "JVMRoute=cluster1&Load=52"

[Mon Jun 24 12:27:28 2013] [debug] mod_manager.c(1404): Processing STATUS

[Mon Jun 24 12:27:28 2013] [debug] proxy_util.c(2044): proxy: ajp: has acquired connection for (cluster1)

[Mon Jun 24 12:27:28 2013] [debug] proxy_util.c(2102): proxy: connecting ajp://cluster1:8009/ to cluster1:8009

[Mon Jun 24 12:27:28 2013] [debug] proxy_util.c(2195): proxy: connected ajp://cluster1:8009/ to cluster1:8009

[Mon Jun 24 12:27:28 2013] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done

[Mon Jun 24 12:27:28 2013] [debug] proxy_util.c(2062): proxy: ajp: has released connection for (cluster1)

[Mon Jun 24 12:27:28 2013] [debug] mod_manager.c(2366): manager_handler STATUS  OK

[Mon Jun 24 12:27:37 2013] [debug] mod_manager.c(1667): manager_trans STATUS (/)

[Mon Jun 24 12:27:37 2013] [debug] mod_manager.c(2323): manager_handler STATUS (/) processing: "JVMRoute=cluster2&Load=0"

[Mon Jun 24 12:27:37 2013] [debug] mod_manager.c(1404): Processing STATUS

[Mon Jun 24 12:27:37 2013] [debug] mod_proxy_cluster.c(1608): proxy_cluster_isup: Can't find worker for 2

[Mon Jun 24 12:27:37 2013] [debug] mod_manager.c(2366): manager_handler STATUS  OK

[Mon Jun 24 12:27:38 2013] [debug] mod_manager.c(1667): manager_trans STATUS (/)

[Mon Jun 24 12:27:38 2013] [debug] mod_manager.c(2323): manager_handler STATUS (/) processing: "JVMRoute=cluster1&Load=58"

[Mon Jun 24 12:27:38 2013] [debug] mod_manager.c(1404): Processing STATUS

[Mon Jun 24 12:27:38 2013] [debug] proxy_util.c(2044): proxy: ajp: has acquired connection for (cluster1)

[Mon Jun 24 12:27:38 2013] [debug] proxy_util.c(2102): proxy: connecting ajp://cluster1:8009/ to cluster1:8009

[Mon Jun 24 12:27:38 2013] [debug] proxy_util.c(2195): proxy: connected ajp://cluster1:8009/ to cluster1:8009

[Mon Jun 24 12:27:38 2013] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done

[Mon Jun 24 12:27:38 2013] [debug] proxy_util.c(2062): proxy: ajp: has released connection for (cluster1)

[Mon Jun 24 12:27:38 2013] [debug] mod_manager.c(2366): manager_handler STATUS  OK

[Mon Jun 24 12:27:47 2013] [debug] mod_manager.c(1667): manager_trans STATUS (/)

[Mon Jun 24 12:27:47 2013] [debug] mod_manager.c(2323): manager_handler STATUS (/) processing: "JVMRoute=cluster2&Load=5"

[Mon Jun 24 12:27:47 2013] [debug] mod_manager.c(1404): Processing STATUS

[Mon Jun 24 12:27:47 2013] [debug] mod_proxy_cluster.c(1608): proxy_cluster_isup: Can't find worker for 2

[Mon Jun 24 12:27:47 2013] [debug] mod_manager.c(2366): manager_handler STATUS  OK

[Mon Jun 24 12:27:48 2013] [debug] mod_manager.c(1667): manager_trans STATUS (/)

[Mon Jun 24 12:27:48 2013] [debug] mod_manager.c(2323): manager_handler STATUS (/) processing: "JVMRoute=cluster1&Load=65"

[Mon Jun 24 12:27:48 2013] [debug] mod_manager.c(1404): Processing STATUS

[Mon Jun 24 12:27:48 2013] [debug] proxy_util.c(2044): proxy: ajp: has acquired connection for (cluster1)

[Mon Jun 24 12:27:48 2013] [debug] proxy_util.c(2102): proxy: connecting ajp://cluster1:8009/ to cluster1:8009

[Mon Jun 24 12:27:48 2013] [debug] proxy_util.c(2195): proxy: connected ajp://cluster1:8009/ to cluster1:8009

[Mon Jun 24 12:27:48 2013] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done

[Mon Jun 24 12:27:48 2013] [debug] proxy_util.c(2062): proxy: ajp: has released connection for (cluster1)

[Mon Jun 24 12:27:48 2013] [debug] mod_manager.c(2366): manager_handler STATUS  OK

[Mon Jun 24 12:27:57 2013] [debug] mod_manager.c(1667): manager_trans STATUS (/)

[Mon Jun 24 12:27:57 2013] [debug] mod_manager.c(2323): manager_handler STATUS (/) processing: "JVMRoute=cluster2&Load=17"

[Mon Jun 24 12:27:57 2013] [debug] mod_manager.c(1404): Processing STATUS

[Mon Jun 24 12:27:57 2013] [debug] mod_proxy_cluster.c(1608): proxy_cluster_isup: Can't find worker for 2

[Mon Jun 24 12:27:57 2013] [debug] mod_manager.c(2366): manager_handler STATUS  OK

[mbabacek]

The message "proxy_cluster_isup: Can't find worker for 2"

means that the balancer failed to look up the structure in the shared memory...

The problem is actually your Apache httpd.  Apache/2.2.3. It is too old for mod_cluster.

Could you try with httpd at least  2.2.8+?  2.2.15+ will be the best, if you can have it.

Edit: Deleted the  "I guess..." part 'cause I'm sure :-D

[benashmead]

Perfect! Upgraded to httpd 2.2.22, and everything works perfectly - both nodes have a sensible load, and failing over from one node to the other is seamless. Thanks for the help