5 Replies Latest reply on Nov 9, 2012 2:21 AM by mbabacek

load balancing in mod_cluster does not seem to work properly with 3 nodes - node 3 is never elected

claudianus Nov 8, 2012 6:30 PM

Hello -

I am seeing some behavior that does not seem right to me with mod_cluster and I was wondering if this is how it is supposed to work.

I am running three EAP 6 nodes fronted by apache 2 on a ubuntu 12.04 OS. Each of the servers (jboss and apache) has its own ip (virtual or physical). I am running mod_cluster 1.2 and EAP 6 for each of the jboss nodes. I am starting them with standalone-ha.xml

I have configured apache to use various settings.

Conf 1

CreateBalancers 1

<IfModule manager_module>

Listen 192.168.10.25:9000

<VirtualHost 192.168.10.25:9000>

ProxyPass / balancer://mycluster/ stickysession=JSESSIONID|jsessionid nofailover=On

ProxyPassReverse / balancer://mycluster

ProxyPreserveHost On

<Directory />

Order deny,allow

Deny from all

Allow from 192.168.10.

</Directory>

<Location /mod_cluster_manager>

SetHandler mod_cluster-manager

Order deny,allow

Deny from all

Allow from 192.168.10.

</Location>

ManagerBalancerName mycluster

KeepAliveTimeout 300

MaxKeepAliveRequests 0

AdvertiseFrequency 5

ServerAdvertise On

EnableMCPMReceive

</VirtualHost>

</IfModule>

Then I configured it like below

Conf 2

CreateBalancers 2

<IfModule manager_module>

Listen 192.168.10.25:9000

<VirtualHost 192.168.10.25:9000>

#ProxyPass / balancer://mycluster/ stickysession=JSESSIONID|jsessionid nofailover=On

#ProxyPassReverse / balancer://mycluster

#ProxyPreserveHost On

<Directory />

Order deny,allow

Deny from all

Allow from 192.168.10.

</Directory>

<Location /mod_cluster_manager>

SetHandler mod_cluster-manager

Order deny,allow

Deny from all

Allow from 192.168.10.

</Location>

ManagerBalancerName mycluster

KeepAliveTimeout 300

MaxKeepAliveRequests 0

AdvertiseFrequency 5

ServerAdvertise On

EnableMCPMReceive

</VirtualHost>

</IfModule>

Both configurations work.

My AS7 config for mod_cluster looks like below for each of the nodes:

conf 1

<subsystem xmlns="urn:jboss:domain:modcluster:1.1">

<mod-cluster-config advertise-socket="modcluster" proxy-list="192.168.10.25:9000" proxy-url="/" advertise="true" excluded-contexts="ROOT,invoker,jbossws,juddi,console" auto-enable-contexts="true" stop-context-timeout="10" socket-timeout="20" sticky-session="true" sticky-session-remove="false" sticky-session-force="false" worker-timeout="-1" max-attempts="1" flush-packets="false" flush-wait="-1" ping="10" smax="-1" ttl="-1" node-timeout="-1" connector="ajp">

<simple-load-provider />

</mod-cluster-config>

</subsystem>

conf 2

<subsystem xmlns="urn:jboss:domain:modcluster:1.1">

<dynamic-load-provider history="0" decay="2">

<load-metric type="busyness"/>

<load-metric type="sessions" weight="2" capacity="512"/>

</dynamic-load-provider>

</mod-cluster-config>

</subsystem>

I have tried various configurations and can never get apache to elect node 3.

It always elect node 1 an then node 2 but almost never node 3.

The only time I saw it elect node 3 was when I had two different web apps deployed and have been hitting the first one which was routed to node 1 or node 2 and then I tried to hit the second app and at that point node 3 was elected and that was the only time I ever saw it elected.

It this how it is supposed to work?

Why wouldn't it elect node 3? I tried various combination of dynamic load balancing policies and also the simple load provider to no avail.

Has anyone else seeing this? What am I missing?

Thanks,

1. Re: load balancing in mod_cluster does not seem to work properly with 3 nodes - node 3 is never elected

mbabacek Nov 8, 2012 7:26 PM (in response to claudianus)

Hi Claudianus, I've done a quick test with a simple configuration and everything seems to be in order.

In my 3 nodes setup, all of them have been utilized quite equally: elected-3-nodes.png

Version being Mod_cluster 1.2.3.Final. Take a look:

httpd

<IfModule manager_module>
  Listen perf05:8888
  ManagerBalancerName qacluster
  <VirtualHost perf05:8888>
    <Location />
     Order deny,allow
     Deny from all
     # Just for playing...quite not suitable for production :-)
     Allow from all
    </Location>

    KeepAliveTimeout 60
    MaxKeepAliveRequests 0
    #ServerAdvertise on http://@IP@:port
    AdvertiseFrequency 5
    #AdvertiseSecurityKey secret
    AdvertiseGroup 224.0.1.105:6666
    EnableMCPMReceive

    <Location /mcm>
       SetHandler mod_cluster-manager
       Order deny,allow
       Deny from all
       # :-)
       Allow from all
    </Location>

  </VirtualHost>
</IfModule>

worker node (AS7)

<subsystem xmlns="urn:jboss:domain:modcluster:1.1">
    <mod-cluster-config advertise-socket="modcluster" advertise="true" sticky-session="true" sticky-session-force="false" sticky-session-remove="false" connector="ajp">
        <dynamic-load-provider decay="2" history="10">
        <!-- Metric #0--><load-metric weight="1" type="busyness"/>
        </dynamic-load-provider>
    </mod-cluster-config>
</subsystem>

I would suggest sticking to defaults as long as you are not up to any special configuration. By adding one config attribute at a time, we might find out what is actually causing your problem. I suppose the load was the same on all the three boxes? Note that the busyness metric calculation is related to the number of available processors (something like 32 * Runtime.getRuntime().availableProcessors() is the default available thread pool...).

HTH

1 of 1 people found this helpful

2. Re: load balancing in mod_cluster does not seem to work properly with 3 nodes - node 3 is never elected

claudianus Nov 8, 2012 8:57 PM (in response to mbabacek)

Hello Michael -
Thank you for the quick reply. I will try 1.2.3 and see how it goes.
I actuallystarted with the default and after mutiple trials I was not able to get node 3 to be elected and that is when I started playing with the configurations.
As for the load yes it is the same or at least supposed to be the same. All 3 EAP 6 nodes and the apache server are running on the same Physiscal box. I just used virtual ip to isolate them. So the traits and processors configuration should be the same.

I will report back once I redo the test with 1.2.3.

BTW is 1.2.3.Final out? I only have 1.2.0.Final

Regards,

C
Actions
3. Re: load balancing in mod_cluster does not seem to work properly with 3 nodes - node 3 is never elected

claudianus Nov 8, 2012 10:39 PM (in response to claudianus)

Hello Michael -
I followed your advice and got the source from github (latest version I built was 1.2.2) and built it but mostly for the native which I then re-installed into apache2.
I then updated my apache conf to look like this

<IfModule manager_module>
    Listen 192.168.10.25:9000

    <VirtualHost 192.168.10.25:9000>

    <Directory />
          Order deny,allow
      Deny from all
      Allow from 192.168.10.
        </Directory>

    <Location /mod_cluster_manager>
      SetHandler mod_cluster-manager
      Order deny,allow
      Deny from all
      Allow from 192.168.10.
    </Location>

    ManagerBalancerName mycluster
        KeepAliveTimeout 60
        MaxKeepAliveRequests 0
        AdvertiseFrequency 5
        ServerAdvertise On
    EnableMCPMReceive

     </VirtualHost>
</IfModule>

I also updated my worker nodes like below

<subsystem xmlns="urn:jboss:domain:modcluster:1.1">
<mod-cluster-config advertise-socket="modcluster" proxy-list="192.168.10.25:9000" proxy-url="/" advertise="true" excluded-contexts="ROOT,invoker,jbossws,juddi,console" auto-enable-contexts="true" stop-context-timeout="10" socket-timeout="20" sticky-session="true" sticky-session-remove="false" sticky-session-force="false" worker-timeout="-1" max-attempts="1" flush-packets="false" flush-wait="-1" ping="10" smax="-1" ttl="-1" node-timeout="-1" connector="ajp">
                <dynamic-load-provider history="10" decay="2">
                    <load-metric type="busyness"/>
                    
                </dynamic-load-provider>
            </mod-cluster-config>

        </subsystem>

However I am still seeing the behavior I described where node 3 is never elected.
To test it have launched my app from 5 to 10 different browsers and most of the calls were routed to node 1 and occasionally to node 2.
I don't know what else I am missing.
Actions
4. Re: load balancing in mod_cluster does not seem to work properly with 3 nodes - node 3 is never elected

claudianus Nov 8, 2012 10:50 PM (in response to claudianus)

Hello Michael -
it looks like it is working now or at least somewhat. Basically I removed the proxy-list from the mod-cluster-fig section of the worker config and restarted apache2 as well as restarting all three EAP 6 nodes. Now node 3 got elected and is actually the dominant node as it has got elected 8 times while node 1 only got elected 2 times and node 2 got elected once.
I will keep testing but for now I will consider this issue resolved.

Regards,

C
1 of 1 people found this helpful
Actions
5. Re: load balancing in mod_cluster does not seem to work properly with 3 nodes - node 3 is never elected

mbabacek Nov 9, 2012 2:21 AM (in response to claudianus)

IMHO you are not getting the picture right: It is not any kind of round robin here. The balancer is trying to send the request to the node that has all the session data on it in order to avoid costly look up on another node (session replication, <distributed/> allows this in you web app).

Don't bother with

has got elected 8 times while node 1 only got elected 2 times and node 2 got elected once
send a coupe of thousands of requests in many simultaneous sessions and then compare node utilization.
Actions

Go to original post