mod_cluster load balancing problem

[tirtha2u]

Hi,

I am running with 4 servers(1101, 1102, 1103 and 1104) in a cluster (JBoss EAP 5.1.1).These servers are LB with Apache and mod_cluster. The problem I am facing is whenever I start 1102, all the request starts going into this box. If I stop the 1102 instance then the loads are equally getting distributed. The boxes are VM boxes, and we tried rebuilding the box 1102 from scratch as well.

Need your help in debugging the cause of this issue. We are using introscope to monitor these servers.

[rhusar]

It is very likely that brining a new mode, most of the new requests are going to go to that node, since this is the new node with no load.

Look at the mod_cluster management console and look at the loads -- what does it say (load field)? Is 1102 really the one with highest factor?

What load-balancing factor provider are you using? (cpu, mem, etc..).

[mbabacek]

+1 To Rado's answer.

I would add another one regarding this statement:

all the request starts going into this box

Do you use stickySession? e.g. something like:

<property name="stickySession">true</property>
<property name="stickySessionForce">false</property>
<property name="stickySessionRemove">false</property>

If it is so, requests that belongs to sessions created prior to server 1102 joining the balancer should indeed keep going to "their" respective servers.

[kuldeep11]

Please share the logs and the mod_cluster screen shot during the issue.

That can help to troubleshoot further.

[mbabacek]

...just a one more piece of info: One must give the system some time, in matter of the amount of new sessions created, to settle and pick other nodes.

An example from an actual environment:

You have 3 nodes with the following Load values:

• Node jboss-6, Load: 20
• Node jboss-6-2, Load: 90
• Node jboss-6-3, Load: 1

Yes, this means that jboss-6-2 is almost not loaded at all whereas jboss-6-3 is desperately overloaded.

Now, I send 1001 requests, each representing a new session (the client is forgetting cookies). The distribution of the requests will be as follows:

• Node jboss-6 served 181 requests
• Node jboss-6-2 served 811 requests
• Node jboss-6-3 served 9 requests

So, generally, yes, the least loaded box received by far the greatest amount of requests, but it did not receive them all. Furthermore, and this concerns your case, for some time from the start, it was jboss-6 who was getting requests.

This whole magic is in place in order to prevent congestion.