It is very likely that brining a new mode, most of the new requests are going to go to that node, since this is the new node with no load.
Look at the mod_cluster management console and look at the loads -- what does it say (load field)? Is 1102 really the one with highest factor?
What load-balancing factor provider are you using? (cpu, mem, etc..).
+1 To Rado's answer.
I would add another one regarding this statement:
all the request starts going into this box
Do you use stickySession? e.g. something like:
<property name="stickySession">true</property> <property name="stickySessionForce">false</property> <property name="stickySessionRemove">false</property>
If it is so, requests that belongs to sessions created prior to server 1102 joining the balancer should indeed keep going to "their" respective servers.
Please share the logs and the mod_cluster screen shot during the issue.
That can help to troubleshoot further.
...just a one more piece of info: One must give the system some time, in matter of the amount of new sessions created, to settle and pick other nodes.
An example from an actual environment:
You have 3 nodes with the following Load values:
- Node jboss-6, Load: 20
- Node jboss-6-2, Load: 90
- Node jboss-6-3, Load: 1
Yes, this means that jboss-6-2 is almost not loaded at all whereas jboss-6-3 is desperately overloaded.
Now, I send 1001 requests, each representing a new session (the client is forgetting cookies). The distribution of the requests will be as follows:
- Node jboss-6 served 181 requests
- Node jboss-6-2 served 811 requests
- Node jboss-6-3 served 9 requests
So, generally, yes, the least loaded box received by far the greatest amount of requests, but it did not receive them all. Furthermore, and this concerns your case, for some time from the start, it was jboss-6 who was getting requests.
This whole magic is in place in order to prevent congestion.