Using distributed mode greater than 100 nodes

[jworrall]

Hello,

If one wants to run distributed mode that contains more than 100 nodes in a cluster, which is the best transport mode? In the guide it suggest to use TCP for distributed mode but recommended for 100 nodes or less, so I am not sure as it is not very clear to me.

Can someone provide me some clarity?

Thanks!

[pruivo]

Hi,

TCP is not recommended for large cluster because it needs a socket for each node to send/receive messages. In a 100 nodes cluster, each node would have 99 socket open. Of course, each socket is not free and it is consuming the machine resources.

UDP is recommended because it uses a single socket to send/receive messages from all the nodes. In other words, it does not depends on the number of nodes in the cluster.

Cheers,

Pedro