Clustering with UDP or TCP - What is best?

[dmurphy2]

Hi - we have started trying to understand the best (most reliable, most performant) networking configuration for clustering with JBOSS AS..

It appears (from the standard, shipped service xml) that JBOSS recommends UDP multicast as the best way to support session replication and distributed caching with tree cache.

But you also provide a sample (commented) TCP based configuration. Could you JBOSS guys elaborate a little on when moving to the TCP config would be recommended? What are the reasons one might move the stack to TCP? I am assuming that JGroups deals internally with the non guaranteed nature of UDP packet receipt.

[brian.stansberry]

First, suggest you have a look at the presentation Bela Ban and I did at JBoss World 2008, which gets into this and other topics you've recently asked about: http://jbossworld.com/downloads/pdf/wednesday/JBOSS_1-150pm_ClusterTuning_Bela_Ban.pdf .

The basic answer to your question is the more servers you need to send a message to, the less well TCP will do, since a server has to send n messages to communicate with n peers, while UDP just sends one. A TCP-based stack works fine for smaller clusters (4-6 or less), and if you use buddy replication you're not sending messages to all peers, so TCP can be suitable even with a bigger cluster.

Note that if you use a TCP-based stack, you either need to list all hosts in the TCPPING protocol's initial_hosts parameter, or you should replace TCPPING with the MPING protocol, which uses UDP multicast for group discovery.

[dmurphy2]

Ok thanks Brian.

My instinct is to go with the default configuration - UDP. And I also understand from your comments why this configuration will be the best for most cases.

The reason I ask is that about 3 years ago (before my time) it was reported that with our app (then using I think JBOSS 2.x) running Windows 2000 that when using UDP for JGroups some app servers would periodically hang while booting. This behavior was intermittent but when they hung there were error messages in the JGroups logs. I don't believe the root cause was ever discovered and so the decision was made to move to a TCP based configuration using the Gossip Router. This apparently stopped the 'hang during boot' behavior but we never understood the original problem.

Since then we have read MS articles saying that Windows networking using multi-homed machines (which our app servers are) can have trouble when talking UDP. MS have a patch for this and we will try that.

My question : have you guys at JBOSS ever heard of problem clustering JBOSS using UDP under Windows? If so, any details (especially resolution steps) you may have come across would be most welcome.

We are now planning to start using full blown JBOSS clustering (session replication, tree cache) etc and I would like to be able to use UDP for all networking configurations.

thx
David

[brian.stansberry]

The one thing I've heard about with Windows is some sort of conflict if you use Network Load Balancer (which is also UDP based). But I don't really know any details; IIRC there was a forum post about a possible conflict, but it didn't really get into specifics.

I've certainly done plenty of UDP clustering of multi-homed Windows machines. If you run the AS clustering testsuite, it deploys two AS instances on the same box and clusters them to run tests. On Windows only way to do that is to multihome the machine.

[billzhou]

Hi,
I am using UDP for clustering. My question is what are the port range for udpPort and udpHttpPort.

JAVA_OPTS="$JAVA_OPTS -Djboss.partition.name=iqueacluster -Djboss.partition.udpGroup=239.255.60.61 -Djboss.partition.udpPort=66666 -Djboss.partition.udpHttpPort=77777 -Dprogram.name=$PROGNAME"

From the above examle, I am using port 66666 and port 77777. However the file /proc/sys/net/ipv4/ip_local_port_range has this value:
32768 61000

Do I need to adjust the above port to between port 32768 and 61000? If I do that, how come jboss port 8080 and 80 are working?

Thanks,

Bill