Clustering / Multicast routing

[lludlow2]

I am running into an issue with runing multiple clustered instances of Jboss. These are on seperate servers (3 to be exact)

Here is what I am trying to do.

1. Route all multicast via a private network

route add -net 224.0.0.0 netmask 240.0.0.0 dev bond1

2. make a copy of the default server and name it foo.bar.com

3. Use custom start and stop scripts for each instance / site

su -l jboss -c '/apps/jboss/bin/run.sh -c foo.bar.com -b 172.25.1.52 -Djboss.partition.name=foo.bar.com --udp=224.10.10.10 > /dev/null 2> /dev/null &'

This should work. I tested that the default server will start and clustering works (needed to set the bind_addr for multicast).

However when I start the servers they do not see each other as cluster members. I have done a tcpdump and do see traffic from both nodes over bond1

I am curious if there is anyhting special I need to do... did I find a bug..

Any help would be greatly appreciated.

[brian.stansberry]

Is 172.25.1.52 associated with bondl? If not, start JBoss like this:

su -l jboss -c '/apps/jboss/bin/run.sh -c foo.bar.com -b 172.25.1.52 -Dbind.address=w.x.y.z -Djboss.partition.name=foo.bar.com --udp=224.10.10.10 > /dev/null 2> /dev/null &'

where w.x.y.z is an IP address associated with bond1.

Also, FYI instead of -Djboss.partition.name=foo.bar.com you can use '-g foo.bar.com' or '--partition foo.bar.com' .

[lludlow2]

I have the bind address set in the cluster-service.xml

do I need this to be different for each instance?? or can they all bind to the same address??

<UDP mcast_addr="${jboss.partition.udpGroup:228.1.2.3}" mcast_port="45566"
 ip_ttl="${jgroups.mcast.ip_ttl:8}" ip_mcast="true"
 mcast_recv_buf_size="2000000" mcast_send_buf_size="640000"
 ucast_recv_buf_size="2000000" ucast_send_buf_size="640000"
 loopback="false" bind_addr="172.25.2.14"/>

I see talking on this interface here is a portion of the tcpdump

16:09:55.374354 IP 172.25.2.14.34585 > 224.10.10.10.45566: UDP, length 118

16:10:08.754060 IP 172.25.2.13.33016 > 224.10.10.10.45566: UDP, length 118
16:08:38.092333 IP 172.25.2.14.34582 > 224.10.10.10.45577: UDP, length 152
16:07:55.047029 IP 172.25.2.13.33016 > 224.10.10.10.45566: UDP, length 118
16:07:55.880378 IP 172.25.2.14.34585 > 224.10.10.10.45566: UDP, length 118

Thanks for the -g hint, will make my scripts a look cleaner

[lludlow2]

Here is my base config

Server1
bond0 172.25.1.13
bond0:1 172.25.1.16
bond1 172.25.2.13

Server2
bond0 172.25.1.14
bond0:1 172.25.1.52
bond1 172.25.2.14

and so on for all 3 nodes. I am going to be running around 50 instances all together per node, all on separate partitions.

[brian.stansberry]

If the IP passed to -b is not the one you want JGroups to use, specifying it in cluster-service.xml will not work unless you are using JGroups 2.2.8 or later and you pass a special 'ignore.bind.address' system property to java. So, you have two choices:

1) Use JGroups 2.2.8 or later (which is a good idea anyway), and

a) in cluster-service.xml

<UDP bind_addr="172.25.2.14" mcast_addr="${jboss.partition.udpGroup:228.1.2.3}" mcast_port="45566" ...... />

b) to start JBoss:

su -l jboss -c '/apps/jboss/bin/run.sh -c foo.bar.com -b 172.25.1.52 -Dignore.bind.address=true -g foo.bar.com -u 224.10.10.10 > /dev/null 2> /dev/null &'

OR

2) With any JGroups version:

su -l jboss -c '/apps/jboss/bin/run.sh -c foo.bar.com -b 172.25.1.52 -Dbind.address=172.25.2.14 -g foo.bar.com -u 224.10.10.10 > /dev/null 2> /dev/null &'

No need to set the bind_addr in cluster-service.xml with option 2.


IMO, option 2 is easier.

Re: running 50 separate instances per machine, all on separate partitions, you can do that and you can use the same -b and -Dbind.address values for all 50. But you must use separate values for:

1) -g (aka -Djboss.partition.name)
2) -u
3) the value for mcast_port in cluster-service.xml's UDP. Recommend you create a system property for this value and pass it in your startup command via -D.

[lludlow2]

In regards to setting the mutlicast port as a variable, do I need to set a variable for both the cluster-service.xml as well as the tomcat cluster config?

Or can I get away with just changing the port for jgoups??

[lludlow2]

Brian, Thanks allot setting the port to a variable, and adding the bind.address in the start script seems to have done the trick.

Now all I have left to do is create a script to do this all for us :) loads of fun.

Larry

[brian.stansberry]

Glad it worked. :-)

I'm curious: why 50 partitions?

[lludlow2]

We have somewhere from 70-80 apps, and we need them to all (well not all but most) be separate, something to do with SAS70 compliance.

My boss came up with the idea of using a separate instance per application, except for some small apps / websites.

I don't exactly agree, but they are willing to give me the hardware I will do it.

[brian.stansberry]

Missed your question about the tomcat cluster. Yes, you should use a separate variable for that :(. With JBoss 5, this will be easier as by default all services will share a single JGroups channel. And there will be a startup switch (a la -u, -b, etc.) for setting the mcast_port for the shared channel.

BTW, the reason for setting the mcast_port to a different value for different groups is discussed here: http://wiki.jboss.org/wiki/Wiki.jsp?page=PromiscuousTraffic . If not for this issue, using different values for -u would be sufficient.

Also, if you aren't using 4.0.4 or later, edit the ClusterName attribute in tc5-cluster-service.xml for it's value is "Tomcat-${jboss.partition.name:DefaultPartition}". That will cause the value you pass to -g to be part of the name, further helping to isolate the 50 clusters. In 4.0.4 and later the default value for ClusterName uses that syntax.

[lludlow2]

we are using 4.0.5 I have done some testing with 4.0.4 in the past, and did see when not using different multicast addresses and ports that I get chatter in the logs from other clusters.

We are using the most recent jdk so I don't think the we will run into the Promiscuous Traffic issue.

Using jdk5 do I need to worry about using different ports?? or just a different address enough?

I will add another var for the tomcat cluster.
Thanks

[brian.stansberry]

The PromiscuousTraffic bug is still listed as open in the Sun database: http://bugs.sun.com/bugdatabase/view_bug.do;:YfiG?bug_id=4701650.

Since the exact cause of this issue is not clear, for safety reasons we recommend using different ports. If you've tested your environment thoroughly and are convinced it's not an issue and won't become one with different equipment configurations you may adopt, then you might get away with not changing it. But we recommend that you do change it.

[lludlow2]

Brian, thanks for all of your help. I do have another question regarding on how we are setting up the 3 nodes.

I am using apache with mod_jk and want to setup the cluster as follows.

Node1 Node2 Node3
Active Standby Active
Buddy node2 Buddy node2

and setup the load balancer to send traffic to either node1 or node3 and only send to node2 if 1 or 3 are down

I want to do this to reduce the amount of memory required for running this many instances.
From my understanding the session state will only be replicated to node2 from either of the other active nodes unless node2 goes down.

Will this work? and is it worth while to do it this way??

[lludlow2]

Bump for some attention :P

[brian.stansberry]

LOL. You're lucky. You tend to ask questions when I've got a few spare minutes to answer. :-)

To get that to work you'd have to do some tricks with the configuration of buddy replication. What you want is:

1) Node 1 always chooses node2 as buddy.
2) Node 2 chooses either 1 or 3.
3) Node 3 always chooses node2 as buddy.

#2 is simple; just include all 3 nodes in the same buddy pool.

For #1 and #3 you would probably need to create your own implementation of the BuddyLocator interface, one that accepts a config option identifying node2 as the preferred buddy.

As to whether it's worthwhile, it depends on how much load your app's usage of session replication puts on a buddy -- how much memory and cpu is used handling providing backup for a buddy.