-
1. Re: Clustering / Multicast routing
brian.stansberry Feb 21, 2007 4:21 PM (in response to lludlow2)Is 172.25.1.52 associated with bondl? If not, start JBoss like this:
su -l jboss -c '/apps/jboss/bin/run.sh -c foo.bar.com -b 172.25.1.52 -Dbind.address=w.x.y.z -Djboss.partition.name=foo.bar.com --udp=224.10.10.10 > /dev/null 2> /dev/null &'
where w.x.y.z is an IP address associated with bond1.
Also, FYI instead of -Djboss.partition.name=foo.bar.com you can use '-g foo.bar.com' or '--partition foo.bar.com' . -
2. Re: Clustering / Multicast routing
lludlow2 Feb 21, 2007 4:29 PM (in response to lludlow2)I have the bind address set in the cluster-service.xml
do I need this to be different for each instance?? or can they all bind to the same address??<UDP mcast_addr="${jboss.partition.udpGroup:228.1.2.3}" mcast_port="45566" ip_ttl="${jgroups.mcast.ip_ttl:8}" ip_mcast="true" mcast_recv_buf_size="2000000" mcast_send_buf_size="640000" ucast_recv_buf_size="2000000" ucast_send_buf_size="640000" loopback="false" bind_addr="172.25.2.14"/>
I see talking on this interface here is a portion of the tcpdump16:09:55.374354 IP 172.25.2.14.34585 > 224.10.10.10.45566: UDP, length 118 16:10:08.754060 IP 172.25.2.13.33016 > 224.10.10.10.45566: UDP, length 118 16:08:38.092333 IP 172.25.2.14.34582 > 224.10.10.10.45577: UDP, length 152 16:07:55.047029 IP 172.25.2.13.33016 > 224.10.10.10.45566: UDP, length 118 16:07:55.880378 IP 172.25.2.14.34585 > 224.10.10.10.45566: UDP, length 118
Thanks for the -g hint, will make my scripts a look cleaner -
3. Re: Clustering / Multicast routing
lludlow2 Feb 21, 2007 4:32 PM (in response to lludlow2)Here is my base config
Server1
bond0 172.25.1.13
bond0:1 172.25.1.16
bond1 172.25.2.13
Server2
bond0 172.25.1.14
bond0:1 172.25.1.52
bond1 172.25.2.14
and so on for all 3 nodes. I am going to be running around 50 instances all together per node, all on separate partitions. -
4. Re: Clustering / Multicast routing
brian.stansberry Feb 21, 2007 4:57 PM (in response to lludlow2)If the IP passed to -b is not the one you want JGroups to use, specifying it in cluster-service.xml will not work unless you are using JGroups 2.2.8 or later and you pass a special 'ignore.bind.address' system property to java. So, you have two choices:
1) Use JGroups 2.2.8 or later (which is a good idea anyway), and
a) in cluster-service.xml<UDP bind_addr="172.25.2.14" mcast_addr="${jboss.partition.udpGroup:228.1.2.3}" mcast_port="45566" ...... />
b) to start JBoss:su -l jboss -c '/apps/jboss/bin/run.sh -c foo.bar.com -b 172.25.1.52 -Dignore.bind.address=true -g foo.bar.com -u 224.10.10.10 > /dev/null 2> /dev/null &'
OR
2) With any JGroups version:su -l jboss -c '/apps/jboss/bin/run.sh -c foo.bar.com -b 172.25.1.52 -Dbind.address=172.25.2.14 -g foo.bar.com -u 224.10.10.10 > /dev/null 2> /dev/null &'
No need to set the bind_addr in cluster-service.xml with option 2.
IMO, option 2 is easier.
Re: running 50 separate instances per machine, all on separate partitions, you can do that and you can use the same -b and -Dbind.address values for all 50. But you must use separate values for:
1) -g (aka -Djboss.partition.name)
2) -u
3) the value for mcast_port in cluster-service.xml's UDP. Recommend you create a system property for this value and pass it in your startup command via -D. -
5. Re: Clustering / Multicast routing
lludlow2 Feb 22, 2007 7:38 AM (in response to lludlow2)In regards to setting the mutlicast port as a variable, do I need to set a variable for both the cluster-service.xml as well as the tomcat cluster config?
Or can I get away with just changing the port for jgoups?? -
6. Re: Clustering / Multicast routing
lludlow2 Feb 22, 2007 9:41 AM (in response to lludlow2)Brian, Thanks allot setting the port to a variable, and adding the bind.address in the start script seems to have done the trick.
Now all I have left to do is create a script to do this all for us :) loads of fun.
Larry -
7. Re: Clustering / Multicast routing
brian.stansberry Feb 22, 2007 9:53 AM (in response to lludlow2)Glad it worked. :-)
I'm curious: why 50 partitions? -
8. Re: Clustering / Multicast routing
lludlow2 Feb 22, 2007 10:01 AM (in response to lludlow2)We have somewhere from 70-80 apps, and we need them to all (well not all but most) be separate, something to do with SAS70 compliance.
My boss came up with the idea of using a separate instance per application, except for some small apps / websites.
I don't exactly agree, but they are willing to give me the hardware I will do it. -
9. Re: Clustering / Multicast routing
brian.stansberry Feb 22, 2007 10:12 AM (in response to lludlow2)Missed your question about the tomcat cluster. Yes, you should use a separate variable for that :(. With JBoss 5, this will be easier as by default all services will share a single JGroups channel. And there will be a startup switch (a la -u, -b, etc.) for setting the mcast_port for the shared channel.
BTW, the reason for setting the mcast_port to a different value for different groups is discussed here: http://wiki.jboss.org/wiki/Wiki.jsp?page=PromiscuousTraffic . If not for this issue, using different values for -u would be sufficient.
Also, if you aren't using 4.0.4 or later, edit the ClusterName attribute in tc5-cluster-service.xml for it's value is "Tomcat-${jboss.partition.name:DefaultPartition}". That will cause the value you pass to -g to be part of the name, further helping to isolate the 50 clusters. In 4.0.4 and later the default value for ClusterName uses that syntax. -
10. Re: Clustering / Multicast routing
lludlow2 Feb 22, 2007 10:21 AM (in response to lludlow2)we are using 4.0.5 I have done some testing with 4.0.4 in the past, and did see when not using different multicast addresses and ports that I get chatter in the logs from other clusters.
We are using the most recent jdk so I don't think the we will run into the Promiscuous Traffic issue.
Using jdk5 do I need to worry about using different ports?? or just a different address enough?
I will add another var for the tomcat cluster.
Thanks -
11. Re: Clustering / Multicast routing
brian.stansberry Feb 22, 2007 11:09 AM (in response to lludlow2)The PromiscuousTraffic bug is still listed as open in the Sun database: http://bugs.sun.com/bugdatabase/view_bug.do;:YfiG?bug_id=4701650.
Since the exact cause of this issue is not clear, for safety reasons we recommend using different ports. If you've tested your environment thoroughly and are convinced it's not an issue and won't become one with different equipment configurations you may adopt, then you might get away with not changing it. But we recommend that you do change it. -
12. Re: Clustering / Multicast routing
lludlow2 Feb 22, 2007 12:48 PM (in response to lludlow2)Brian, thanks for all of your help. I do have another question regarding on how we are setting up the 3 nodes.
I am using apache with mod_jk and want to setup the cluster as follows.Node1 Node2 Node3 Active Standby Active Buddy node2 Buddy node2
and setup the load balancer to send traffic to either node1 or node3 and only send to node2 if 1 or 3 are down
I want to do this to reduce the amount of memory required for running this many instances.
From my understanding the session state will only be replicated to node2 from either of the other active nodes unless node2 goes down.
Will this work? and is it worth while to do it this way?? -
13. Re: Clustering / Multicast routing
lludlow2 Feb 28, 2007 7:29 AM (in response to lludlow2)Bump for some attention :P
-
14. Re: Clustering / Multicast routing
brian.stansberry Feb 28, 2007 11:05 AM (in response to lludlow2)LOL. You're lucky. You tend to ask questions when I've got a few spare minutes to answer. :-)
To get that to work you'd have to do some tricks with the configuration of buddy replication. What you want is:
1) Node 1 always chooses node2 as buddy.
2) Node 2 chooses either 1 or 3.
3) Node 3 always chooses node2 as buddy.
#2 is simple; just include all 3 nodes in the same buddy pool.
For #1 and #3 you would probably need to create your own implementation of the BuddyLocator interface, one that accepts a config option identifying node2 as the preferred buddy.
As to whether it's worthwhile, it depends on how much load your app's usage of session replication puts on a buddy -- how much memory and cpu is used handling providing backup for a buddy.