1 2 Previous Next 16 Replies Latest reply on Jun 12, 2007 6:10 PM by Brian Stansberry

    Clustering / Multicast routing

    Larry Ludlow Newbie

      I am running into an issue with runing multiple clustered instances of Jboss. These are on seperate servers (3 to be exact)

      Here is what I am trying to do.

      1. Route all multicast via a private network

      route add -net 224.0.0.0 netmask 240.0.0.0 dev bond1


      2. make a copy of the default server and name it foo.bar.com

      3. Use custom start and stop scripts for each instance / site
      su -l jboss -c '/apps/jboss/bin/run.sh -c foo.bar.com -b 172.25.1.52 -Djboss.partition.name=foo.bar.com --udp=224.10.10.10 > /dev/null 2> /dev/null &'


      This should work. I tested that the default server will start and clustering works (needed to set the bind_addr for multicast).

      However when I start the servers they do not see each other as cluster members. I have done a tcpdump and do see traffic from both nodes over bond1

      I am curious if there is anyhting special I need to do... did I find a bug..

      Any help would be greatly appreciated.

        • 1. Re: Clustering / Multicast routing
          Brian Stansberry Master

          Is 172.25.1.52 associated with bondl? If not, start JBoss like this:

          su -l jboss -c '/apps/jboss/bin/run.sh -c foo.bar.com -b 172.25.1.52 -Dbind.address=w.x.y.z -Djboss.partition.name=foo.bar.com --udp=224.10.10.10 > /dev/null 2> /dev/null &'


          where w.x.y.z is an IP address associated with bond1.

          Also, FYI instead of -Djboss.partition.name=foo.bar.com you can use '-g foo.bar.com' or '--partition foo.bar.com' .

          • 2. Re: Clustering / Multicast routing
            Larry Ludlow Newbie

            I have the bind address set in the cluster-service.xml

            do I need this to be different for each instance?? or can they all bind to the same address??

            <UDP mcast_addr="${jboss.partition.udpGroup:228.1.2.3}" mcast_port="45566"
             ip_ttl="${jgroups.mcast.ip_ttl:8}" ip_mcast="true"
             mcast_recv_buf_size="2000000" mcast_send_buf_size="640000"
             ucast_recv_buf_size="2000000" ucast_send_buf_size="640000"
             loopback="false" bind_addr="172.25.2.14"/>
            


            I see talking on this interface here is a portion of the tcpdump

            16:09:55.374354 IP 172.25.2.14.34585 > 224.10.10.10.45566: UDP, length 118
            
            16:10:08.754060 IP 172.25.2.13.33016 > 224.10.10.10.45566: UDP, length 118
            16:08:38.092333 IP 172.25.2.14.34582 > 224.10.10.10.45577: UDP, length 152
            16:07:55.047029 IP 172.25.2.13.33016 > 224.10.10.10.45566: UDP, length 118
            16:07:55.880378 IP 172.25.2.14.34585 > 224.10.10.10.45566: UDP, length 118
            
            


            Thanks for the -g hint, will make my scripts a look cleaner

            • 3. Re: Clustering / Multicast routing
              Larry Ludlow Newbie

              Here is my base config

              Server1
              bond0 172.25.1.13
              bond0:1 172.25.1.16
              bond1 172.25.2.13

              Server2
              bond0 172.25.1.14
              bond0:1 172.25.1.52
              bond1 172.25.2.14

              and so on for all 3 nodes. I am going to be running around 50 instances all together per node, all on separate partitions.

              • 4. Re: Clustering / Multicast routing
                Brian Stansberry Master

                If the IP passed to -b is not the one you want JGroups to use, specifying it in cluster-service.xml will not work unless you are using JGroups 2.2.8 or later and you pass a special 'ignore.bind.address' system property to java. So, you have two choices:

                1) Use JGroups 2.2.8 or later (which is a good idea anyway), and

                a) in cluster-service.xml

                <UDP bind_addr="172.25.2.14" mcast_addr="${jboss.partition.udpGroup:228.1.2.3}" mcast_port="45566" ...... />
                


                b) to start JBoss:

                su -l jboss -c '/apps/jboss/bin/run.sh -c foo.bar.com -b 172.25.1.52 -Dignore.bind.address=true -g foo.bar.com -u 224.10.10.10 > /dev/null 2> /dev/null &'



                OR

                2) With any JGroups version:

                su -l jboss -c '/apps/jboss/bin/run.sh -c foo.bar.com -b 172.25.1.52 -Dbind.address=172.25.2.14 -g foo.bar.com -u 224.10.10.10 > /dev/null 2> /dev/null &'


                No need to set the bind_addr in cluster-service.xml with option 2.


                IMO, option 2 is easier.

                Re: running 50 separate instances per machine, all on separate partitions, you can do that and you can use the same -b and -Dbind.address values for all 50. But you must use separate values for:

                1) -g (aka -Djboss.partition.name)
                2) -u
                3) the value for mcast_port in cluster-service.xml's UDP. Recommend you create a system property for this value and pass it in your startup command via -D.


                • 5. Re: Clustering / Multicast routing
                  Larry Ludlow Newbie

                  In regards to setting the mutlicast port as a variable, do I need to set a variable for both the cluster-service.xml as well as the tomcat cluster config?

                  Or can I get away with just changing the port for jgoups??

                  • 6. Re: Clustering / Multicast routing
                    Larry Ludlow Newbie

                    Brian, Thanks allot setting the port to a variable, and adding the bind.address in the start script seems to have done the trick.

                    Now all I have left to do is create a script to do this all for us :) loads of fun.

                    Larry

                    • 7. Re: Clustering / Multicast routing
                      Brian Stansberry Master

                      Glad it worked. :-)

                      I'm curious: why 50 partitions?

                      • 8. Re: Clustering / Multicast routing
                        Larry Ludlow Newbie

                        We have somewhere from 70-80 apps, and we need them to all (well not all but most) be separate, something to do with SAS70 compliance.

                        My boss came up with the idea of using a separate instance per application, except for some small apps / websites.

                        I don't exactly agree, but they are willing to give me the hardware I will do it.

                        • 9. Re: Clustering / Multicast routing
                          Brian Stansberry Master

                          Missed your question about the tomcat cluster. Yes, you should use a separate variable for that :(. With JBoss 5, this will be easier as by default all services will share a single JGroups channel. And there will be a startup switch (a la -u, -b, etc.) for setting the mcast_port for the shared channel.

                          BTW, the reason for setting the mcast_port to a different value for different groups is discussed here: http://wiki.jboss.org/wiki/Wiki.jsp?page=PromiscuousTraffic . If not for this issue, using different values for -u would be sufficient.

                          Also, if you aren't using 4.0.4 or later, edit the ClusterName attribute in tc5-cluster-service.xml for it's value is "Tomcat-${jboss.partition.name:DefaultPartition}". That will cause the value you pass to -g to be part of the name, further helping to isolate the 50 clusters. In 4.0.4 and later the default value for ClusterName uses that syntax.

                          • 10. Re: Clustering / Multicast routing
                            Larry Ludlow Newbie

                            we are using 4.0.5 I have done some testing with 4.0.4 in the past, and did see when not using different multicast addresses and ports that I get chatter in the logs from other clusters.

                            We are using the most recent jdk so I don't think the we will run into the Promiscuous Traffic issue.

                            Using jdk5 do I need to worry about using different ports?? or just a different address enough?

                            I will add another var for the tomcat cluster.
                            Thanks

                            • 11. Re: Clustering / Multicast routing
                              Brian Stansberry Master

                              The PromiscuousTraffic bug is still listed as open in the Sun database: http://bugs.sun.com/bugdatabase/view_bug.do;:YfiG?bug_id=4701650.

                              Since the exact cause of this issue is not clear, for safety reasons we recommend using different ports. If you've tested your environment thoroughly and are convinced it's not an issue and won't become one with different equipment configurations you may adopt, then you might get away with not changing it. But we recommend that you do change it.

                              • 12. Re: Clustering / Multicast routing
                                Larry Ludlow Newbie

                                Brian, thanks for all of your help. I do have another question regarding on how we are setting up the 3 nodes.

                                I am using apache with mod_jk and want to setup the cluster as follows.

                                Node1 Node2 Node3
                                Active Standby Active
                                Buddy node2 Buddy node2
                                

                                and setup the load balancer to send traffic to either node1 or node3 and only send to node2 if 1 or 3 are down

                                I want to do this to reduce the amount of memory required for running this many instances.
                                From my understanding the session state will only be replicated to node2 from either of the other active nodes unless node2 goes down.

                                Will this work? and is it worth while to do it this way??

                                • 13. Re: Clustering / Multicast routing
                                  Larry Ludlow Newbie

                                  Bump for some attention :P

                                  • 14. Re: Clustering / Multicast routing
                                    Brian Stansberry Master

                                    LOL. You're lucky. You tend to ask questions when I've got a few spare minutes to answer. :-)

                                    To get that to work you'd have to do some tricks with the configuration of buddy replication. What you want is:

                                    1) Node 1 always chooses node2 as buddy.
                                    2) Node 2 chooses either 1 or 3.
                                    3) Node 3 always chooses node2 as buddy.

                                    #2 is simple; just include all 3 nodes in the same buddy pool.

                                    For #1 and #3 you would probably need to create your own implementation of the BuddyLocator interface, one that accepts a config option identifying node2 as the preferred buddy.

                                    As to whether it's worthwhile, it depends on how much load your app's usage of session replication puts on a buddy -- how much memory and cpu is used handling providing backup for a buddy.

                                    1 2 Previous Next