Note: this document is intended as JBoss Application Server documentation and not as general JGroups documentation. Some of the content may be useful in a general sense, but it is written from the point of view of configuring JBoss AS clusters.

I want to have two JBoss AS clusters on the same network and I don't want them to see each other

First, what do we mean by a cluster here? Short answer: we're talking about a JGroups Channel.

Longer answer: We're talking about a service, an instance of which is deployed on several JBoss AS instances, where the different instances communicate with each other in order to provide a unified view to clients.

Within the AS, there are a number such services -- 3 different JBoss Cache services (used for HttpSession replication, EJB3 SFSB replication and EJB3 entity replication) along with a general purpose clustering service called HAPartition that underlies most other JBossHA services.

Each instance of these services creates a JGroups Channel and uses it to communicate with its peers.

In AS 5 or EAP 4.3 the JBoss Messaging service opens a further two channels.

It is critical that these channels only communicate with their intended peers; not with the channels used by other services and not with channels for the same service opened on machines not meant to be part of the group.

Isolating JGroups Channels

A JGroups Channel is defined by its group name, multicast address, and multicast port.

To isolate JGroups channels for different services on the same set of AS instances from each other, you MUST use a unique group name for each and either 1) use a different multicast port for each or 2) have the channels share a JGroups shared transport (available beginning with AS 5.0.0) .

For example, say we have a production cluster of 3 machines, each of which has an HAPartition deployed along with a JBoss Cache used for web session clustering.  The HAPartition channels should not communicate with the JBoss Cache channels.  Those channels should use a different group name and should either use a different multicast port or share a JGroups transport. They can use the same multicast address, although they don't need to. The default configs that ship in AS 4.0.4 and later will use different group names and multicast ports for the standard AS clustering services, while in AS 5.0.0 and later they will use a shared transport. So isolating the channels for different services requires no effort.

To isolate JGroups channels for the same service from other instances of the service on the network, you MUST change ALL three values on 4.x and earlier. Each channel must have its own group name, multicast address, and multicast port. Due to improvements in JGroups, in AS 5.x and later it is sufficient to change just the group name and multicast address; changing multicast ports should not be necessary in AS 5.

For example, say we have a production cluster of 3 machines, each of which has an HAPartition deployed.  On the same network there is also a QA cluster of 3 machines, which also has an HAPartition deployed. The HAPartition group name, multicast address, and multicast port (in AS 4.x and earlier) for the production machines must be different from those used on the QA machines.

Changing the Group Name

The group name for a JGroups channel is configured via the service that starts the channel.  Unfortunately, different services use different attribute names for configuring this. For HAPartition and related services configured in the deploy/cluster-service.xml file (deploy/cluster/hapartition-jboss-beans.xml in AS 5), this is configured via a PartitionName attribute.  For JBoss Cache services, the name of the attribute is ClusterName.

Starting with JBoss AS 4.0.2, you can easily change the cluster-service.xml PartitionName by specifying it as a system property at startup.

./run.sh -c all -Djboss.partition.name=MyPartition

Starting with JBoss AS 4.0.3, you can use a command line switch instead of setting a system property:

./run.sh -c all -g MyPartition

or

./run.sh -c all --partition MyPartition

Behind the scenes, both of these set the jboss.partition.name system property.

See Changing a Cluster PartitionName for more on this.

Starting with JBoss AS 4.0.4, the value passed at startup via -g or --partition or -Djboss.partition.name is used as part of the ClusterName used by JBoss Cache for the web session replication cache.  For earlier releases, the tc5-cluster-service.xml file can easily be modified to provide this behavior:

<attribute name="ClusterName">Tomcat-${jboss.partition.name:Cluster}</attribute>

Changing the Multicast Address

Starting with JBoss AS 4.0.3, the jboss.partition.udpGroup system property can be used to control the multicast address used by the JGroups channel opened by all standard AS services. A command line option to set this property was also added:

./run.sh -c all -u 233.3.4.5

or

./run.sh -c all --udp 233.3.4.5

Behind the scenes, both of these set the jboss.partition.udpGroup system property.

Changing the Multicast Port

(Note: improvements in JGroups mean changing multicast ports should not be necessary in AS 5.)

Changing the multicast port involves changing the protocol stack configuration for the various JGroups channels. The configuration for any JGroups channel that uses the UDP or MPING protocols should be changed. (MPING is not used in any of the standard protocol stack configurations in AS 4.x or 3.x; it is used in AS 5).  With both UDP and MPING, it is the mcast_port attribute that needs to be changed:

....
<config>
   <UDP mcast_addr="${jboss.partition.udpGroup:228.1.2.3}" 
        mcast_port="45566"
        ip_ttl="8" ip_mcast="true"
        mcast_send_buf_size="800000" mcast_recv_buf_size="150000"
        ucast_send_buf_size="800000" ucast_recv_buf_size="150000"
        loopback="false"/>
    ....

MPING is used in a TCP-based stack:

....
<config>
   <TCP start_port="7600"
        ....
   />
   <MPING timeout="3000"
          num_initial_members="3"
          mcast_addr="${jboss.partition.udpGroup:230.11.11.11}"
          mcast_port="45700"
          ip_ttl="${jgroups.udp.ip_ttl:2}"/>   
....

Where the JGroups channel configurations are located varies between AS versions:

JBoss AS 5

Improvements in JGroups mean changing multicast ports should not be necessary in AS 5. However, the following information on how to change multicast ports in AS 5 is provided here as general documentation.

In AS 5, all JGroups protocol stack configuration is isolated in a single file: /deploy/cluster/jgroups-channelfactory.sar/META-INF/jgroups-channelfactory-stacks.xml. This file contains a number of protocol stack configurations used by the AS's JGroups ChannelFactory. Not all stacks described in that file are actually used; by default in AS 5.0.0 only the udp, jbm-control and jbm-data stacks are used. (Services like the clustered HttpSession cache that use JGroups can be configured to use other stacks in the file besides those three default ones.)  There is no need to change the ports for stacks that you are not using.

It shouldn't be necessary to actually modify this file to control the ports used; this is because each port configuration uses system property substitution syntax to allow you to set the property from the command line using -D. For example, the UDP protocol mcast_port for the udp and jbm-control stacks is configured via the jboss.jgroups.udp.mcast_port system property:

mcast_port="${jboss.jgroups.udp.mcast_port:45688}"

while the MPING protocol mcast_port in the jbm-data stack uses:

mcast_port="${jboss.messaging.datachanneludpport:45710}"

The jboss.jgroups.udp.mcast_port property is a bit of a special case; it can be set from the command line using the shorthand -m command line option, e.g.:

./run.sh -c all -m 54321 -Djboss.messaging.datachanneludpport=12345

4.2.x and Earlier

In AS 4.2.x and earlier, JGroups channel configurations are embedded in the configurations for the services that use the channel.

For the HAPartition service, the JGroups channel configuration is found in deploy/cluster-service.xml.

Fort HttpSession replication, a JGroups channel configuration is found in deploy/tc5-cluster-service.xml. (In 4.0.4 and 4.0.5, this is instead located in deploy/tc5-cluster.sar/META-INF/jboss-service.xml. In the 4.2 series it is located in deploy/jboss-web-cluster.sar/META-INF/jboss-service.xml).

For EJB3 clustering, JGroups channel configurations are found in the deploy/ejb3-clustered-sfsbcache-service.xml and deploy/ejb3-entity-cache-service.xml files.

For clustered JBoss Messaging in EAP 4.3, JGroups channel configurations are found in the deploy/jboss-messaging.sar/clustered-hsqldb-persistence-service.xml file.

Configuration via System Property in 4.x

In the AS 4.2 and EAP 4.x releases, the mcast_port configurations for the services noted above can all be controlled via a system property by using -D when starting JBoss.The properties and the services they configure are as follows:

For releases prior to 4.2, you can also define the multicast ports via system properties but you need to modify the corresponding xml file to use system property substitution. For this example, we have chosen cluster-service.xml:

         ....
         <Config>
            <UDP mcast_addr="${jboss.partition.udpGroup:228.1.2.3}" 
               mcast_port="${jboss.hapartition.mcast_port:45566}"
               ip_ttl="8" ip_mcast="true"
               mcast_send_buf_size="800000" mcast_recv_buf_size="150000"
               ucast_send_buf_size="800000" ucast_recv_buf_size="150000"
               loopback="false"></UDP>
         ....

You will have to do a similar thing for any clustered service, such as HTTP session replication, EJB3 entity bean cache...etc. Use a different system property for each. Please, do not use the same port for all services due to the problems explained in the promiscuous traffic wiki.

Isolating Channels in JBoss AS 5

An excellent piece of news is that isolating clusters in AS 5.0.1 should no longer require changing multicast ports. Actually, simply starting each cluster with a different value passed to -u (as described above) should be sufficient. Even setting different partition names via -g isn't absolutely required (altough it's simple to do and still recommended.) As the above process of changing the multicast ports is by far the most cumbersome aspect of cluster isolation this is good news indeed.

In JGroups 2.6.5 and 2.6.6 fixes were made to the UDP and MPING protocols that resolve the issue that lead to the need to change multicast ports.

And an added bit of good news is a simple tweak to correct a mistake in the default AS 5.0.0.GA configuration will allow you to skip changing multicast port values in AS 5.0.0.GA as well.  The correction to AS 5.0.0.GA involves a simple edit to the $JBOSS_HOME/server/all/deploy/cluster/jgroups-channelfactory.sar/META-INF/jgroups-channelfactory-stacks.xml file to edit the jbm-data stack configuration:

<stack name="jbm-data"
       description="Stack optimized for the JBoss Messaging Data Channel">
  <config>
     <TCP singleton_name="jbm-data"
          start_port="${jboss.messaging.datachanneltcpport:7900}"
           .....
           oob_thread_pool.queue_max_size="100"
           oob_thread_pool.rejection_policy="run"/>
     <MPING timeout="5000"
            mcast_addr="${jboss.messaging.datachanneludpaddress,jboss.partition.udpGroup:228.6.6.6}"
            .....

Change the MPING.mcast_addr config to:

mcast_addr="${jboss.partition.udpGroup:228.6.6.6}"

Basically, the JGroups XML parser doesn't handle the more complex system property substitution syntax, so reduce it to the simpler syntax. Eliminate the first system property, not the second.

There are several other places in the jgroups-channelfactory-stacks.xml file that have the same problem, although this is the only one that causes problems with a default JBoss AS 5.0.0.GA. But, fixing making the same fix to the others is recommended as well.

This problem will be eliminated in the first release after AS 5.0.0.GA.

Other Notes

Using ip_ttl to isolate clusters

Another tip: if your clusters are on different subnets, configure the switches between your VLANs/subnets to drop IP multicast packets. If that is not an option, you can achieve much the same thing by reducing the UDP (and MPING) protocol's ip_ttl to a value where the multicast doesn't leave the subnet, e.g. 2.

Beginning with JBoss AS 4.0.5, if you set the jgroups.mcast.ip_ttl system property, that value will be picked up by the standard JGroups channel configurations.

This is sometimes a useful technique for developer workstations as well; i.e. set ip_ttl to 0 or 1 to prevent developers' workstations from clustering with each other. This doesn't always work though; it depends on how the specific network hardware deals with the TTL.  Another approach for isolating developer workstations is to start JBoss with -b localhost.

Why isn't it sufficient to change the group name?

If channels with different group names share the same multicast address and port, the lower level JGroups protocols in each channel will see, process and eventually discard messages intended for the other group.  This will at a minimum hurt performance and can lead to anomalous behavior.

Why isn't it sufficient to change the group name and multicast address? Why do I need to change the multicast port if I change the address?

First off, beginning with JGroups 2.6.6 it probably isn't necessary. But for those using earlier JGroups releases, or those interested in details...

There is an issue on several operating systems whereby packets addressed to a particular multicast port are delivered to all listeners on that port, regardless of the multicast address they are listening on.  If you know your OS doesn't have this issue, only changing the multicast address is sufficient.  See PromiscuousTraffic for more on this, including a test you can run to see if your OS exhibits this behavior.

In JGroups 2.6.5 and 2.6.6 there are fixes to the JGroups UDP and MPING protocols such that on Linux systems multicast sockets are created in such a way that the promiscuous traffic problem doesn't occur. An InetAddress indicating the multicast address is passed to the java.net.MulticastSocket constructor; that avoids the problem. However, constructing a MulticastSocket that way on other systems will lead to failures, so before doing this JGroups checks the os.name system property to confirm that Linux is the OS.

EJB clients

If you have EJB clients, see this forum post for info on restricting EJB clients to a particular partition.

The jboss.partition.udpGroup and jboss.partition.udpPort System Properties

There are two special system properties,  and , that JGroups will programatically check as part of the process of setting up a channel.  If the former is found, the value of the property will be used to set the channel's multicast address, overriding any value set in the configuration file.  If the latter is found the value of the property will be used to set the channel's multicast port, again overriding any value set in the configuration file.

Do not set both of these properties!!! If you do, all the channels on the server will use the same address/port combination and will thus see each others' traffic.  We recommend setting , which is automatically done if you start JBoss AS using the  switch.  If you do that, don't set .

More information

Here is more information related to cross talking.