Overview

JBoss AS 5 brings a number of major changes to the way the standard AS clustering services work with JGroups.  These changes can be summarized as follows:

• JGroups Channels needed by clustering services (e.g. a channel used by a distributed HttpSession cache) are no longer configured in detail as part of the consuming service's configuration, and are no longer directly instantiated by the consuming service. Instead, a new ChannelFactory service is used as a registry for named Channel configurations and as a factory for Channel instances. A service that needs a channel requests the channel from the ChannelFactory, passing in the name of the desired configuration.
• Different Channels used by different clustering services can share resources (e.g. network sockets and threads for handling incoming messages) using the JGroups shared transport.

Each of these areas are covered in more detail below.

ChannelFactory Service

AS 5 provides a new ChannelFactory service, deployed in the $JBOSS_HOME/server/all/deploy/cluster/jgroups-channelfactory.sar.  This service is an implementation of the JGroups ChannelFactory interface. It acts as a centralized registry of named JGroups channel protocol stack configurations and as a factory for Channel objects configured per a named configuration.  On startup the ChannelFactory service parses the deploy/cluster/jgroups-channelfactory.sar/META-INF/jgroups-channelfactory-stacks.xml file, which includes various standard JGroups configurations identified by name (e.g "udp" or "tcp"). Services needing a channel access the channel factory and ask for a channel with a particular named configuration.

The standard clustering services that ship with AS 5 (i.e. the HAPartition, the various services like session clustering that use JBoss Cache  under the covers, as well as JBoss Messaging) all get their JGroups channels by requesting them from the ChannelFactory.

Advantages of Using the Channel Factory

Configuration of a service that needs a channel no longer requires verbose specification of the JGroups protocol stack configuration. Instead, the service configuration simply needs to include the injection of the ChannelFactory and the name of the desired protocol stack configuration. This JBoss Cache configuration:

<bean name="ExampleCacheConfig" class="org.jboss.cache.config.Configuration">          

        <!-- Name of cluster. Needs to be the same for all members -->
        <property name="clusterName">ExampleCache</property>

        <!-- Use a UDP (multicast) based stack optimized for REPL_ASYNC -->
        <property name="clusterConfig">
          <config>
          <UDP
             singleton_name="udp-async"
             mcast_port="${jboss.jgroups.udp_async.mcast_port:45689}"
             mcast_addr="${jgroups.udp.mcast_addr:228.11.11.11}"
             tos="8"
             ucast_recv_buf_size="20000000"
             ucast_send_buf_size="640000"
             mcast_recv_buf_size="25000000"
             mcast_send_buf_size="640000"
             loopback="true"
             discard_incompatible_packets="true"
             enable_bundling="false"
             max_bundle_size="64000"
             max_bundle_timeout="30"
             use_incoming_packet_handler="true"
             ip_ttl="${jgroups.udp.ip_ttl:2}"
             thread_naming_pattern="cl"
             timer.num_threads="12"
                 
             use_concurrent_stack="true"

             thread_pool.enabled="true"
             thread_pool.min_threads="8"
             thread_pool.max_threads="200"
             thread_pool.keep_alive_time="5000"
             thread_pool.queue_enabled="true"
             thread_pool.queue_max_size="1000"
             thread_pool.rejection_policy="discard"
      
             oob_thread_pool.enabled="true"
             oob_thread_pool.min_threads="1"
             oob_thread_pool.max_threads="8"
             oob_thread_pool.keep_alive_time="5000"
             oob_thread_pool.queue_enabled="false"
             oob_thread_pool.queue_max_size="100"
             oob_thread_pool.rejection_policy="run"/>
          <PING timeout="2000" num_initial_members="3"/>
          <MERGE2 max_interval="100000" min_interval="20000"/>
          <FD_SOCK/>
          <FD timeout="6000" max_tries="5" shun="true"/>
          <VERIFY_SUSPECT timeout="1500"/>
          <pbcast.NAKACK use_mcast_xmit="false" gc_lag="0"
                   retransmit_timeout="300,600,1200,2400,4800"
                   discard_delivered_msgs="true"/>
          <UNICAST timeout="300,600,1200,2400,3600"/>
          <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
                   max_bytes="400000"/>
          <pbcast.GMS print_local_addr="true" join_timeout="3000"
                   shun="true"
                   view_bundling="true"
                   view_ack_collection_timeout="5000"/>
          <FC max_credits="2000000" min_threshold="0.10"/>
          <FRAG2 frag_size="60000"/>
          <!-- pbcast.STREAMING_STATE_TRANSFER use_reading_thread="true"/ -->
          <pbcast.STATE_TRANSFER/>
          <pbcast.FLUSH timeout="0"/>
          </config>
        </property>

        <!-- other unrelated configuration elements -->
</bean>

becomes this:

<bean name="ExampleCacheConfig" class="org.jboss.cache.config.Configuration">               

         <!-- Name of cluster. Needs to be the same for all members -->

         <property name="clusterName">ExampleCache</property>         

         <!-- Inject the ChannelFactory service -->
         <property name="runtimeConfig">
            <bean class="org.jboss.cache.config.RuntimeConfig">
               <property name="muxChannelFactory"><inject bean="JChannelFactory"/></property>
            </bean>
         </property>

         <!-- Use a UDP (multicast) based stack optimized for REPL_ASYNC.  -->
         <property name="multiplexerStack">udp-async</property>

         <!--other unrelated configuration elements -->
</bean>

(The above example shows using the JBoss Microcontainer schema to build a JBoss Cache Configuration.)

Using the ChannelFactory also allows re-use of the configuration.  Multiple services can that need a channel with, for example, the characteristics of the 'udp' configuration can request a 'udp'-configured channel from the ChannelFactory. No need for redundant and error prone re-declaration of the same configuration information in multiple locations.

It's important to note that if several services request a channel with the same configuration name from the ChannelFactory, they are not handed a reference to the same underlying Channel. Each receives its own Channel, but the channels will have an identical configuration. A logical question is how those channels avoid forming a group with each other if each, for example, is using the same multicast address and port. The answer is that when a consuming service connects its Channel, it passes a unique-to-that-service cluster_name argument to the Channel.connect(String cluster_name) method. The Channel uses that cluster_name as one of the factors that determine whether a particular message received over the network is intended for it.

Standard Protocol Stack Configurations

The standard protocol stack configurations that ship with AS 5.0.0.GA are described below. Note that not all of these are actually used; many are included as a convenience to users who may wish to alter the default server configuration. The configurations actually used in a stock AS 5.0.0.GA 'all' config are udp, jbm-control and jbm-data, with all clustering services other than JBoss Messaging using udp.

You can add a new stack configuration by adding a new <stack> element to the deploy/cluster/jgroups-channelfactory.sar/META-INF/jgroups-channelfactory-stacks.xml file. You can alter the behavior of an existing configuration by editing this file. Before doing this though, have a look at the other standard configurations the AS ships; perhaps one of those meets your needs.  Also, please note that before editing a configuration you should understand what services are using that configuration; make sure the change you are making is appropriate for all affected services.  If the change isn't appropriate for a particular service, perhaps its better to create a new configuration and change some services to use that new configuration.

Behavior Specific to the JBoss AS ChannelFactory Implementation

The ChannelFactory service in JBoss AS 5 includes a number of behavioral modifications that make it distinct from JGroups' standard implementation of the ChannelFactory interface, the JChannelFactory class.

• The AS ChannelFactory passes a config event to all newly created channels containing "additional_data" that will be associated with the JGroups IpAddress for the peer. Used to provide logical addresses to cluster peers that remain consistent across channel and server restarts. By default, this "additional_data" is a string concatentation of the value of the jboss.bind.address system property (i.e. the value passed via the -b startup switch) and the port on which the naming service is listening (by default 1099). For example, 192.168.1.111:1099.
  
• The AS ChannelFactory never returns instances of org.jgroups.mux.MuxChannel from the ChannelFactory.createMultiplexerChannel(...) methods.  Instead it always returns a channel with a shared transport protocol, for reasons discussed here.
• The AS ChannelFactory configures the channel's thread pools and thread factories to ensure that application thread context classloaders don't leak to the channel threads.

For those interested in the details, the source for the AS-specific ChannelFactory implementation is available in SVN.

Shared Transport

As the number of JGroups-based clustering services running in the AS has risen over the years, the need to share the resources (particularly sockets and threads) used by these channels has become a glaring problem.  A stock AS 5 'all' config will connect 4 JGroups channels during startup, and a total of 7 or 8 will be connected if distributable web apps, clustered EJB3 SFSBs and a clustered JPA/Hibernate second level cache are all used. So many channels can consume a lot of resources, and can be a real configuration nightmare if the network environment requires configuration to ensure cluster isolation.

Beginning with the JGroups 2.6.2 release, JGroups supports sharing of transport protocol instances between channels. A JGroups channel is composed of a stack of individual protocols, each of which is responsible for one aspect of the channel's behavior. A transport protocol is a protocol that is responsible for actually sending messages on the network and receiving them from the network. The resources that are most desirable for sharing (sockets and thread pools) are managed by the transport protocol, so sharing a transport protocol between channels efficiently accomplishes JGroups resource sharing.

To configure a transport protocol for sharing, simply add a singleton_name="someName" attribute to the protocol's configuration. All channels whose transport protocol config uses the same singleton_name value will share their transport.  All other protocols in the stack will not be shared. The following illustrates 4 services running in a VM, each with its own channel. Three of the services are sharing a transport; the fourth is using its own transport.

The protocol stack configurations used by the AS 5 ChannelFactory all have a singleton_name configured. In fact, if you add a stack to the ChannelFactory that doesn't include a singleton_name, before creating any channels for that stack, the ChannelFactory will synthetically create a singleton_name by concatenating the stack name to the string "unnamed_", e.g. unnamed_customStack.

See the JGroups manual for further documentation on the shared transport.

Use of Shared Transport instead of Multiplexer

The JGroups Multiplexer was an earlier attempt at providing shared JGroups resources.  The JGroups development team has deprecated the Multiplexer in favor of the shared transport for reasons discussed here.  As noted above, the JBoss AS 5 ChannelFactory implementation will not return channels based on the deprecated technology from its ChannelFactory.createMultiplexerChannel(...) methods.  Instead it always returns a channel with a shared transport protocol.