Used of JGroups shared transport in AS 5

[brian.stansberry]

First in a series of posts re: use of the JGroups "shared transport" in AS 5 instead of the JGroups multiplexer. There's been some discussion of this on the public jbosscache-dev mailing list, and too many private discussions, but this needs to go in front of a wider audience.

JIRA for this is http://jira.jboss.com/jira/browse/JBAS-5329

Shared JGroups Resources

The purpose of both the multiplexer and the shared transport is to make it possible for different services that need to use JGroups to share some of the resources JGroups uses. In AS 4 and earlier, each clustered service needed to open its own JGroups channel; no resources were shared. This has become a bigger and bigger issue as the number of clustered services has grown.

The resources most desirable for sharing are:

1) Network sockets. Sharing these between services simplifies configuration and administration and saves memory (i.e. network buffers).
2) Threads. A JGroups channel creates a thread pool for passing incoming messages up to the clustered service. A pool that is shared across services can more effectively manage the number of threads in use.

The JGroups multiplexer was the original way JGroups sought to provide sharable resources. Basically, an entire underlying JChannel was shared, with an adapter (Multiplexer+MuxChannel) on top that multiplexed/demultiplexed messages and passed them to the appropriate service. See http://www.jgroups.org/javagroupsnew/docs/manual/html_single/index.html#d0e2203 for details.

The shared transport was added in JGroups 2.6.2. Here the shared object is not an entire JChannel, but rather the transport protocol (UDP or TCP) that makes up the bottommost element in its protocol stack. The network sockets and the thread pool are all managed by the transport protocol, so just sharing this protocol lets us achieve the most desirable sharing. See http://www.jgroups.org/javagroupsnew/docs/manual/html_single/index.html#d0e2325 for more.

With both approaches, a key goal is that an application using JGroups does not need to know if it is using shared resources or not. The application codes to the abstract org.jgroups.Channel class' API; whether that API is implemented using a JChannel+Multiplexer+MuxChannel or by a shared transport JChannel or just by a plain JChannel should be transparent to the application.

Advantages of Shared Transport over Multiplexer

The shared transport has a number of major advantages over the multiplexer:

1) No "impedence mismatch". A number of JChannel behaviors, particularly around view management, need to be massaged/hidden from the application if the Multiplexer+MuxChannel is used. E.g. the underlying JChannel sees the group membership as {A, B, C}. But, if Service1 connects a MuxChannel on nodes A and B but not C, Service1 should get a view of just {A, B}. The Multiplexer+MuxChannel needs to have some pretty complex (and fragile) logic to resolve the impedence mismatch between what the JChannel says is the view and what the application needs to see as the view.

2) Testability. A transport protocol has a much more limited set of behaviors than a full JChannel. It also has a known and limited set of configuration options, whereas a JChannel is infinitely configurable. As a result, rigorous testing of sharing behavior is much more manageable with a transport protocol.

3) Greater configuration independence. Different services that wish to share a transport protocol need only agree on the configuration of that transport protocol. The rest of their protocol stack can be completely different. It was clear that getting agreement on a multiplexed channel's protocol stack for all AS 5 services was either not going to happen or would force a kind of lowest common denominator config.

4) FLUSH. The FLUSH protocol stops all activity on a channel for a period. Mostly happens around view changes and state transfer. This is somewhat disruptive to the service using the channel. With the multiplexer, a FLUSH initiated by Service1 also effects Service2, Service3 and Service4, with no benefit to those other services. With a shared transport, a FLUSH on Service1's channel is transparent to the channels used by Service2, Service3 and Service4.

Advantages of Multiplexer over Shared Transport

The one advantage of the multiplexer is improved startup times. Until a few members have joined a JChannel's group, the discovery sequence takes a few seconds. With the multiplexer there is only one JChannel so the discovery sequence only occurs once per VM. With shared transport, it occurs once per service.

I want to do some stuff to help mask this, particularly lazy initializing the AS's clustered caches. This will speed the standard start time; the cost of the discovery is only incurred if the user deploys things that need the clustered caches.

Bottom Line

I think the advantages of the shared transport far outweigh the disadvantages, and intend to use it in AS 5. Bela strongly agrees. We both lack confidence in the multiplexer as a reliable solution.

[brian.stansberry]

Next, some implementation details.

In JGroups .2.6.2 the API for getting a shared transport channel vs. a multiplexed one isn't particularly clean.

Channel's in AS 5 will be gotten from the deploy/cluster/jgroups-channelfactory.sar's "JChannelFactory" bean. That implements the org.jgroups.ChannelFactory interface; clients should code to that interface.

The interface exposes the following methods for getting a shared resource channel:

Channel createMultiplexerChannel(String stack_name, String id) throws Exception;
Channel createMultiplexerChannel(String stack_name, String id,
 boolean register_for_state_transfer,
 String substate_id) throws Exception;

The name of the method and the javadoc somewhat imply these methods will return a MuxChannel+Multiplexer+JChannel. However, they do not explicitly state this. The return type is just "Channel", an abstract parent of both JChannel and MuxChannel. Clients should of course be coding to the Channel API and not assuming any further implementation details.

The std ChannelFactory impl, JChannelFactory exposes a method createChannel(String stack_name) which would be the more correct API to call if you just wanted a plain JChannel (shared transport or not). Unfortunately, in JG 2.6.2 this method was inadvertently left out of the ChannelFactory interface (fixed in 2.6.3). JG 2.6.2 will be used in AS 5, so AS 5 services cannot code to this method. They are stuck coding to one of the "createMultiplexerChannel" methods above.

In any case, JBC and probably JBM are too far along in their release cycle to code to a different method anyway.

So, how to get a shared transport channel rather than a multiplexed one?

The AS's channel factory bean is already an AS-specific impl of the ChannelFactory interface. I intend to alter the impl of the "createMultiplexerChannel" methods so they analyze the configuration of the specified protocol stack. If it supports a shared transport, I will return a shared transport channel. If not, I will return a MuxChannel+Multiplexer+JChannel. This is a bit hacky, given the name of the method, but I think the benefits are worth it.

[galder.zamarreno]

Brian, thanks very much for the detailed information on shared transport vs multiplexer.

[brian.stansberry]

Thanks; glad it's useful. :-)

FYI, my second post at http://www.jboss.com/index.html?module=bb&op=viewtopic&t=132161 gets into some of the configuration nuances when different services have shared transport available.

[brian.stansberry]

The AS's channel factory bean is already an AS-specific impl of the ChannelFactory interface. I intend to alter the impl of the "createMultiplexerChannel" methods so they analyze the configuration of the specified protocol stack. If it supports a shared transport, I will return a shared transport channel. If not, I will return a MuxChannel+Multiplexer+JChannel. This is a bit hacky, given the name of the method, but I think the benefits are worth it.

This has evolved further. The AS ChannelFactory will never return a MuxChannel. If a user invokes one of the createMultiplexerChannel methods and the TP config doesn't include a singleton_name, I'm going to create a singleton name for that channel and inject it into its config, log a WARN and return a shared transport channel.

Created name will be "unamed_" + stack_name.

http://jira.jboss.com/jira/browse/JBAS-5442

Re: the hackiness of the AS approach given the name of the method, I suppose it's a bit less hacky than I was thinking. A shared transport channel is still a "multiplexer channel". The multiplexing happens in the transport itself, when it chooses what up-protocol to pass an incoming message to. The fact that JG had another impl that involved an implementation detail class called "Multiplexer" doesn't mean the word multiplexer is forever reserved for that particular impl.

The AS fix is still hacky though, as the method signatures of the two createMultiplexerChannel methods include other params besides the stack name that are now totally ignored.

[galder.zamarreno]

The AS fix is still hacky though, as the method signatures of the two createMultiplexerChannel methods include other params besides the stack name that are now totally ignored.

Hmmm, I suppose you can always indicate this in the AS's ChannelFactory javadoc? As long as it's noted somewhere, doesn't look that bad. It's certainly safer than returning different type of objects depending on whether its shared transport or multiplexer.

[brian.stansberry]

Yes, I'd changed the AS factory's javadoc before to reflect some of the AS-specific behavior; need to update it again. It will actually be cleaner now, no more "if this is the case this happens, if not, that happens." Now just "this happens".