Notes on design of handling of JBoss Remoting based management calls

Version 2

    This document is intended as a guide to how the various classes and interfaces involved in dealing with JBoss Remoting based WildFly management invocations work. This reflects the classes in the WildFly Core master branch as of late June 2016 and is generally applicable to WildFly releases prior to that. It is possible things will change significantly as Elytron is integrated into WildFly Core management.

     

    Basic Background

     

    This article assumes knowledge of JBoss Remoting and of the general use cases involved with WildFly management.

     

    WildFly management often isn't based on simple request/response patterns. Invoking management operations often involves a number of message exchanges between processes. For example:

    • Client requests may have a stream associated with the DMR operation, with a separate series of messages exchanged to read the stream from the client to the server.
    • Responses may also have streams associated with them, with a separate series of messages exchanged to read the stream from the server to the client.
    • Internally to a managed domain, operations that cross processes involve a transactional protocol, where the controlling process sends the initial request message, the target process sends a prepared or failure response message, in the prepared case the controlling process sends a commit or rollback message, and then the target process sends the final result.
    • Streams also have to be handled between processes in the domain.
    • There is a currently-disabled feature to send asynchronous status messages back to management callers (originally conceived to handle things like report on progress of rolling the op out across a domain).
    • Operations can be cancelled by the client, or, in the case of shutdown, by domain processes. Cancellation is implemented via separate messages.
    • Operations that change the config of a slave HostController can trigger messages back to the DC to bring down additional configuration or content that is not present on the slave but is now required.

     

    The blocking behavior of all of this needs to be carefully watched, with a basic design goal that a separate thread pool is used for all blocking activity.

     

    ManagementChannelHandler

     

    The central class in management operation handling is the ManagementChannelHandler. Writes to and reads from the underlying remoting Channel are coordinated by an instance of this class. The various management clients and request handling services all have an instance of this class.

     

    ManagementChannelHandler.png

    ManagementChannelHandler implements 3 different interfaces. This division of interfaces largely serves to give callers an API that is specific to their needs without exposing irrelevant methods.

     

    • ManagementChannelShutdownHandle -- this is used by code that wishes to shut down the ManagementChannelHandler, allowing said code to signal that shutdown should begin and then wait for completion of in flight operations.
    • ManagementMessageHandler -- this is the API for handling incoming messages from the remoting Channel. ManagementChannelHandler implements this interface and passes a reference to itself to ManagementChannelReceiver, which is an implementation of remoting's Channel.Receiver interface. The ManagementChannelReceiver is then passed to the Channel's receiveMessage() method. ManagementChannelReceiver reads the management protocol header for a message, directly handles a couple of minor cases, and for the rest calls on the ManagementChannelHandler.
    • ManagementChannelAssociation -- this is the interface that callers who wish to initiate requests use. It is a poorly named and not very clean interface. It's not very clean because it exposes the Channel itself via a getChannel() method.

     

    A ManagementChannelHandler will be configured with one or more instances of ManagementRequestHandlerFactory. When it is asked to handle an incoming message it will query its associated factories looking for a ManagementRequestHandler that can handle the request; it then passes the request to that handler. Different ManagementChannelHandler instances will have different sets of ManagementRequestHandlerFactory, depending on what types of messages it needs to handle (e.g. an end user client has no need to handle messages related to slave host controller registration with the master DC.)


    ManagementChannelHandler needs access to a remoting Channel. To get this, it is provided at construction with an instance of the ManagementClientChannelStrategy interface. Via this interface the details of controlling the lifecycle of the channel are hidden from the ManagementChannelHandler. See the "ManagementClientChannelStrategy" section below for more on this.


    It's not noted on the image above, but ManagementChannelHandler also implements remoting's CloseHandler<Channel> interface. It is registered with the channel and responds to channel close notifications by cancelling any in progress operations.


    ActiveOperation interface


    As noted in the Basic Background section, management operations can involve the exchange of a number of messages before completion. Unique operations are assigned an integer id when created and this id is included as part of the management protocol header for each request. Through this mechanism related messages can be correlated. In each process an instance of the ActiveOperation interface is created. State and behavior associated with the overall operation can be obtained via this object. The ActiveOperation interface is commonly seen as a parameter to or return type from the methods in the various interfaces involved in management operation handling, particularly ManagementChannelAssociation and ManagementRequestHandler. The javadoc for the interface is fairly self-explanatory so I won't get into it in detail here.


    ManagementChannelAssociation Uses


    The following classes use ManagementChannelAssociation, i.e. initiate requests of some form:


    • AbstractModelControllerClient -- send typical management operation requests. Subclasses include:
      • RemotingModelControllerClient -- standard client created by ModelControllerClient.Factory, used by custom clients
      • CLIModelControllerClient -- ModelControllerClient variant used by the CLI, with a bit of special connection management logic that we haven't been willing to put in ModelControllerClient itself
      • ExistingChannelModelControllerClient -- used in some cases where a management Channel or Connection already exists and now another ModelControllerClient using it is wanted.
        • RemoteDomainConnection -- slave HCs invoke operations against the master (e.g. to request domain wide config data that the slave was ignoring but now needs). The primary use of the channel is to receive request from the master.
        • JConsoleCLIPlugin -- CLI GUI plugin invokes management ops using a Channel created from an existing remoting Connection (one used for remoting-jmx based JSR-166 invocations)
      • TestControllerClient -- special purpose client for a couple managed domain tests
      • DomainTestClient -- special purpose client for managed domain tests
    • OperationAttachmentsProxy -- server side requests attachments associated with the operation (i.e. streams) from the client.
    • OperationResponseProxy -- client side requests attachments associated with the operation response (i.e. streams) from the client.
    • OperationMessageHandlerProxy -- send out of band messages to the client. This feature is disabled.
    • TransactionalProtocolClientImpl -- HostController sending operations to either a slave HC or to a server. These can either be proxied requests (i.e. client connects to an HC but the address of the op is another HC or a server) or requests associated with rolling out an operation to the full domain.
    • TransactionalProtocolOperationHandler -- checks for an attachment that tells it whether it needs to read a Subject from request messages. This use has a bit of code smell.

     

    ManagementClientChannelStrategy


    As noted above, instantiators of ManagementChannelHandler provide an instance of the ManagementClientChannelStrategy which the handler uses to obtain a reference to the remoting Channel. There are a number of implementations of this interface:


    ManagementClientChannelStrategy.png

     

    These implementations fall into 3 basic categories:

     

    • The "establishing" variants. These strategies are all based on opening a remoting Connection and using it to create the Channel. They all descend from the FutureManagementChannel class, which uses a ProtocolConnectionManager to create the Connection. The variants:
      • Establishing. The standard "establishing" variant. Delegated to by a couple other impls that add some special behavior:
        • Anonymous class in RemotingModelControllerClient
        • Anonymous class in CLIModelControllerClient
      • HostControllerConnection -- managed domain server initiated connection back to the HostController process that spawned it
      • RemoteDomainConnection -- slave HC initiated connection to the master HC
      • TestControllerClient.ChannelStrategy -- special purpose client for a couple managed domain tests
    • DomainTestConnection.ChannelStrategy -- test utility
    • The "Existing" variant. Here the Channel is already created so the ManagementChannelHandler will use it. The services that create these are all invoked by an instance of ManagementChannelOpenListenerService, which implements the org.jboss.remoting3.OpenListener interface.
      • ModelControllerClientOperationHandlerFactoryService -- this service is for handling requests over end user initiated channels
      • MasterDomainControllerOperationHandlerService -- this service is for the master HC handling slave HostController registration requests and establishing the channel for passing management requests between the master and the slave. The ManagementChannelHandler created here is also used to create a TransactionalProtocolClientImpl (see ManagementChannelAssociation section above)
      • ServerToHostOperationHandlerFactoryService -- this service is for a HostController handling server registration requests and establishing the channel for passing management requests between the HC and the server. The ManagementChannelHandler created here is also used to create a TransactionalProtocolClientImpl (see ManagementChannelAssociation section above)
      • ExistingChannelModelControllerClient -- see discussion in ManagementChannelAssociation section above
      • 7 different uses in the unit tests in the org.jboss.as.controller module.