This document outlines an initial design for a cluster subsystem (/subsystem=cluster) for Wildfly.

Purpose

Clustered (or clustering) applications depend on the JGroups (/subsystem=jgroups) and Infinispan (/subsystem=infinispan) subsystems for group communication and distributed cache support, respectively.  Clustered applications include:

• web applications which are marked as <distributable/> and which support HTTP session replication and failover
• EJB applications which are marked as @Clustered and which support SFSB replication and failover
• JPA applications which use Hibernate to support a 2nd level cache

In such cases, each clustered application is made up of a number of clustered application peers, each the result of a deployment of the clustered application on a different node.

Monitoring clustered applications (checking current state, diagnosing  problems) in practice reduces to monitoring group communication and  distributed cache activity, across the cluster; in other words,  monitoring group communication and distributed cache activity at each  node participating in the clustered application.

The JGroups and Infinispan subsystems currently expose read-only runtime metrics for group communication channels and caches, but only for group communication and distributed cache peers running on a single node (i.e. the node that we have connected to with the management API). The main idea behind the cluster subsystem is to extend this view of run-time information to all nodes in a cluster, providing a cluster-wide view of runtime information for an application, accessible from a single management client.

In addition to being able to display runtime information on group communication and cache activity, the cluster subsystem is also a good location for diagnostic tools for clustered applications (e.g. testing multicast connectivity, probing JGroups nodes, etc)

The initial discussions for a such tools can be found in WFLY-1236 (https://issues.jboss.org/browse/WFLY-1236).

First cut of information to be presented

The JGroups and Infinispan subsystems present a wide range of information concerning group communication and distributed cache related activity; however, we consider the subset of information below as suitable for a first cut:

JGroups

• cluster view, cluster view history
• unicast / multicast / RPC statistics (transmissions, retransmissions)

Infinispan

• cache view, cache view history
• put / get / remove statistics
• transaction statistics
• hashing / data distribution statistics

Clustering 

• HTTP sessions and their distribution
• EJB SFSB sessions and their distribution
• JPA related information

The aim here is to present enough information to get a good view of what is going on in the cluster.

Organization of clustering services

A clustered application is made up of a collection of services, at the JGroups, Infinispan and Clustering levels and these services have well defined relationships. The presentation of addressable resources in the cluster subsystem will be based on these relationships, so we review them here.

JGroups services

• channel factory service (jboss.jgroups.stack.<stack-name>)
• channel service (jboss.jgroups.channel.<channel-name>)
• defaults service (jboss.jgroups.defaults)

Channel factory services are created based on JGroups subsystem configuration (read at startup, or defined via CLI) and they represent JGroups stack configurations. Channel services are defined when clustered applications are deployed and represent channel instances configured with a particular stack definition. Channel services are named according to the cache container which they support (e.g. for a cache container jboss.infinispan.web, its corresponding channel will be jboss.jgroups.channel.web). Channel services also have a channel id which is configurable (based on the cluster attribute of cache container transport if defined, or defaults to the cache container name). NOTE: users may define stack configurations, but creation of channels is restricted to internal use at present.

Infinispan services

• cache container service (jboss.infinispan.<container-name>)
• cache service (jboss.infinispan.<container-name>.<cache-name>)
• cache configuration service (jboss.infinispan.<container-name>.<cache-name>.config)

Cache container services are created to support a collection of clustered applications (i.e. the web container service supports clustered web apps; the ejb container service supports clustered EJB apps, the hibernate container service supports JPA apps which use a Hibernate 2nd level cache). Cache services are created to support a clustered application's replication requirement and the actual service names are application deployment dependent.  NOTE: cache containers and caches may be defined and accessed outside of the scope of <distributable/> web apps, @Clustered EJB apps and JPA applications, but this is generally unsupported usage.

Clustering services (e.g. web application services - EJB SFSB and JPA cases similar)

• web application service (jboss.web.deployment.<vhost>/<context>)
• session manager factory service (jboss.web.deployment.<vhost>/<context>.session)

The war application deployer installs a SessionManagerFactory service for the web app, which in turn creates the following services:

• session cache configuration service (jboss.infinispan.web.<vhost>/<context>.config)
• cache container service (jboss.infinispan.web)
• cache service (jboss.infinispan.web.<vhost>/<context>)

All of these services are tied to the web application as dependencies.

What is a cluster?

So, what do we mean by a cluster? For now, we take a cluster to be:

• a  set of cache containers on different nodes with the same service name  (e.g. jboss.infinispan.web) ; or alternatively,  because each cache  container is associated with a channel of the same name, we can think of  a cluster as set of channels with the same service name  (jboss.jgroups.channel.web)
• for  each cluster, there will be zero or more caches defined within the  cluster, corresponding to a deployed application (e.g.  org.jboss.web.<vhost>/<context>)

Presentation of addressable resources

Given the above relationships between services for clustered applications, here is one organization of the addressable resources in the cluster subsystem which could be used to present the run-time information in the cluster subsystem in a coherent way:

# collect together JGroups and Infinispan container related metrics per cluster name
/subsystem=cluster/name=X

#  define attributes for cluster wide run-time information (e.g. cluster view, view history, unicat / multicast / RPC stats)

# collect together Infinispan cache related metrics per deployment name
/subsystem=cluster/name=X/deployment=Y

# define attributes for cluster-wide runtime information (e.g. cache view, cache view history, put / get / remove stats, etc)

Reading each attribute returns a list containing the corresponding attributes for each node in the cluster.

Implementation

There are some implementation issues to resolve.

Collecting information from all nodes in the cluster

The cluster subsystem depends on having a mechanism to retrieve run-time metrics from all nodes in a cluster and return them to the node on which the cluster subsystem is being controlled from. This is easily achieved using the existing RPC mechanism provided by org.jboss.as.clustering.impl.CoreGroupCommunicationService.

The RPC handler on the remote end can obtain the required information by either:

• connecting to the management intrerface on the remote node via a managemnt client and performing a management operation
• looking up services on the remote node and extracting the information from the services directly

NOTE: for the moment, the addressable resources in the cluster subsystem are fixed, so any use of a management client to lookup information on a remote node would be internal use only (i.e. not available as part of the interface).

Security

Access to the management interface on any node need to be access controlled. Any RPC mechanism used to retrieve data using the management interface needs to be subject to this control.

Progress

See WFLY-1430 (https://issues.jboss.org/browse/WFLY-1430)

Comments / Issues Missed?

Feedback is appreciated.