Infinispan Subsystem Management API

    Notes on the Infinispan subsystem management API

    This document aims to present an overview of some of the issues involved in the design/implementation of the Infinispan management API for AS7.

    0. Background


    The Infinispan subsystem of AS7 is made up of the following:

    • a runtime, namely a collection of Modular Service Container (MSC) service instances (EmbeddedCacheManagerService, CacheService and EmbeddedCacheManagerDefaultsService) providing Infinispan-related services to the AS as a whole
    • a persistent configuration of the subsystem in standalone.xml (or domain.xml) describing the configuration of the various services
    • services upon which the Infinispan subsystem is depeneent (e.g. JGroups services, transaction services, naming services, etc)
    Management of services

    The management API allows defining addressable resources (entities with attributes and operations) to represent these elements of the Infinispan subsystem and allow the user to interact with them. It also allows specifying what needs to happen in response to any user interaction with the subsystem elements. For example, if we change a configuration attribute of a service, a restart of that service may be required.


    The key questions to be decided are:

    • what do we want users to be able to do with the management API when interacting with the Infinispan subsystem? (add cache manager instances? add cache instances? add new configurations only? change an existing configuration?)
    • how do we want them to be able to do it? (e.g. add a cache configuration with a single operation execution, or over multiple operation executions)
    • how are we going to deal with the impact of their actions on the running system? (e.g. are restarts of some or all services required upon operation execution?)


    Another consideration is that the "final" functionality of the Infinispan management API will probably evolve over several releases as requirements are established.

    1. Proposed scope of initial release

    The following provides a summary of the proposed features to be included in the inital version of the managemnt API for Infinispan and provides some discussion of related issues.


    1.1 Initial Features


    A user should be able to do the following from the management API (either programmatically or via CLI):

    • cache manager operations
      • add a new configured cache manager to the system
      • remove an existing cache manager from the system
    • cache operations
      • add a new configured cache to the system, under the control of an existing cache manager
      • remove an existing cache from the system
    • MBean style access to operations and attributes


    1.2 Useability Issues


    The Infinispan subsystem is initially started from a persistent configuration in XML (i.e. see <subsystem xmlns="urn:jboss:domain:infinispan:1.0"/> in standalone.xml). This configuration describes the cache manager instances and their cache instances which will be installed as services in thye Infinispan subsystem. In addition, each cache manager and cache element specifies overrides to the clustering-specific default values for many (but not all) cache manager and cache configuration settings. Configuration of this initial state system state via XML is simple.


    Providing the same configuration ability from the command line is less easy. Some issues:


    1. handling cache manager/cache definitions - cache manager and cache definitions are large in size (and getting larger) and contain many nested elements. Inputting a cache definition from a command line in one operation execution presents problems; in particular, how to represent nested attributes.
    2. separating cache manager/cache definition from cache manager/cache start - do we define and start the cache manager/cache in one operation, or define and start separately. In the initial startup from XML, services are defined and started in one shot.  
    3. separate cache definitions - in the current XML, cache manager/cache service instances are tied to their definitions. Being able to reference a definition when starting a cache could be useful. (possibly in later release)


    1.3 Issues concerning effect of management operations on runtime


    Each management operation has the potential to affect both the persistent configuration of the subsystem on disk as well as the current runtime state of the services. In particular, making changes to the persistent configuration of a running service may necessitate the restart of the service itself and other services in the system. For example, changing the transport of a cache manager from stack="udp" to stack="tcp" wiil requie retsrating the cache manager, the caches it contains as well as the applications which depend on those caches. Changing other cache manager attributes may not be so intrusive.


    To handle these situations, the management API provides an attribute "restart-required" for operations which can affect the runtime state and will indicate to the user the appropriate restart actions. Possible values are:


    • no services - applying the operation to the runtime does not require the restart of any services
    • all-services - applying the operation to the runtime requires a restart of all services in the system before it will take effect (e.g. via management command /:reload)
    • jvm - applying the operation to the runtime requires a restart of the entire JVM before it will take effect (e.g. via management command /:shutdown followed by command line startup)
    • resource-services - applying the operation to the runtime will generally require a restart of all services in the system except if the operation specifies "allow-resource-service-restart" => true, in which case the operation may restart any services required "on-the-fly"


    So, with respect to management commands we provide, there are issues:

    1. CacheManager and Cache restart requirements- what are the general restart requirements for CacheManager and Cache instances when we change their configurations on-the-fly?
    2. determining restart requirements and policy for specific operations - for each operation, we need to be aware of its restart requirements and the intrusiveness on the system


    For example, it should be possible to add a new cache manager to the system without restarting existing cache services. On the other hand, modifying an existing cache manager may or may not require a restart of that cache manager and its caches. Removing a cache should probably not require restarting the associated cache manager.


    2. Addressable resources


    The current plan is to make the following addressable resources, where cache resources are organized by cache mode:


    <!-- Infinispan subsystem -->



    <!-- cache containers (with appropriate attributes, operations)-->



    <!-- caches by type (with appropriate attributes, operations) -->






    <!-- cache components (with appropriate attributes, operations) -->






    In the case of cache-containers and caches, the basic operations are add, remove and read-atribute, write-attribute. 


    3. Dealing with large configurations


    There are two approaches possible for adding / removing configured cache managers and caches which differ in the way that configuration options are specified:

    • single-step configuration, where the cache manager (resp. cache) is created, configured and started on demand in one step by passing all configuration settings as paranmeters to the add command
    • multi-step configuration, where a cache manager (resp. cache) is created, configured and started in multiple steps

    Single-step configuration

    In this approach, in a single step, the cache manager (resp. cache) is:

    • created
    • configured
    • installed as a service and started on demand


    Nested configuration elements are flattened (prefixed by their enclosing element name) so that they may be spefied on the command line.


    <!-- add a new cache container named X (with top-level and nested configuration attributes specified)  ->

    /subsystem=infinispan/cache-container=X:add(default-cache=default, listener-executor=W, transport.stack=udp, transport.lock-timeout=100)



    There is a problem with cache stores with this approach: cache stores can have nested lists of properties of the form <property name="X">Y</property> which are difficult (impossible) to represent with the flattened elements approach and the managemnt interface's requirements for parameter descritions. An alternative is to pass a complex structure representing the property list:


    <!-- add a new cache container named X (with top-level and nested configuration attributes specified)  ->

    /subsystem=infinispan/cache-container=X/local-cache=Y:add(file-store.relative-to="/tmp", file-store.path="blah",{"X"="x", "Y"="y", ...})



    This can then be parsed and setup as required.


    Multi-step configuration

    In this approach, in individual steps, the cache manager (resp. cache) is:

    • created but not installed nor started (the model is updated though)
    • configured but not installed nor started (the model is updated though)
    • installed as a service and started on demand


    Nested configuration elements are available as addressable resources under the submodel name configuration=<element name> and attributes can be set using write-attribute.


    <!-- create a new cache container but do not start it -->

    /subsystem=infinispan/cache-container=X:add(default-cache=Y, jndi-name=Z)


    <!-- for top-level cache container attributes (i.e. attributes of XML element cache-container), set them via write-attribute -->

    /subsystem=infinispan/cache-container=X:write-attribute(name=jndi-name, value=W)


    <!-- for nested nested cache-container attributes (i.e. attributes of child-elements of cache-container), access them as an addressable resource and seth them via write-attribute -->

    /subsystem=infinispan/cache-container=X/transport=TRANSPORT:write-attribute(name=stack, value=udp)

    /subsystem=infinispan/cache-container=X/transport=TRANSPORT:write-attribute(name=lock-timeout, value=100)


    <!-- once configuration is done, start the cache container-->




    The idea behind this approach is to update only the model for all operations but start. For the operation start, read the model to build the runtime services and install them on demand.


    4. Handling Service Interdependencies


    Cache containers and caches have various dependencies on one another; for example:


    • clustering - when a cache is clustered (invalidation-cache, replicated-cache, distributed-cache), the cache-container needs to initialise a JGroups transport
    • transactions - when a cache is transactional, uses syncrhonziation for transactions or transaction recovery, the cache-container needs dependencies on related services


    These dependencies need to be handled once we start to create cache-containers and caches independently of each other, as is the case when we introduce add commands for creating cache-managers and caches. For example, we may add a clustered cache once a cache-manager which manages only local caches has already been started.