Version 15
Created by brian.stansberry on Sep 26, 2012 8:49 PM. Last modified by aloubyansky on Feb 23, 2016 3:47 PM.

    This article outlines design notes for the single instance patching feature developed for AS 7.2 and WildFly.

     

    The patching feature allows to apply a patch to a single installation of the AS (i.e. a single unzip of the AS distribution.) That single installation may be used for a standalone server or for a managed domain Host Controller and servers. However, the feature described in this article will not support coordinated patching by the patching tool of multiple installation in a managed domain. The user can of course independently patch multiple installations in a domain, coordinating the application of those patches themselves.

     

    The basic patching of a running server process consists of:

     

    1. The user obtains a patch file.
    2. The user uses the CLI tool to connect to the standalone server or Host Controller that is running on the installation that is to be patched.
      1. The server or Host Controller may be running in admin-only mode or running normally. However, some patches may not be able to be applied if the target process isn't running in admin-only mode
    3. The user invokes a command on the CLI providing the location of the patch file. The CLI invokes an operation(s) on the target process telling it to stage the patch.
    4. The target process stages the patch by applying the patch contents to the filesystem.
      1. The changes made to the filesystem during staging should not affect the running operation of the server. If this is not possible, the server should reject the patch with an error message indicating that it needs to be placed in admin-only mode before applying the patch.
    5. The user uses the CLI tool to restart the target process. Upon restart the staged patch is visible in the runtime.

     

    It is also possible to patch the installation of a not running server. In that case the the process will consists of:

     

    1. The user obtains a patch file.
    2. The user launches the CLI tool from the installation that is to be patched in the disconnected mode.
    3. The user invokes the same CLI command as in the previous scenario.
    4. The CLI command handler apply the instructions from the patch file to the installation.
    5. The user can start the patched server as usually.

     

    Patches can also be rolled back. As in case of patch application, a roll back can be performed on a running server as well as on the installation of a non-running server. For that:

     

    1. The CLI tool can be used to instruct the target process to revert a patch.
    2. Patch reversion results in changes to the filesystem that are staged in the same manner as what is done with a patch installation.
      1. The changes made to the filesystem during patch rollback staging should not affect the running operation of the server. If this is not possible, the server should reject the patch rollback with an error message indicating that it needs to be placed in admin-only mode before rolling back the patch.
    3. The CLI tool is used to restart the target process. Upon restart the rolled back patch is no longer visible in the runtime.

     

    Patch File

     

    A patch file is a zip archive that contains a patch.xml file that is used to describe the patch along with the updated modules (organized by layers and add-ons) and miscellaneous files that comprise the patch. It's structure is as follows:

     

    + patch.xml

    + layer

    ++ patch modules in the same structure as they appear in the modules dir in the normal AS dist

    + add-on

    ++ patch modules in the same structure as they appear in the modules dir in the normal AS dist

    + misc

    ++ misc files that need to be updated, organized in a directory structure that matches the directory structure of the AS dist

     

    The patch.xml File

     

    The patch.xml file describes the patch. It includes basic information about the patch along with metadata about the modules and misc files in the patch.

     

    Basic patch metadata
    • The name of the patch
    • The type of the patch (one-off vs cumulative)
    • The version to which the patch applies
    • The version after the patch has been applied (for cumulative patches)
    • A text description of the patch

     

    Module metadata
    • The name of the module
    • The slot of the module
    • The hash of the module.xml file of the previous version of the module. Not present if the relevant module did not exist in the version being patched.

     

    Removed module metadata

    Identifies modules that were available in the version being patched but which should no longer be accessible once the patch is applied

    • The name of the removed module
    • The slot of the removed module

     

    Misc file metadata
    • The path of the misc file, relative to the root of the AS distribution
    • The hash of the version of the file that was in the version being patched. Not present if the relevant file did not exist in the version being patched, or if the relevant file is a directory.
    • A boolean indicator as to whether the relevant file is a directory.
    • A boolean indicator as to whether the file is in active use by a non-admin-mode Host Controller or server

     

    Removed misc file metadata

    Identifies misc files that were available in the version being patched but which should no longer be accessible once the patch is applied

    • The path of the misc file, relative to the root of the AS distribution
    • The hash of the version of the file that was in the version being patched. Not present if the relevant file is a directory.
    • A boolean indicator as to whether the relevant file is a directory.
    • A boolean indicator as to whether the file is in active use by a non-admin-mode Host Controller or server

     

     

    Patch "Staging"

     

    The target process when it applies a patch will create an .overlays directory (unless it already exists) and a patch directory under it directly underneath each layer and add-on of the AS distribution targeted by the patch and patches directory underneath the .installation directory of the target AS distribution. The .overlays directory is used for the module content of the patch. The patches directory is used to store the miscellaneous content files and patch metadata.

     

    Before patch staging begins, a check is made as to whether the target process is in admin-only mode. If it is not, and an unresolvable conflict is detected between a module or miscellaneous file contained in the patch and a user-modification of the corresponding item in the installation being patched, the patch staging process will be aborted with an error message.  See "Patch Conflict Detection" below.

    Modules

     

    Patch modules will be staged by copying them to a patch-specific subdirectory of the layer's .overlays directory. Information about the location of this directory will be stored in a location visible to the module loader used by JBoss Modules at boot. This patch-specific subdirectory is not visible to the module loader currently in use by the target process.

     

    For modules that the patch "removes", the patch-specific subdirectory of the layer's .overlays directory will include that module, but with a special module.xml file and no other contents. The module.xml file will include the tag <missing/>. The JBoss Modules release used with this will interpret that <missing/> tag and if seen will throw a ModuleNotFoundException (which is what it would throw if it were unable to locate a module.  The nice thing about this approach is if a later patch adds back the removed module, that patch will take precedence in the module path, and the "missing" module.xml will be irrelevant.

    Misc files

     

    Misc files are staged by copying them directly to their normal location on the filesystem. A copy of the existing file at that location will be stored in the patch-specific subdirectory of the patches directory.

     

    Configuration files

     

    Patches will not modify the configuration files of the target process. Configuration patching will not be supported in this version of the feature (and may never be supported.)  Configuration files belong to the user. However, a copy of all existing configuration files will be made in a patch-specific subdirectory of the patches directory. During patch rollback, these configuration files will be restored. This backup is necessary because once the patch is applied to the runtime, any management operation that results in persistence of the configuration will use the xml schemas associated with the patched version of the core AS and any subsystems. If the patch is reverted, those schemas may not be intelligible to the version of the code to which the runtime has been reverted.

     

     

    Patch Application

     

    A patch is "applied" by stopping the target process and then restarting it in a normal manner; e.g. from the command line or by using the :shutdown(restart=true) command from the CLI. The patch contents that have been staged are then used by the restarted process.

    Modules

     

    The module loader works using a notion of a "module path", which is an ordered list of directories in which modules are located. When the module loader needs to find a module, it searches the directories in that path in order, and once it finds the desired module the search ends. Thus, directories earlier in the path listing take precedence over those found later. This is all very similar to how an OS uses the $PATH environment variable.

     

    The module loader at start-up examines the patching metadata that was persisted in the "staging" phase and from that determines which of the patch-specific subdirectories of the .overlays directories need to be prepended to the module path. In addition, for modules that have been removed in patches, the module loader knows to not load those modules, even though the modules will still remain on the filesystem.

     

    Misc Files

    Application of miscellaneous files to the runtime is straightforward. The staging step places the patch's miscellaneous files in the normal location that the runtime services will expect, so when the target process is restarted, the runtime services see them.

     

    Patch rollback

     

    Rolling back a patch works in a similar to the applying process. The target process has to be in --admin-only mode or completely shutdown. The backup copies of the miscellaneous files stored during the "Patch Staging" process will be restored to their normal locations. Any misc files that were added as part of the "Patch Staging" process (i.e. those that were new in the patch) will be removed. Modules overlay directories will get removed when the staging process is completed.

     

    A patch rollback is "applied" by stopping the target process and then restarting it in a normal manner; e.g. from the command line or by using the :shutdown(restart=true) command from the CLI. The modules and miscellaneous files that have been staged are then used by the restarted process.


    Configuration

     

    If the user instructed to restore the backed up configuration, the backup copies of the configuration files stored during the "Patch Staging" process will be restored. Any configuration changes made while the patch was in place will be lost.

    Modules

     

    See the discussion in "Patch Application" above of how the module path works. When a patch is rolled back, the metadata the custom module loader uses to construct the module path will no longer indicate the patch's modules should be on the path, so they will not be available for loading.

    Misc Files

     

    As was the case in the discussion in "Patch Application" above, the patch rollback staging step places the correct miscellaneous files in the normal location that the runtime services will expect, so when the target process is restarted, the runtime services see them.

     

    Patch Conflict Detection

     

    At the beginning of the patch staging process, a check will be made for conflicts between the patch and any user modifications to the same item. The user when executing the patch command can provide information to guide the tool in resolving conflicts. Any conflicts that cannot be resolved will result in the patch staging being aborted with no changes to the filesystem. Conflict types are:

     

    • Modules. If the patch is updating or removing an existing module, the patch.xml file will include the hash of the existing module's contents. The patching process will hash the current module's contents to check for modifications. If any affected module is found to have been modified, the patch command will have to have been provided with a parameter indicating that "overriding" modified modules is ok. No option will be given to optionally override some modules and leave others unmodified. If the user wishes to retain some modification they made to a module, they will need to make an equivalent change to the patch module. The sole purpose of this "module conflict detection" is to provide information to users.
    • Misc files: If the patch is updating an existing misc file, the patch.xml file will include the hash of the existing file. If the patch is removing an existing directory, the patch.xml file will include the hash of the existing directory's contents. The patching process will hash the current file or directory to check for modifications. If any affected file/directory is found to have been modified, the patch command will have to have been provided with a permission indicating that "overriding" modified modules is ok. If no permission is available for a particular file or directory, the conflict will be treated as unresolved. These permissions can come in four forms:
      • A global permission to update all conflicting misc files, provided as a param to the CLI tool's patching command
      • A global permission to not update any conflicting misc files, provided as a param to the CLI tool's patching command
      • A file containing
        • a list of paths (relative to $JBOSS_HOME) for which update permission is granted.
        • a list of paths (relative to $JBOSS_HOME) for which update permission is denied, meaning the existing file should be retained.

     

    Patch Generation Tool

     

    • Accepts basic patch metadata along with the location of a distribution of the new version along with a distribution of the old version
      • Also includes details on any misc files that are expected to be in active use
    • Generates a patch.xml file
    • Can generate a full patch file

     

    (Layered) Directory structure

     ${JBOSS_HOME}
 |-- bin
 |-- docs
 |-- modules
 |   |-- layers.conf (xyz,vuw)
 |   |-- system (system modules contains only modules, no patches metadata)
 |   |   |-- layers
 |   |   |   |-- xyz
 |   |   |   |    `-- .overlays
 |   |   |   |       |-- patch-xyz-1
 |   |   |   |       `-- patch-xyz-2
 |   |   |   |-- vuw
 |   |   |   |    `-- .overlays
 |   |   |   |        `-- patch-vuw-1
 |   |   |   ` -- base
 |   |   |        |-- .overlays
 |   |   |        |   |-- patch-base-1
 |   |   |        |   `-- patch-base-2
 |   |   |        |-- org/jboss/as/...
 |   |   |        `-- org/jboss/as/server/main/module.xml
 |   |   `-- add-ons
 |   |       `-- def
 |   |           `-- .overlays
 |   |               |-- patch-def-1
 |   |               `-- patch-def-2
 |   |
 |   `-- my/own/module/root/repo
 |
 |-- .installation (metadata directory for the installation)
 |   |-- identity.conf (patched state for the installed identity)
 |   `-- patches  (history of the patches applied to the identity)
 |       `-- patch-identity-1
 |           |-- patch.xml
 |           |-- rollback.xml
 |           |-- timestamp
 |           |-- configuration   (configuration backup)
 |           `-- misc            (misc backup)
 |       |-- layers (metadata for patched layers)
 |       |   |-- base
 |       |   |   `-- layer.conf (patched state for the layer)
 |       |   |-- xyz
 |       |   |   `-- layer.conf
 |       |   |-- vuw
 |       |   |   `-- layer.conf
 |       `-- add-ons (metadata for patched add-ons)
 |           `-- def
 |               `-- layer.conf
 `-- jboss-modules.jar

    For more information on layering: https://community.jboss.org/wiki/LayeredDistributionsAndModulePathOrganization

     

    Patching Deployments

    There has been some confusion related to how the patching tool can be used to patch a deployment that comes as part of some set of functionality that gets added onto the base AS (e.g. by a layered product that builds on top of EAP 6.) This section will attempt to address some of that confusion.
    See the Extending AS7 wiki page for basic information on different approaches to adding more functionality to the AS.
    First, this patching tool is irrelevant when it comes to end user application deployments. Once the patching feature is available, end users should install an updated version of their application using the same mechanisms they've used for updating their application deployments in previous AS releases (CLI, deployment scanner, admin-console, JON.) The patching tool is not intended to replace or supplement these mechanisms.
    Second, the patching tool will be capable of patching deployment archives provided by some add-on that uses the underlying AS. However, there is an important consideration that needs to be understood. It must be possible to stage a patch by copying the patch contents onto the filesystem without having that updated content become immediately visible to a running server that is reading that filesystem. The staged content should sit there having no affect on the running server until the user chooses to restart the server. If a deployment archive is installed into the server by placing it in the deployments/ folder (or any other folder periodically scanned by a deployment scanner), this requirement cannot be reliably met. The scanner will notice the new deployment as soon as it is staged and will deploy it. Similarly, if the deployment archive is exploded, even if it is not installed by a deployment scanner, if the server is using the original exploded directory as the source of the live deployment, the "staging requirement" cannot be met. When the patch tool updates a file in the exploded archive, it will immediately be visible to the runtime services (e.g. to the running deployment's classloader.)
    How can AS add-ons install services while still supporting stageable patches?  There are 4 approaches, two based on using modules, two that don't use modules
    Using modules:
    1. Create an AS Extension implementation. Package it in a module. Have the Extension implement install your runtime services. This is the preferred approach wherever practical.
    2. Create an AS Extension implementation. Package it in a module. Include the deployment archive inside the module. Have the Extension implementation programatically deploy it when the subsystem starts. See "A Mixed Approach" on the Extending AS7 wiki page for an example extension that does this.

     

    Both of these approaches have no problems with patch staging, because when a module is patched, the patch is not visible to the running server's module loader until the server is restarted.

     

    If the deployment archive is not packaged inside a module, then it is a "Misc File" in terms of the previous discussion on this document. Therefore the rules for staging an update to a misc file described above apply. Given this, there are two mechanisms for making such a deployment "patchable":

     

    1. Place the deployment in some location on the filesystem, but not in a directory scanned by a deployment-scanner.  Create an AS Extension implementation. Package it in a module. Have the Extension implementation, when the subsystem is added, copy the deployment content into an internal location (i.e. the data/ dir) and then invoke management operations to deploy the content from that location. The portal team was investigating installing their ear (which must be exploded and end user-modifiable) in this manner. The original deployment can be patched because it's the copy of the deployment that is being used by the running server.
    2. Accept that the deployment archive cannot be truly staged. Include in the patch.xml file metadata indicating that the file is in active use by a non-admin-mode Host Controller or server. When the patching tool goes to stage the patch, if it sees this metadata, it will check if the process is in admin-only mode. If it is not, it will refuse to install the patch until the process is placed in admin-only mode.

     

    This last mechanism of not truly supporting patch staging is only intended to serve as a fallback scenario for cases where no alternative is available for patching a particular file. It was not intended to serve as a mechanism for patching deployments installed by layered products, for which the other three alternatives are available.

    8672 Views Categories: Tags: