JON 2.3 Control Grouped Servers Via The CLI

Version 3

    1. Preface

    Oracle and IBM have a concept of Managed Servers. Besides other things it means that you have a central instance which takes care on delivering deployment artifacts to all registered managed application servers. This really helps administrators to make sure that all active application servers have the same version of the deployment artifacts. Really great would be the ability of being able to use a scripting interface to manage those managed servers and to also gather required measurements like memory consumption and such.

     

    This article is an attempt to show how you could do this with JBoss AS and JBoss Operations Network (JON). It is based on the experience you have by reading the following article:

    http://www.jboss.org/community/wiki/JON23ScriptedGroupDeploymentsUsingTheCLIAPI

     

     

    2. Requirements

    You need the following things to be able to execute the scripts:

    - JON 2.3 or better

    - JBoss EAP 4.x or 5.x

    - JDK 1.6 for the rhq cli

    - Linux to execute the wrapper script

     

     

    3. Creating The Wrapper Script

    To ease the use of scripts for JON you should create a simple wrapper script based on your favorite OS. I am using Fedora but it should easily be possible to use other OSses for this.

     

    The purpose of this script is to take the command line arguments and to call the JON CLI with one of the scripts as argument.

     

    #!/bin/bash
    #
    # groupcontrol
    # ------------
    # This is a simple wrapper script for all the java script scripts in this folder.
    # Start this script with some parameters to automate group handling from within the
    # command line.
    # 
    # With groupcontrol you can do the following:
    #   deploy: Deploys an application to all AS instances specified by group name
    #   create: Create a new group
    #   delete: Delete an existing group
    #   start : start all EAP instances specified by group name
    #   stop  : stop all EAP instances specified by group name
    #   add   : Add a new EAP instance to the specified group
    #   remove: Remove an existing EAP instance from the specified group
    #   status: Print the status of all resources of a group
    #
    # 
    
    ## Should not be run as root.
    if [ "$EUID" = "0" ]; then
       echo " Please use a normal user account and not the root account"
       exit 1
    fi
    
    ## Figure out script home
    MY_HOME=$(cd `dirname $0` && pwd)
    SCRIPT_HOME=$MY_HOME/scripts
    
    ## Source some defaults
    . $MY_HOME/groupcontrol.conf
    
    ## Check to see if we have a valid CLI home
    if [ ! -d ${JON_CLI_HOME} ]; then
         echo "JON_CLI_HOME not correctly set. Please do so in the file"
         echo $MY_HOME/groupcontrol.conf
         exit 1
    fi
    
    RHQ_OPTS="-s $JON_HOST -u $JON_USER -t $JON_PORT"
    # If JON_PWD is given then use it as argument. Else let the user enter the password
    if [ "x$JON_PWD" == "x" ]; then
         RHQ_OPTS="$RHQ_OPTS -P"
    else
         RHQ_OPTS="$RHQ_OPTS -p $JON_PWD"
    fi
    
    #echo "Calling groupcontrol with $RHQ_OPTS"
    
    usage() {
         echo "  Usage $0:"
         echo "  Use this tool to control most group related tasks with a simple script."
         echo "  ------------------------------------------------------------------------- "
         <....>
    }
    
    doDeploy() {
         $JON_CLI_HOME/bin/rhq-cli.sh $RHQ_OPTS -f $SCRIPT_HOME/deployToGroup.js $2 $3
    }
    
    
    case "$1" in
    'deploy')
            doDeploy $*
            ;;     
    *)
            usage $*
            ;;
    esac
    
    

     

    This script needs a groupcontrol.conf file which gets "sourced" into the execution context of this script. the .conf file looks like this:

     

    ##
    ## This file contains some defaults for the groupcontrol script
    ##
    JON_CLI_HOME=/home/wanja/Programs/JON/jon-2.3/rhq-remoting-cli-1.3.0.GA
    JON_HOST=localhost
    JON_PORT=7080
    
    # The user you want to connect with
    JON_USER=rhqadmin
    
    # if you omit the password here, you'll be prompted for it.
    JON_PWD=rhqadmin
    

     

     

    4. Creating The JON Scripts

    Within this chapter we will discuss how to use the API to implement necessary functionality.

     

    4.1 Creating A Compatible Group

    In order to be able to simulate this managed server functionality, we need to properly group all the resources. A group could be a set of servers which belong to the productive environment. Or a group could also be a set of servers of one domain of my productive environment.

     

    So how could you create a group using the API. This is really easy:

     

    // Now just create the group
    var rg = new ResourceGroup(groupName, resType);
    rg.setRecursive(true);
    rg.setDescription("Created via groupcontrol scripts on " + new java.util.Date().toString());
    
    ResourceGroupManager.createResourceGroup(rg);
    

     

    Of course you need to first check if the group already exists or not. We are creating a compatible group, which basically means that only resources of the same type are grouped together. This is controlled with resType which you get as follows:

     

    // First find resourceType specified by pluginName
    var resType = ResourceTypeManager.getResourceTypeByNameAndPlugin("JBossAS Server", pluginName);
    

     

    And pluginName is either "JBossAS" or "JBossAS5".

     

     

    4.2 Adding Resources To A Group

    Now you simply have a blank group which has nothing in it. Now how can we add resources to it? This is also quite easy:

     

    // now, search for EAP resources based on criteria
    criteria = new ResourceCriteria();
    criteria.addFilterName(searchPattern);
    criteria.addFilterResourceTypeName("JBossAS Server");
    
    var resources = ResourceManager.findResourcesByCriteria(criteria);
    

     

    This means we just want to have JBoss AS Server instances based on the given searchPattern. If you have those two instances in your JON repository:

     

    tolnedra.belgariad JBoss EAP 4.3.0.GA_CP03 node1 (192.168.100.50:1099)
    tolnedra.belgariad JBoss EAP 4.3.0.GA_CP03 node2 (192.168.100.51:1099)
    

     

    And you search for "node" you will find both instances. If you search for "node1" you'll find just node1.

     

    We just want to add a resource if just a single instance was found. Otherwise it could be too dangerous. So we need to test this:

     

    if( resources != null ) {
      if( resources.size() > 1 ) {
            println("Found more than one JBossAS Server instance. Try to specialize.");
         for( i =0; i < resources.size(); ++i) {
              var resource = resources.get(i);
              println("  found " + resource.name );
         }
      }
      else if( resources.size() == 1 ) {
         resource = resources.get(0);
         println("Found one JBossAS Server instance. Trying to add it.");
         println("  " + resource.name );
            ResourceGroupManager.addResourcesToGroup(group.id, [resource.id]);
         println("  Added to Group!");
      }
      else {
            println("Did not find any JBossAS Server instance matching your pattern. Try again.");
      }
    }
    

     

     

    4.3 Getting Inventory And Status Information Out

    Now we are able to create a group and to add resources to it. We now would like to have some kind of inventory information: Which resource is part of the group. And what is the status of it? Is it running?

     

    This is done by using the AvailabilityManager:

     

    // get server resource to start/stop it and to redeploy application
    var server = ProxyFactory.getResource(res.id);
    var avail  = AvailabilityManager.getCurrentAvailabilityForResource(server.id);
         
    println("  " + server.name );
    println("    - Availability: " + avail.availabilityType.getName());
    println("    - Started     : " + avail.startTime.toUTCString());
    println("");
    

     

     

    4.4 Deployment

    This was already explained here:

    http://www.jboss.org/community/wiki/JON23ScriptedGroupDeploymentsUsingTheCLIAPI

     

    The only difference will be that we don't want to stop the server first, deploy the new application and then start the server again. Instead we need to make sure if the server is already started. This is done with the following fragment:

     

         // we need check to see if the given server is up and running
         var avail = AvailabilityManager.getCurrentAvailabilityForResource(server.id);
         
         // infortunately, we can only proceed with deployment if the server is running. Why?
         if( avail.availabilityType.toString() == "DOWN" ) {
              println("  Server is DOWN. Please first start the server and run this script again!");
              println("");
              continue;
         }
    

     

     

    4.5 Start/Stop/Restart


    Starting / Stopping your servers is also quite easy. Just iterate through all you resources in a specific group and issue the corresponding operation (shutdown(), start() or restart()) on it:

     

    var resourcesArray = group.explicitResources.toArray();
    
    for( i in resourcesArray ) {
         var res = resourcesArray[i];
         var resType = res.resourceType.name;
         println("  Found resource " + res.name + " of type " + resType + " and ID " + res.id);
         
         if( resType != "JBossAS Server") {
              println("    ---> Resource not of required type. Exiting!");
              usage();
         }
         
         // get server resource to start/stop it and to redeploy application
         var server = ProxyFactory.getResource(res.id);
         println("    Starting " + server.name + "....");
         try {
              server.start();
         }
         catch( ex ) {
              println("   --> Caught " + ex );
         }
    }
    

     

     

    4.6 Scheduling An Operation

    Now that we're able to start/stop our resources, we will soon figure out that it will take some time until the operation was really executed and that it will then need some time until JON detects the change. This is due to the fact that an availabilty report needs to be sent from the agent to the server. We can now use the OperationManager to schedule a "executeAvailabilityScan" operation with all agents.

     

    First we need to get a list of all RHQ Agent server resources:

     

         println("Scanning all RHQ Agent instances");
         var rc = ResourceCriteria();
         var resType = ResourceTypeManager.getResourceTypeByNameAndPlugin("RHQ Agent", "RHQAgent");
         rc.addFilterResourceTypeName(resType.name);          
         resources = ResourceManager.findResourcesByCriteria(rc).toArray();
         
         var idx=0;
         for( i in resources ) {
              if( resources[i].resourceType.id == resType.id ) {
                   agents[idx] = resources[i];
                   idx = idx + 1;
              }
          }
    
    

     

    Now we are able to traverse the agents array and schedule the required operation:

     

         for( a in agents ) {
              var agent = agents[a];
    
              println("  executing availability scan on agent" );
              println("    -> " + agent.name + " / " + agent.id);
              var config = new Configuration();
              config.put(new PropertySimple("changesOnly", "true") );
    
              var ros = OperationManager.scheduleResourceOperation(
                   agent.id, 
                   "executeAvailabilityScan", 
                   0,   // delay
                   1,   // repeatInterval
                   0,   // repeat Count
                   10000000,  // timeOut
                   config,    // config
                   "test from cli" // description
              );
    
              println(ros);
              println("");
         }
    

     

    You might wonder how you could get the name of the operation. Well, this is quite easily. There is a method on the OperationManger which helps you to get a list of all operation definitions:

     

    var odc = new OperationDefinitionCriteria();
    odc.addFilterResourceTypeName("RHQ Agent");
    var oppList = OperationManager.findOperationDefinitionsByCriteria(odc);
    

     

    oppList now contains a set of OperationDefinition's, where you find the name of the operation, the description and if the opperation needs to get some parameters. Nearly all resource types do have operations defined that can be issued / scheduled in that way. This means it is also easy to invoke the agent's "executePromptCommand". But this operation needs a Configuration element (which command etc.)

     

     

    4.7 Gathering Metric Data Of Managed Servers

    Now we are able to create a group, to add resources to a group and to schedule operations. Another thing which is of high interest to an administrator is the health status of an application server. We would now like to create a script which is able to connect to the Measurement interface of JON. This is kind of complex so I am trying to explain more detailed how it works.

     

    JON is designed for complex monitoring as you can see if you have a look at the GUI. Nearly all existing resources do have a set of metrics which get collected by an active agent. The agent collects those metrics and sends a summary to the connected JON server which will then store the data into a database.

     

    JON is also designed to be as generic as possible. That means, you're free to write a plugin which collects own metrics.

     

    In order to be able to store those generic metrics, JON needs to have a set of metadata which describes a metric. This metadata could be "numeric", "performance", "string based" etc.

     

    To retrieve some life metric data, you first need to know for which metric you would like to see the value. This is done by using the MeasurementDefinitionManager:

     

              var mdc = MeasurementDefinitionCriteria();
              mdc.addFilterResourceTypeName("JBossAS Server"); 
              mdc.addFilterDisplayName(metricName);
              var mdefs = MeasurementDefinitionManager.findMeasurementDefinitionsByCriteria(mdc); 
    

     

    Again, you're building and filling a Criteria object. Here we would like to have metric definitions of resources of type "JBossAS Server". It should also be filtered by the metric display name, like "JVM Total Memory".

     

    Once you have the metric definition, you can use the MeasurementDataManager to find live data for the metric and the given resource:

     

              var metrics = MeasurementDataManager.findLiveData(resId, [metricDefId]); 
              if( metrics == null || metrics.size() == 0 ) {
                   return 0;
              }
              else {
                   var metric = metrics.toArray()[0];
                   if( metric != null && metric.value != null ) {
                        return metric.value.longValue();
                   }
                   else {
                        return 0;
                   }
              }
    

     

    As said, the value of the metric could be anything. In this case it is a numeric value. Another important "need-to-know" is that findLiveData() really tries to find live metric data. That means it only works if the Agent of the resource is up and running. To ensure this, you can check the Agent's status field. If it returns "-1", the agent is not connected. If it is "0", everything is fine and you can call the method.

     

     

    5. The Scripts

    The attached groupcontrol.zip file contains all the files necessary to start playing with your own set of scripts. Just unzip the file, change the groupcontrol.conf configuration for your environment and start the groupcontrol.sh script.

     

    For example:

     

    [wanja@tolnedra groupcontrol]$ ./groupcontrol.sh list
    Remote server version is: 1.3.0.GA(5192)
    Login successful
    Name                                     Version         HostName             Listen Addr          C-Time   A Last UP-Time      Total K Used KB % Free
    ------------------------------------------------------------------------------------------------------------------------------------------------------
    RHQ Server,  JBoss AS 4.2.3.GA default   4.2.3.GA        tolnedra.belgariad   0.0.0.0:2099         10/26/09 U 11/28/09 11:51:08 1021.56  185.46  81.85
    JBoss EAP 4.3.0.GA_CP03 node1            4.3.0.GA_CP03   tolnedra.belgariad   192.168.100.50:1099  10/27/09 U 11/28/09 14:26:55  127.88   35.73  72.06
    JBoss EAP 4.3.0.GA_CP03 node2            4.3.0.GA_CP03   tolnedra.belgariad   192.168.100.51:1099  11/11/09 D 11/27/09 11:26:11    0.00    0.00    NaN
    JBoss EAP 5.0.0.FCS web1                 5.0.0.FCS       jboss-01             0.0.0.0:1099         10/26/09 D 10/27/09 07:15:45    0.00    0.00   0.00
    JBoss EAP 5.0.0.FCS web1                 5.0.0.FCS       tolnedra.belgariad   0.0.0.0:1099         11/13/09 D 11/13/09 14:49:49    0.00    0.00    NaN
    
    

     

     

    6. Disclaimer

    Use at your own risk!