Version 7

    The ActiveAlarmTable Service


    Introduced in JBoss v4.0.3, the ActiveAlarmTable service is a table

    that can be used to hold the current (active) system alarms. Assuming that

    the occurence of an alarm is indicated by a JMX notification, the ActiveAlarmTable

    is essentially a NotificationListener based on ListenerServiceMBeanSupport

    that can subscribe for any JMX notification in the system, in order to feed

    the alarm table.


    The main goal of providing this service is to make it easier to discover

    the collective fault status of the system. If the alarm table is updated

    correctly it should help answering the question "what is wrong with my system, now".


    Alarm Types & Severity


    Alarm is an indication of an error condition in the system. The distinguishing

    factor of alarms from other system events, is that alarms have a "severity"

    (or it can be that their severity is "unknown").


    According to most management standards the available severities are:


    • NORMAL


    • MINOR

    • MAJOR




    Another characteristic of alarms is that they can be stateful or stateless.


    A stateless alarm indicates an instantanious error condition, e.g. a security

    violation, or a deployment failure. The mbean that produces the alarm doesn't

    keep track of the error condition to report its clearance (or this may be

    simply impossible). The state of a stateless alarm is always NONE, while the

    severity can be anything other than NORMAL.


    A stateful alarm represents an error condition in the system, e.g. "memory low",

    that can clear itself out. The mbean that produces the alarm can, after a while,

    decide that the free memory is back to normal levels, so the alarm can be

    cleared. To do that, the mbean has to keep track of the alarm status. The

    state changes for stateful alarms are indicated with emitted alarm notifications

    that carry the alarm states CREATED, CHANGED, or CLEARED. A CLEARED alarm

    must carry a severity of type NORMAL.


    A common source of confusion is that we often associate an alarm with the Notification

    carrying the alarm. A notification is simply an "event in transit". In the case

    of stateless alarms, the notification carries the full state of the alarm,

    because every instantiation of the alarm represents a separate event. With stateful

    alarms, the notification merely carries the state change of the alarm, i.e.

    you can have 3 notifications referring to the same logical alarm condition.

    So for example, an alarm can be first CREATED with a severity WARNING,

    then CHANGED with an escallated severity of CRITICAL, and finally CLEARED

    with a NORMAL severity.


    How the ActiveAlarmTable interprets received notifications.


    Every notification received by the ActiveAlarmTable is considered to be an alarm.

    Depending on the type of notification we have three cases:


    • A subclass of org.jboss.monitor.alarm.AlarmNotification that carries

    an alarmState of CREATED, CHANGED or CLEARED. This is considered to be

    a stateful alarm. The notification source is combined with the notification

    type to form an Id. Multiple notifications with the same Id are stored

    as a single entry in the table, since they represent a single error condition.

    • A subclass of org.jboss.monitor.alarm.AlarmNotification that carries

    an alarmState of NONE. This is treated as a stateless alarm and is always stored

    in the table as a new entry.

    • Any other Notification is treated as a stateless alarm. Since its severity

    is not known (which is the case with an AlarmNotification) we assume a

    severity of UNKNOWN.


    So it is easier for stateless alarm to fill the table, since every notification

    is considered a different alarm. For example, a security manager mbean may

    throw multiple notifications of type "" and each is

    a different fault.


    On the other hand, stateful notifications of the same type

    and the same source create a single entry in the table. For example,

    multiple notifications of type "jboss.alarm.memory.low" from the same instance

    jboss.monitor:service=MemoryMonitor refer to the same fault. If an mbean

    wants to produce multiple stateful alarms it needs to use different notification

    types, e.g. to represent the failed or otherwise normal status of a pool of

    3 connection we could use the types:








    Acknowledgement of Alarms


    The alarms are kept in the table for somebody (usually an administrator) to

    see and react. An alarm can be acknowledged to indicate that it's been handled.

    In that case:


    • A stateles alarm is removed from the table

    • A stateful alarm, if it has cleared itself out, it is removed from the table,

    otherwsie, it is just marked as acknowledged, in its current state. Every

    change in its state resets the acknowledge flag.


    Configuring the ActiveAlarmTable


    The following attributes can be configured:


    • MaxTableSize, can be used to set an upper limit to the number of

    entries kept in the table in order to avoid overflow. When the table

    is full, new entries are ignored. Alarm tables shouldn't have to keep more

    than a few hundred entries. The default is 1000.

    • LogLevel, the log level to use for received notification, can be

    set to NONE, default is DEBUG

    • SubscriptionList, can be used to subscribe for any notification

    in the system.



    Retrieving the active alarms


    The active alarm table can be retrieved from the service in the form

    of a AlarmTableNotification array. AlarmTableNotification is an

    extension of AlarmNotification, that adds acknowledge info and uses

    the payload (UserData) field, to store the original alarm notification.


    The ActiveAlarmTable itself produces jboss.alarm.table.update notifications

    whenever updated.


    Feeding the table with alarms


    The table can be fed with any notification in the system but there is already

    a number of services that can actually produce AlarmNotification notifications:


    • ScriptingListener, a scripting facility that allows you to map

    ordinary notifications to alarm notifications; however this is a generic

    service and it can be used for other purposes.

    • JMXNotificationAppender, a simple log4j appender that transforms logging

    events to JMX notifications, so they can feed the alarm table.

    • MemoryMonitor, a simple free memory monitor that produces stateful

    alarm notifications.


    Producing alarms from your own MBeans


    If you want your mbeans to produce natively stateful/stateless AlarmNotification

    notifications, that can feed the alarm table directly, you may want to

    consider using the AlarmManager helper, that greatly simplifies this task

    (especially when it comes to stateful notifications).




    The ActiveAlarmTable is disabled by default. To enable


    1. edit conf/jboss-service.xml, uncomment the entry:

       <mbean code="" 
          <attribute name="SubscriptionList">
                <mbean name="jboss.monitor:*">
                   <notification type="jboss.alarm"></notification>
                   <notification type="JBOSS_MONITOR_NOTIFICATION"></notification>               
                <mbean name="jboss.system:service=Logging,type=JMXNotificationAppender"></mbean>


    2. edit conf/log4j.xml and uncomment the JMXNotificationAppender

       <!--  Emit events as JMX notifications -->
       <appender name="JMX" class="">
          <errorHandler class="org.jboss.logging.util.OnlyOnceErrorHandler"></errorHandler>
          <param name="Threshold" value="WARN"/>
          <param name="ObjectName" value="jboss.system:service=Logging,type=JMXNotificationAppender"/>
          <layout class="org.apache.log4j.PatternLayout">
             <param name="ConversionPattern" value="%d %-5p [%c] %m"/>

    Also, remember to add the JMX appender in the root category

          <appender-ref ref="CONSOLE"></appender-ref>
          <appender-ref ref="FILE"></appender-ref>
          <appender-ref ref="JMX"></appender-ref>    <!-- ADD HERE -->      

    This appender transforms log4j messages to JMX notification so they can

    be fed to the table.


    3. for an example of a stateful alarm, edit deploy/monitoring-service.xml

    and uncomment the MemoryMonitor:

      <mbean code=""
        <attribute name="FreeMemoryWarningThreshold">95M</attribute>
        <attribute name="FreeMemoryCriticalThreshold">80M</attribute>    


    4. just for the sake of the example, create a faulty deployment, e.g.

    create an empty directory deploy/bad.sar


    5. restart the server!


    6. Go to the web-console http://localhost:8080/web-console/


    Under the Monitoring tab, there should be a new entry, "Alarm Table". By

    clicking on this you should get something similar to the picture below.

    In this example we see an error thrown by the URLDeploymentScanner

    when the initial deployment round was complete, pointing out bad.sar

    as a failed deployment. This is a stateless alarm and it will just go away

    if you Ack(nowledge) it.


    Another stateful alarm may be created depending on the settings of the

    MemoryMonitor service. The state of this alarm changes depending on

    the free memory of the system. Acknowledging the alarm will make it

    go away, if it CLEARED by the system, too.


    There should be another alarm showing the stackstrace of the bad.sar

    deployment, but I have already ack'ed this in the following picture:








    Referenced by: