JBoss Monitoring

In JBoss 3.2.4, we've implemented some new JMX MBean monitoring and a nice GUI around this infrastructure as well as a way to plug in how alerts are processed.  Why didn't we use the JMX Gauge and String monitors that come implemented with the JMX spec?  3 reasons:

1. They are not integrated with our Service architecture

2. They have no way of determining which monitors have sent an alert and are currently disabled because an alert was reached

3. They have no way of reseting an alert

So, what infrastructure have we built?  First let's look at configuring a JMX MBean monitor through the plain old -service.xml interface that you would to create any MBean.

All Monitors are MBeans that start a thread to watch the values of another MBean's attribute.  When a monitoring threshold is reached, the Monitor will send out an MBean Notification to a set of registered listeners.  Currently in JBoss, the are two types of listeners, a dumb Console listener, and an Email listener which will send out an email whenever an alert is received.  Of course, the messages are fully configurable.

Numeric Attribute Monitors

The first type of monitor is a org.jboss.monitor.ThresholdMonitor that is used to track Numeric MBean attributes.  Here is an example configuration of a monitor of free available memory.  It will send a JMX Notification when free memory goes below one megabyte.  The MBean it is watching over that has this particular stat is jboss.system:type=ServerInfo.

File: FreeMemory_Monitor-service.xml

<?xml version="1.0" encoding="UTF-8"?>
<server>
<mbean code="org.jboss.monitor.ThresholdMonitor"
       name="jboss.monitor:service=FreeMemory">
  <attribute name="MonitorName">FreeMemory Monitor</attribute>
  <attribute name="ObservedObject">jboss.system:type=ServerInfo</attribute>
  <attribute name="ObservedAttribute">FreeMemory</attribute>
  <attribute name="Threshold">1000000</attribute>
  <attribute name="CompareTo">1</attribute>
  <attribute name="Period">1000</attribute>
  <attribute name="Enabled">true</attribute>
  <depends-list optional-attribute-name="AlertListeners">

<depends-list-element>jboss.alerts:service=ConsoleAlertListener</depends-list-element>

<depends-list-element>jboss.alerts:service=EmailAlertListener</depends-list-element>
  </depends-list>
</mbean>
</server>

Let's walk through each attribute:

  <attribute name="MonitorName">FreeMemory</attribute>

The display name in which this monitor will be shown in the web-console.  If you create monitors by hand you can have it managed by the web-console if you have one monitor defined in one file only and the name of the file is the same name as the monitor and the file lives in ./deploy/management/monitors/  So The above example the filename should be FreeMemory_Monitor-service.xml, notice that whitespace is substituted with an '_' character.

  <attribute name="ObservedObject">jboss.system:type=ServerInfo</attribute>
  <attribute name="ObservedAttribute">FreeMemory</attribute>

These are the MBean and the MBean attribute this monitor will watch.  If the MBean is a ServiceMBean then you should make this a so that this monitor doesn't start before the watched MBean is loaded.

  <attribute name="Threshold">1000000</attribute>
  <attribute name="CompareTo">1</attribute>

The Threshold is the value threshold of the attribute.  The CompareTo is a numeric value, -1 means > (greater than), 0 means = (equal), 1 means (less than).  These are the same values used by Java comparators.  So in this example, when the FreeMemory attribute is less than 1000000 a JMX notification will be sent.

  <attribute name="Period">1000</attribute>

The MBean creates a thread that will wake up every so often to check the threshold against the MBean attribute it is watching.  The Period is the time in milliseconds this thread will sleep.

  <attribute name="Enabled">true</attribute>

Enabled determines whether or not this monitor should actually monitor.  It is the on/off switch of the monitor.

  <depends-list optional-attribute-name="AlertListeners">

<depends-list-element>jboss.alerts:service=ConsoleAlertListener</depends-list-element>

<depends-list-element>jboss.alerts:service=EmailAlertListener</depends-list-element>
  </depends-list>

When the threshold is triggered, the Monitor will send  JMX NOtification to all registered listeners.  For this particular MOnitor, a Console and Email listener have been registered.  We'll look into how to configure the Email and Console listener in a bit, but when a threshold is breached, a message will be sent to the JBoss console (System.out via the ConsoleAlertListener) and via an Email with the EmailAlertListener.

That's it!

StringThresholdMonitor

You can also monitor string values:

<?xml version="1.0" encoding="UTF-8"?>
<server>
<mbean code="org.jboss.monitor.StringThresholdMonitor"
       name="jboss.monitor:service=BindAddress_Monitor">
  <attribute name="MonitorName">BindAddress Monitor</attribute>
  <depends optional-attribute-name="ObservedObject">jboss:service=Naming</depends>
  <attribute name="ObservedAttribute">BindAddress</attribute>
  <depends-list optional-attribute-name="AlertListeners">
      <depends-list-element>jboss.alerts:service=ConsoleAlertListener     </depends-list-element>
  </depends-list>
  <attribute name="Threshold">0.0.0.0</attribute>
  <attribute name="Period">1000</attribute>
  <attribute name="EqualityTriggersAlert">true</attribute>
  <attribute name="Enabled">true</attribute>
</mbean>
</server>

The config is the same as Threshold except that there is an "EqualityTriggersAlert" attribute.  If this is true and the attribute equals the Threshold value, then an alert will be triggered

EmailAlertListener_

To configure email you must

1. Configure ./deploy/mail-service.xml to work with your mail server

2. Edit ./deploy/monitoring-service.xml and uncomment out the EmailAlertListener:

monitoring-service.xml:

<server>

  <mbean code="org.jboss.monitor.alerts.ConsoleAlertListener"
         name="jboss.alerts:service=ConsoleAlertListener">
    <attribute name="MessageTemplate"><![CDATA[%(MONITOR_NAME) was triggered for attribute %(ATTRIBUTE).\]\]\></attribute>
    <attribute name="AlertName">Console Alert</attribute>
  </mbean>

<!-- Enable after filling in correct to, from, and reply to

  <mbean code="org.jboss.monitor.alerts.EmailAlertListener"
         name="jboss.alerts:service=EmailAlertListener">
    <depends>jboss:service=Mail</depends>
    <attribute name="AlertName">Email Alert</attribute>
    <attribute name="To">somebody@somewhere.org</attribute>
    <attribute name="From">noreply@nowhere.org</attribute>
    <attribute name="ReplyTo">noreply@nowhere.org</attribute>
    <attribute name="SubjectTemplate"><![CDATA[[jboss-alert] %(MONITOR_NAME)\]\]\></attribute>
    <attribute name="MessageTemplate"><![CDATA[%(MONITOR_NAME) was triggered for attribute %(ATTRIBUTE).\]\]\></attribute>
  </mbean>

-->

</server>

Let's walk through each attribute of the email listener.

    <attribute name="AlertName">Email Alert</attribute>

This is the display name of the Listener within the web-console gui.

    <attribute name="To">somebody@somewhere.org</attribute>
    <attribute name="From">noreply@nowhere.org</attribute>
    <attribute name="ReplyTo">noreply@nowhere.org</attribute>

Are the email addresses.  "To" can be multiple emails comma delimited. From and ReplyTo must be a single email address.

    <attribute name="SubjectTemplate"><![CDATA[[jboss-alert] %(MONITOR_NAME)\]\]\></attribute>
    <attribute name="MessageTemplate"><![CDATA[%(MONITOR_NAME) was triggered for attribute %(ATTRIBUTE).\]\]\></attribute>

The SubjectTemplate is the string template for the Email Subject.  The MessageTemplate is a string template for the Email message body.  There are some substitution characters you can use:

For System Properties you can use the standard MBean service subtition of

${some.system.property}

There are also some Monitoring-specific substitions via a different delimeter:

%(MONITOR_NAME)

The thing referenced within the %(...) parens will be transfered to one of these values:

There is also a GUI for monitoring through the JBoss Management Console accessed at http://host:8080/web-console.  If you go to the System tree and click down to the JMX view on a particular MBean, open that up and you should see a list of attributes.  Right click on an attribute should give you a menu list of "graph" and "create monitor", select "create monitor" and you will be brought to the creation screen.  Monitors will show up under the Monitors tree item in the management console.