Define alert conditions for JBoss AS

[swiderski.maciej]

Hi,

after successful installation and configuration of RHQ/JON I started to prepare it for real production usage - meaning define alerts. So have some basic alerts that I define via alert templates so they will be applied to all instances of given resource. Let's take JBoss AS as an example. I would like to define alerts with following conditions (among others that I know how to do):

• data source/connection pool exhausted meaning active count is equal or close to max pool size setting
• CPU utilization by JVM is higher than 85%

How I can define such conditions? It looks like it's only possible to reason over just one metric and cannot compare two metrics even if one of them is sort of constant (settings). How would you recommend to achieve described conditions?

And another related question is about dampening - if I set ti to occurrences - how often it will evaluate the metric/conditions before it fires? For example I have a http service that pings given url in app and checks if it's up and running. Then I define the alert to fire when it's down but would like to avoid situation that it fires when for example application is redeployed so would like to see it fired when it is down for 5 consecutive checks but then how much time would it mean in reality?

Thanks

Maciej

[jayshaughnessy]

Hi Maciej,

Unfortunately you're right, we don't yet support conditions that compare values of multiple metrics.  The way that gets done today is typically via the creation of another metric that calculates the desired value (like pool-percent-used) and report/alert against the single metric.  If we don't already provide the desired metric it could be added as a customization.  That involves adding the new metric definition to the plugin descriptor (rhq-plugin.xml) for the plugin, possibly adding code to collect the metric, and redeploying the updated plugin.

As for dampening, if you set "consecutive occurrences to N", that means that the alert will fire only if the conditions are met N evaluations in a row.  The condition set is evaluated each time a relevant condition is evaluated.  The amount of time it takes to do that *is variable*.  It depends on the collection intervals for the relevant metrics.  See Monitoring->Schedule subtab in the GUI to see the list of metrics, their enabled/disabled state, and their collection intervals.  By default many metrics are disabled out-of-box, and if enabled are set to conservative collection intervals.  For a particular alerting use case a metric may need to be enabled, and possibly collected more frequently.

Now, for Availability alerting (UP, DOWN, etc) you need to be careful because availability checking only reports *changes* in availability.  So a dampening rule on Goes DOWN, like "consecutive occurrences = 5" is not a good idea.  This is because DOWN will be reported once, when it goes DOWN.  The next availability report will likely be Goes UP, when it comes back up.  So it won't be reported DOWN 5 times in a row.  What people usually use in this case is an Availability Duration alert condition,like "Goes DOWN and stays DOWN for N minutes".  And then you don't set dampening at all.   Again, collection interval is relevant.  Each resource will have an "Availability" metric.  By default the Availability metric collection interval is set to 1 minute for Servers and 10 minutes for services.

Jay

[swiderski.maciej]

Jay,

could you please provide some more details on how to define metric that calculate value? Does that need to be done with custom implementation as well?

Cheers

Maciej

[genman]

You need to create custom code.

For example, take a look at the code that calculates the heap percent free for the JMX plugin.

See: rhq/modules/plugins/jmx/src/main/java/org/rhq/plugins/jmx/MBeanResourceComponent.java (RHQ source code)

You can probably do a better job than this. I think the calculation code doesn't really belong in this class.

[swiderski.maciej]

thanks guys, now I have all the details required.

Maciej