I am seeing behavior that is against the JMX specification for monitors sending duplicate notifications AFAIK. For instance...

Once sent, Counter monitors should not send until the new threshold is reached.

Once a low threshold is sent, Gauge monitors should not send again until the high threshold has been reached and then the low threshold is reached again. Vice versa for high.

Am I experiencing a bug, configuring something wrong, or just not understanding how the duplicate notifications are supposed to be handled?

thanks!



I have the following setup:
JBoss Version: 4.0.2 (JBoss Java Management implementation with that install)
JDK 1.4.2

1. A GagueMonitor that observes a custom MBean for "memory % used" and notifies on low = 20% and high = 90%.

NotifyHigh = True
NotifyLow = True
Active = True
DifferenceMode = false
GranularityPeriod = 10000
HighThreshold = 90
LowThreshold = 20
ObservedObjects[] = application.system:service=JVMMemory
ObservedAttribute = PercentMemoryUsed



2. A CounterMonitor that observes JMS QueueDepth reporting on threshold of 5.

Notify = True
Active = True
DifferenceMode = false
GranularityPeriod = 30000
DerivedGauge=0
OffSet=5
Modulus=0
InitThreshold = 5
Threshold=5
ObservedObjects[] = jboss.mq.destintaion:service=Queue,name=TestQueue, (and 10 other Queues as well)
ObservedAttribute = QueueDepth

3. An MBean NotificationListener called "AlertCollectorMBean" that is registered to each. AlertCollector adds the javax.management.monitor.MonitorNotification to a Set each time one is received. It logs ERROR if a duplicate is received based on Set.add() returning 'false' (e.g., Set already has the exact MonitorNotification). A client program periodically retrieves the cached alerts in AlertCollectorMBean and then clears the Set afterwards

4. The monitored "TestQueue" contains 6 messages (e.g., QueueDepth=6 thereby breaking the threshold of 5). It ALWAYS stays at QueueDepth = 6 because there are no publishers or consumers on the Queue.

Expected Behavior
When JBoss starts, it should to send 2 monitor notifications:
a. QueueDepth threshold reached since QueueDepth is 6 and threshold is 5
b. JVM Memory low threshold reached since it generally goes below 20% used.

Unfortunately, as the logs indicate, I am getting duplicate monitor notifications sent (so fast actually some are logging in the same millisecond!). It happens on both Counter and GaugeMonitor. When I get a duplicate, I include a dummy Exception for stack trace information.


I get the following logging when running the application:

DEBUG [2006-01-18 10:55:08,635] AlertCollector: [application.alerts:service=AlertCollector] getAlerts: size = 0
DEBUG [2006-01-18 10:55:38,931] AlertCollector: [application.alerts:service=AlertCollector] handleNotification: javax.management.monitor.MonitorNotification: type=jmx.monitor.counter.threshold source=application.monitors:name=QueueDepthCounterMonitor,type=JMS sequence=0 time=1137606938931 message=threshold exceeded derivedGauge=6 observedAttribute=QueueDepth observedObject=jboss.mq.destination:name=TestQueue,service=Queue trigger=5
DEBUG [2006-01-18 10:55:40,587] AlertCollector: [application.alerts:service=AlertCollector] clearAlerts: size = 1
DEBUG [2006-01-18 10:55:59,946] AlertCollector: [application.alerts:service=AlertCollector] handleNotification: javax.management.monitor.MonitorNotification: type=jmx.monitor.gauge.low source=application.monitors:name=JVMMemoryPercentUsedMonitor,type=SERVER sequence=0 time=1137606959946 message=low threshold exceeded derivedGauge=20 observedAttribute=PercentMemoryUsed observedObject=application.system:service=JVMMemory trigger=20
DEBUG [2006-01-18 11:12:07,212] AlertCollector: [application.alerts:service=AlertCollector] getAlerts: size = 1
DEBUG [2006-01-18 11:15:14,346] AlertCollector: [application.alerts:service=AlertCollector] handleNotification: javax.management.monitor.MonitorNotification: type=jmx.monitor.counter.threshold source=application.monitors:name=QueueDepthCounterMonitor,type=JMS sequence=1 time=1137608114346 message=threshold exceeded derivedGauge=6 observedAttribute=QueueDepth observedObject=jboss.mq.destination:name=TestQueue,service=Queue trigger=5
ERROR [2006-01-18 11:15:14,409] AlertCollector: [application.alerts:service=AlertCollector] JMX monitoring sent duplicate alert javax.management.monitor.MonitorNotification: type=jmx.monitor.counter.threshold source=application.monitors:name=QueueDepthCounterMonitor,type=JMS sequence=1 time=1137608114346 message=threshold exceeded derivedGauge=6 observedAttribute=QueueDepth observedObject=jboss.mq.destination:name=TestQueue,service=Queue trigger=5
java.lang.Exception: STACKTRACE VERIFIER
at cdot.monitor.AlertCollector.handleNotification(AlertCollector.java:140)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.mx.notification.NotificationListenerProxy.invoke(NotificationListenerProxy.java:138)
at $Proxy558.handleNotification(Unknown Source)
at org.jboss.mx.util.JBossNotificationBroadcasterSupport.handleNotification(JBossNotificationBroadcasterSupport.java:112)
at org.jboss.mx.util.JBossNotificationBroadcasterSupport.sendNotification(JBossNotificationBroadcasterSupport.java:95)
at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:58)
at javax.management.monitor.Monitor.sendNotification(Monitor.java:316)
at javax.management.monitor.CounterMonitor.sendThresholdExceededNotification(CounterMonitor.java:452)
at javax.management.monitor.CounterMonitor.monitor(CounterMonitor.java:359)
at javax.management.monitor.CounterMonitor$1.monitorCallback(CounterMonitor.java:260)
at org.jboss.mx.util.MonitorRunnable.runMonitor(MonitorRunnable.java:145)
at org.jboss.mx.util.MonitorRunnable.runMonitor(MonitorRunnable.java:175)
at org.jboss.mx.util.MonitorRunnable.doRun(MonitorRunnable.java:161)
at org.jboss.mx.util.SchedulableRunnable.run(SchedulableRunnable.java:172)
at org.jboss.mx.util.ThreadPool$Worker.run(ThreadPool.java:238)
DEBUG [2006-01-18 11:15:17,049] AlertCollector: [application.alerts:service=AlertCollector] getAlerts: size = 2
DEBUG [2006-01-18 11:15:17,065] AlertCollector: [application.alerts:service=AlertCollector] clearAlerts: size = 2
DEBUG [2006-01-18 11:16:17,750] AlertCollector: [application.alerts:service=AlertCollector] getAlerts: size = 0
DEBUG [2006-01-18 11:16:17,766] AlertCollector: [application.alerts:service=AlertCollector] clearAlerts: size = 0
DEBUG [2006-01-18 11:17:15,358] AlertCollector: [application.alerts:service=AlertCollector] handleNotification: javax.management.monitor.MonitorNotification: type=jmx.monitor.gauge.low source=application.monitors:name=JVMMemoryPercentUsedMonitor,type=SERVER sequence=1 time=1137608235358 message=low threshold exceeded derivedGauge=20 observedAttribute=PercentMemoryUsed observedObject=application.system:service=JVMMemory trigger=20
ERROR [2006-01-18 11:17:15,358] AlertCollector: [application.alerts:service=AlertCollector] JMX monitoring sent duplicate alert javax.management.monitor.MonitorNotification: type=jmx.monitor.gauge.low source=application.monitors:name=JVMMemoryPercentUsedMonitor,type=SERVER sequence=1 time=1137608235358 message=low threshold exceeded derivedGauge=20 observedAttribute=PercentMemoryUsed observedObject=application.system:service=JVMMemory trigger=20
java.lang.Exception: STACKTRACE VERIFIER
at cdot.monitor.AlertCollector.handleNotification(AlertCollector.java:140)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.mx.notification.NotificationListenerProxy.invoke(NotificationListenerProxy.java:138)
at $Proxy558.handleNotification(Unknown Source)
at org.jboss.mx.util.JBossNotificationBroadcasterSupport.handleNotification(JBossNotificationBroadcasterSupport.java:112)
at org.jboss.mx.util.JBossNotificationBroadcasterSupport.sendNotification(JBossNotificationBroadcasterSupport.java:95)
at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:58)
at javax.management.monitor.Monitor.sendNotification(Monitor.java:316)
at javax.management.monitor.GaugeMonitor.sendThresholdLowExceededNotification(GaugeMonitor.java:379)
at javax.management.monitor.GaugeMonitor.monitor(GaugeMonitor.java:305)
at javax.management.monitor.GaugeMonitor$1.monitorCallback(GaugeMonitor.java:222)
at org.jboss.mx.util.MonitorRunnable.runMonitor(MonitorRunnable.java:145)
at org.jboss.mx.util.MonitorRunnable.runMonitor(MonitorRunnable.java:175)
at org.jboss.mx.util.MonitorRunnable.doRun(MonitorRunnable.java:161)
at org.jboss.mx.util.SchedulableRunnable.run(SchedulableRunnable.java:172)
at org.jboss.mx.util.ThreadPool$Worker.run(ThreadPool.java:238)
DEBUG [2006-01-18 11:17:18,452] AlertCollector: [application.alerts:service=AlertCollector] getAlerts: size = 1
DEBUG [2006-01-18 11:17:18,467] AlertCollector: [application.alerts:service=AlertCollector] clearAlerts: size = 1