Duplicate monitor notifications received
jason1 Jan 18, 2006 2:05 PMI am seeing behavior that is against the JMX specification for monitors sending duplicate notifications AFAIK. For instance...
Once sent, Counter monitors should not send until the new threshold is reached.
Once a low threshold is sent, Gauge monitors should not send again until the high threshold has been reached and then the low threshold is reached again. Vice versa for high.
Am I experiencing a bug, configuring something wrong, or just not understanding how the duplicate notifications are supposed to be handled?
thanks!
I have the following setup:
JBoss Version: 4.0.2 (JBoss Java Management implementation with that install)
JDK 1.4.2
1. A GagueMonitor that observes a custom MBean for "memory % used" and notifies on low = 20% and high = 90%.
NotifyHigh = True
NotifyLow = True
Active = True
DifferenceMode = false
GranularityPeriod = 10000
HighThreshold = 90
LowThreshold = 20
ObservedObjects[] = application.system:service=JVMMemory
ObservedAttribute = PercentMemoryUsed
2. A CounterMonitor that observes JMS QueueDepth reporting on threshold of 5.
Notify = True
Active = True
DifferenceMode = false
GranularityPeriod = 30000
DerivedGauge=0
OffSet=5
Modulus=0
InitThreshold = 5
Threshold=5
ObservedObjects[] = jboss.mq.destintaion:service=Queue,name=TestQueue, (and 10 other Queues as well)
ObservedAttribute = QueueDepth
3. An MBean NotificationListener called "AlertCollectorMBean" that is registered to each. AlertCollector adds the javax.management.monitor.MonitorNotification to a Set each time one is received. It logs ERROR if a duplicate is received based on Set.add() returning 'false' (e.g., Set already has the exact MonitorNotification). A client program periodically retrieves the cached alerts in AlertCollectorMBean and then clears the Set afterwards
4. The monitored "TestQueue" contains 6 messages (e.g., QueueDepth=6 thereby breaking the threshold of 5). It ALWAYS stays at QueueDepth = 6 because there are no publishers or consumers on the Queue.
Expected Behavior
When JBoss starts, it should to send 2 monitor notifications:
a. QueueDepth threshold reached since QueueDepth is 6 and threshold is 5
b. JVM Memory low threshold reached since it generally goes below 20% used.
Unfortunately, as the logs indicate, I am getting duplicate monitor notifications sent (so fast actually some are logging in the same millisecond!). It happens on both Counter and GaugeMonitor. When I get a duplicate, I include a dummy Exception for stack trace information.
I get the following logging when running the application:
DEBUG [2006-01-18 10:55:08,635] AlertCollector: [application.alerts:service=AlertCollector] getAlerts: size = 0
DEBUG [2006-01-18 10:55:38,931] AlertCollector: [application.alerts:service=AlertCollector] handleNotification: javax.management.monitor.MonitorNotification: type=jmx.monitor.counter.threshold source=application.monitors:name=QueueDepthCounterMonitor,type=JMS sequence=0 time=1137606938931 message=threshold exceeded derivedGauge=6 observedAttribute=QueueDepth observedObject=jboss.mq.destination:name=TestQueue,service=Queue trigger=5
DEBUG [2006-01-18 10:55:40,587] AlertCollector: [application.alerts:service=AlertCollector] clearAlerts: size = 1
DEBUG [2006-01-18 10:55:59,946] AlertCollector: [application.alerts:service=AlertCollector] handleNotification: javax.management.monitor.MonitorNotification: type=jmx.monitor.gauge.low source=application.monitors:name=JVMMemoryPercentUsedMonitor,type=SERVER sequence=0 time=1137606959946 message=low threshold exceeded derivedGauge=20 observedAttribute=PercentMemoryUsed observedObject=application.system:service=JVMMemory trigger=20
DEBUG [2006-01-18 11:12:07,212] AlertCollector: [application.alerts:service=AlertCollector] getAlerts: size = 1
DEBUG [2006-01-18 11:15:14,346] AlertCollector: [application.alerts:service=AlertCollector] handleNotification: javax.management.monitor.MonitorNotification: type=jmx.monitor.counter.threshold source=application.monitors:name=QueueDepthCounterMonitor,type=JMS sequence=1 time=1137608114346 message=threshold exceeded derivedGauge=6 observedAttribute=QueueDepth observedObject=jboss.mq.destination:name=TestQueue,service=Queue trigger=5
ERROR [2006-01-18 11:15:14,409] AlertCollector: [application.alerts:service=AlertCollector] JMX monitoring sent duplicate alert javax.management.monitor.MonitorNotification: type=jmx.monitor.counter.threshold source=application.monitors:name=QueueDepthCounterMonitor,type=JMS sequence=1 time=1137608114346 message=threshold exceeded derivedGauge=6 observedAttribute=QueueDepth observedObject=jboss.mq.destination:name=TestQueue,service=Queue trigger=5
java.lang.Exception: STACKTRACE VERIFIER
at cdot.monitor.AlertCollector.handleNotification(AlertCollector.java:140)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.mx.notification.NotificationListenerProxy.invoke(NotificationListenerProxy.java:138)
at $Proxy558.handleNotification(Unknown Source)
at org.jboss.mx.util.JBossNotificationBroadcasterSupport.handleNotification(JBossNotificationBroadcasterSupport.java:112)
at org.jboss.mx.util.JBossNotificationBroadcasterSupport.sendNotification(JBossNotificationBroadcasterSupport.java:95)
at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:58)
at javax.management.monitor.Monitor.sendNotification(Monitor.java:316)
at javax.management.monitor.CounterMonitor.sendThresholdExceededNotification(CounterMonitor.java:452)
at javax.management.monitor.CounterMonitor.monitor(CounterMonitor.java:359)
at javax.management.monitor.CounterMonitor$1.monitorCallback(CounterMonitor.java:260)
at org.jboss.mx.util.MonitorRunnable.runMonitor(MonitorRunnable.java:145)
at org.jboss.mx.util.MonitorRunnable.runMonitor(MonitorRunnable.java:175)
at org.jboss.mx.util.MonitorRunnable.doRun(MonitorRunnable.java:161)
at org.jboss.mx.util.SchedulableRunnable.run(SchedulableRunnable.java:172)
at org.jboss.mx.util.ThreadPool$Worker.run(ThreadPool.java:238)
DEBUG [2006-01-18 11:15:17,049] AlertCollector: [application.alerts:service=AlertCollector] getAlerts: size = 2
DEBUG [2006-01-18 11:15:17,065] AlertCollector: [application.alerts:service=AlertCollector] clearAlerts: size = 2
DEBUG [2006-01-18 11:16:17,750] AlertCollector: [application.alerts:service=AlertCollector] getAlerts: size = 0
DEBUG [2006-01-18 11:16:17,766] AlertCollector: [application.alerts:service=AlertCollector] clearAlerts: size = 0
DEBUG [2006-01-18 11:17:15,358] AlertCollector: [application.alerts:service=AlertCollector] handleNotification: javax.management.monitor.MonitorNotification: type=jmx.monitor.gauge.low source=application.monitors:name=JVMMemoryPercentUsedMonitor,type=SERVER sequence=1 time=1137608235358 message=low threshold exceeded derivedGauge=20 observedAttribute=PercentMemoryUsed observedObject=application.system:service=JVMMemory trigger=20
ERROR [2006-01-18 11:17:15,358] AlertCollector: [application.alerts:service=AlertCollector] JMX monitoring sent duplicate alert javax.management.monitor.MonitorNotification: type=jmx.monitor.gauge.low source=application.monitors:name=JVMMemoryPercentUsedMonitor,type=SERVER sequence=1 time=1137608235358 message=low threshold exceeded derivedGauge=20 observedAttribute=PercentMemoryUsed observedObject=application.system:service=JVMMemory trigger=20
java.lang.Exception: STACKTRACE VERIFIER
at cdot.monitor.AlertCollector.handleNotification(AlertCollector.java:140)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.mx.notification.NotificationListenerProxy.invoke(NotificationListenerProxy.java:138)
at $Proxy558.handleNotification(Unknown Source)
at org.jboss.mx.util.JBossNotificationBroadcasterSupport.handleNotification(JBossNotificationBroadcasterSupport.java:112)
at org.jboss.mx.util.JBossNotificationBroadcasterSupport.sendNotification(JBossNotificationBroadcasterSupport.java:95)
at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:58)
at javax.management.monitor.Monitor.sendNotification(Monitor.java:316)
at javax.management.monitor.GaugeMonitor.sendThresholdLowExceededNotification(GaugeMonitor.java:379)
at javax.management.monitor.GaugeMonitor.monitor(GaugeMonitor.java:305)
at javax.management.monitor.GaugeMonitor$1.monitorCallback(GaugeMonitor.java:222)
at org.jboss.mx.util.MonitorRunnable.runMonitor(MonitorRunnable.java:145)
at org.jboss.mx.util.MonitorRunnable.runMonitor(MonitorRunnable.java:175)
at org.jboss.mx.util.MonitorRunnable.doRun(MonitorRunnable.java:161)
at org.jboss.mx.util.SchedulableRunnable.run(SchedulableRunnable.java:172)
at org.jboss.mx.util.ThreadPool$Worker.run(ThreadPool.java:238)
DEBUG [2006-01-18 11:17:18,452] AlertCollector: [application.alerts:service=AlertCollector] getAlerts: size = 1
DEBUG [2006-01-18 11:17:18,467] AlertCollector: [application.alerts:service=AlertCollector] clearAlerts: size = 1