2 Replies Latest reply on Jan 18, 2006 4:21 PM by Dimitris Andreadis

    Duplicate monitor notifications received

    jason1 Newbie

      I am seeing behavior that is against the JMX specification for monitors sending duplicate notifications AFAIK. For instance...

      Once sent, Counter monitors should not send until the new threshold is reached.

      Once a low threshold is sent, Gauge monitors should not send again until the high threshold has been reached and then the low threshold is reached again. Vice versa for high.

      Am I experiencing a bug, configuring something wrong, or just not understanding how the duplicate notifications are supposed to be handled?

      thanks!



      I have the following setup:
      JBoss Version: 4.0.2 (JBoss Java Management implementation with that install)
      JDK 1.4.2

      1. A GagueMonitor that observes a custom MBean for "memory % used" and notifies on low = 20% and high = 90%.

      NotifyHigh = True
      NotifyLow = True
      Active = True
      DifferenceMode = false
      GranularityPeriod = 10000
      HighThreshold = 90
      LowThreshold = 20
      ObservedObjects[] = application.system:service=JVMMemory
      ObservedAttribute = PercentMemoryUsed



      2. A CounterMonitor that observes JMS QueueDepth reporting on threshold of 5.

      Notify = True
      Active = True
      DifferenceMode = false
      GranularityPeriod = 30000
      DerivedGauge=0
      OffSet=5
      Modulus=0
      InitThreshold = 5
      Threshold=5
      ObservedObjects[] = jboss.mq.destintaion:service=Queue,name=TestQueue, (and 10 other Queues as well)
      ObservedAttribute = QueueDepth

      3. An MBean NotificationListener called "AlertCollectorMBean" that is registered to each. AlertCollector adds the javax.management.monitor.MonitorNotification to a Set each time one is received. It logs ERROR if a duplicate is received based on Set.add() returning 'false' (e.g., Set already has the exact MonitorNotification). A client program periodically retrieves the cached alerts in AlertCollectorMBean and then clears the Set afterwards

      4. The monitored "TestQueue" contains 6 messages (e.g., QueueDepth=6 thereby breaking the threshold of 5). It ALWAYS stays at QueueDepth = 6 because there are no publishers or consumers on the Queue.

      Expected Behavior
      When JBoss starts, it should to send 2 monitor notifications:
      a. QueueDepth threshold reached since QueueDepth is 6 and threshold is 5
      b. JVM Memory low threshold reached since it generally goes below 20% used.

      Unfortunately, as the logs indicate, I am getting duplicate monitor notifications sent (so fast actually some are logging in the same millisecond!). It happens on both Counter and GaugeMonitor. When I get a duplicate, I include a dummy Exception for stack trace information.


      I get the following logging when running the application:

      DEBUG [2006-01-18 10:55:08,635] AlertCollector: [application.alerts:service=AlertCollector] getAlerts: size = 0
      DEBUG [2006-01-18 10:55:38,931] AlertCollector: [application.alerts:service=AlertCollector] handleNotification: javax.management.monitor.MonitorNotification: type=jmx.monitor.counter.threshold source=application.monitors:name=QueueDepthCounterMonitor,type=JMS sequence=0 time=1137606938931 message=threshold exceeded derivedGauge=6 observedAttribute=QueueDepth observedObject=jboss.mq.destination:name=TestQueue,service=Queue trigger=5
      DEBUG [2006-01-18 10:55:40,587] AlertCollector: [application.alerts:service=AlertCollector] clearAlerts: size = 1
      DEBUG [2006-01-18 10:55:59,946] AlertCollector: [application.alerts:service=AlertCollector] handleNotification: javax.management.monitor.MonitorNotification: type=jmx.monitor.gauge.low source=application.monitors:name=JVMMemoryPercentUsedMonitor,type=SERVER sequence=0 time=1137606959946 message=low threshold exceeded derivedGauge=20 observedAttribute=PercentMemoryUsed observedObject=application.system:service=JVMMemory trigger=20
      DEBUG [2006-01-18 11:12:07,212] AlertCollector: [application.alerts:service=AlertCollector] getAlerts: size = 1
      DEBUG [2006-01-18 11:15:14,346] AlertCollector: [application.alerts:service=AlertCollector] handleNotification: javax.management.monitor.MonitorNotification: type=jmx.monitor.counter.threshold source=application.monitors:name=QueueDepthCounterMonitor,type=JMS sequence=1 time=1137608114346 message=threshold exceeded derivedGauge=6 observedAttribute=QueueDepth observedObject=jboss.mq.destination:name=TestQueue,service=Queue trigger=5
      ERROR [2006-01-18 11:15:14,409] AlertCollector: [application.alerts:service=AlertCollector] JMX monitoring sent duplicate alert javax.management.monitor.MonitorNotification: type=jmx.monitor.counter.threshold source=application.monitors:name=QueueDepthCounterMonitor,type=JMS sequence=1 time=1137608114346 message=threshold exceeded derivedGauge=6 observedAttribute=QueueDepth observedObject=jboss.mq.destination:name=TestQueue,service=Queue trigger=5
      java.lang.Exception: STACKTRACE VERIFIER
      at cdot.monitor.AlertCollector.handleNotification(AlertCollector.java:140)
      at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.mx.notification.NotificationListenerProxy.invoke(NotificationListenerProxy.java:138)
      at $Proxy558.handleNotification(Unknown Source)
      at org.jboss.mx.util.JBossNotificationBroadcasterSupport.handleNotification(JBossNotificationBroadcasterSupport.java:112)
      at org.jboss.mx.util.JBossNotificationBroadcasterSupport.sendNotification(JBossNotificationBroadcasterSupport.java:95)
      at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:58)
      at javax.management.monitor.Monitor.sendNotification(Monitor.java:316)
      at javax.management.monitor.CounterMonitor.sendThresholdExceededNotification(CounterMonitor.java:452)
      at javax.management.monitor.CounterMonitor.monitor(CounterMonitor.java:359)
      at javax.management.monitor.CounterMonitor$1.monitorCallback(CounterMonitor.java:260)
      at org.jboss.mx.util.MonitorRunnable.runMonitor(MonitorRunnable.java:145)
      at org.jboss.mx.util.MonitorRunnable.runMonitor(MonitorRunnable.java:175)
      at org.jboss.mx.util.MonitorRunnable.doRun(MonitorRunnable.java:161)
      at org.jboss.mx.util.SchedulableRunnable.run(SchedulableRunnable.java:172)
      at org.jboss.mx.util.ThreadPool$Worker.run(ThreadPool.java:238)

      DEBUG [2006-01-18 11:15:17,049] AlertCollector: [application.alerts:service=AlertCollector] getAlerts: size = 2
      DEBUG [2006-01-18 11:15:17,065] AlertCollector: [application.alerts:service=AlertCollector] clearAlerts: size = 2
      DEBUG [2006-01-18 11:16:17,750] AlertCollector: [application.alerts:service=AlertCollector] getAlerts: size = 0
      DEBUG [2006-01-18 11:16:17,766] AlertCollector: [application.alerts:service=AlertCollector] clearAlerts: size = 0
      DEBUG [2006-01-18 11:17:15,358] AlertCollector: [application.alerts:service=AlertCollector] handleNotification: javax.management.monitor.MonitorNotification: type=jmx.monitor.gauge.low source=application.monitors:name=JVMMemoryPercentUsedMonitor,type=SERVER sequence=1 time=1137608235358 message=low threshold exceeded derivedGauge=20 observedAttribute=PercentMemoryUsed observedObject=application.system:service=JVMMemory trigger=20
      ERROR [2006-01-18 11:17:15,358] AlertCollector: [application.alerts:service=AlertCollector] JMX monitoring sent duplicate alert javax.management.monitor.MonitorNotification: type=jmx.monitor.gauge.low source=application.monitors:name=JVMMemoryPercentUsedMonitor,type=SERVER sequence=1 time=1137608235358 message=low threshold exceeded derivedGauge=20 observedAttribute=PercentMemoryUsed observedObject=application.system:service=JVMMemory trigger=20
      java.lang.Exception: STACKTRACE VERIFIER
      at cdot.monitor.AlertCollector.handleNotification(AlertCollector.java:140)
      at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.mx.notification.NotificationListenerProxy.invoke(NotificationListenerProxy.java:138)
      at $Proxy558.handleNotification(Unknown Source)
      at org.jboss.mx.util.JBossNotificationBroadcasterSupport.handleNotification(JBossNotificationBroadcasterSupport.java:112)
      at org.jboss.mx.util.JBossNotificationBroadcasterSupport.sendNotification(JBossNotificationBroadcasterSupport.java:95)
      at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:58)
      at javax.management.monitor.Monitor.sendNotification(Monitor.java:316)
      at javax.management.monitor.GaugeMonitor.sendThresholdLowExceededNotification(GaugeMonitor.java:379)
      at javax.management.monitor.GaugeMonitor.monitor(GaugeMonitor.java:305)
      at javax.management.monitor.GaugeMonitor$1.monitorCallback(GaugeMonitor.java:222)
      at org.jboss.mx.util.MonitorRunnable.runMonitor(MonitorRunnable.java:145)
      at org.jboss.mx.util.MonitorRunnable.runMonitor(MonitorRunnable.java:175)
      at org.jboss.mx.util.MonitorRunnable.doRun(MonitorRunnable.java:161)
      at org.jboss.mx.util.SchedulableRunnable.run(SchedulableRunnable.java:172)
      at org.jboss.mx.util.ThreadPool$Worker.run(ThreadPool.java:238)

      DEBUG [2006-01-18 11:17:18,452] AlertCollector: [application.alerts:service=AlertCollector] getAlerts: size = 1
      DEBUG [2006-01-18 11:17:18,467] AlertCollector: [application.alerts:service=AlertCollector] clearAlerts: size = 1