6 Replies Latest reply on Jan 15, 2009 11:17 AM by joe.marques

    DataSource Alerts doesn't work

    rafaelcba

      Hi All.

      I define a DataSource Alert Templates for JBoss AS resource but it doesn't work - don't alert :x .

      My env is:

      O.S: Linux Debian
      Jopr 2.1
      JBoss AS 4.0.3
      JDK 5


      I observe a specific DataSource that has possible connection leak problems and RHQ doesn't send me an alert about it.

      The condition set configuratios for my Alert Template like this:

      If Condition: Active Connections > 90.0% of Baseline Value
      OR If Condition: Available Connections < 10.0% of Baseline Value
      Dampening Rule: Each time condition set is true
      Action Filters: Disable alert until re-enabled manually or by recovery alert : false




      When the DataSource has no more Available Connections this alert isn't trigged.
      I see this exception on RHQ Server log

      Select all
       2008-11-19 12:26:18,226 ERROR [org.rhq.enterprise.server.alert.engine.AlertConditionCache] Failed to calculate baseline for [conditionId=500052, baselineId=35896]: optionStatus string was 'max', but the corresponding baseline value was null
       2008-11-19 12:26:18,228 ERROR [org.rhq.enterprise.server.alert.engine.AlertConditionCache] Error loading cache for agent[id=500550]
       java.lang.NullPointerException
       at org.rhq.enterprise.server.alert.engine.AlertConditionCache.getCalculatedBaselineValue_helper(AlertConditionCache.java:1025)
       at org.rhq.enterprise.server.alert.engine.AlertConditionCache.getCalculatedBaselineMeanValue(AlertConditionCache.java:988)
       at org.rhq.enterprise.server.alert.engine.AlertConditionCache.insertAlertConditionComposite(AlertConditionCache.java:849)
       at org.rhq.enterprise.server.alert.engine.AlertConditionCache.loadCachesForAgent(AlertConditionCache.java:332)
       at org.rhq.enterprise.server.alert.engine.AlertConditionCache.reloadCachesForAgent(AlertConditionCache.java:244)
       at org.rhq.enterprise.server.alert.engine.AlertConditionCacheManagerBean.reloadCachesForAgent(AlertConditionCacheManagerBean.java:84)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:112)
       at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
       at org.rhq.enterprise.server.authz.RequiredPermissionsInterceptor.checkRequiredPermissions(RequiredPermissionsInterceptor.java:153)
       at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:118)
       at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
       at org.jboss.aspects.tx.TxInterceptor$RequiresNew.invoke(TxInterceptor.java:262)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:62)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
       at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:106)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
       at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
       at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:214)
       at org.jboss.ejb3.stateless.StatelessContainer.localInvoke(StatelessContainer.java:184)
       at org.jboss.ejb3.stateless.StatelessLocalProxy.invoke(StatelessLocalProxy.java:81)
       at $Proxy195.reloadCachesForAgent(Unknown Source)
       at org.rhq.enterprise.server.core.CoreServerServiceImpl.connectAgent(CoreServerServiceImpl.java:190)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.rhq.enterprise.communications.command.impl.remotepojo.server.RemotePojoInvocationCommandService.execute(RemotePojoInvocationCommandService.java:184)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201)
       at $Proxy350.execute(Unknown Source)
       at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:358)
       at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:253)
       at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:734)
       at org.jboss.remoting.transport.servlet.ServletServerInvoker.processRequest(ServletServerInvoker.java:231)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:585)
       at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
       at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
       at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
       at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
       at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
       at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201)
       at $Proxy343.processRequest(Unknown Source)
       at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:128)
       at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.doPost(ServerInvokerServlet.java:157)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:113)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
       at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
       at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
       at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
       at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
       at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
       at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
       at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
       at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
       at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
       at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      



      I think RHQ Agent send the alert to RQH Server but anythis is wrong on Server.

      Other alerts work fine! Only Datasource's alert doesn't wonk here.

      Any Idea about this?

        • 1. Re: DataSource Alerts doesn't work

          This might be an issue if the system has not calculated baselines yet (you can configuration the amount of data to use and the frequency of baseline calculation in the Administration section). In the latest released version of RHQ (1.1.2.GA), there wasn't sufficient error-handling when your resource did not have it's baselines calculated before you created the alert definition:

          http://jira.rhq-project.org/browse/RHQ-1059
          http://jira.rhq-project.org/browse/RHQ-1060

          Also, due to a known bug in 1.1.2.GA, you will not be able to create alerts off of the baseline minimum or maximum. It will only work if you create it off of the baseline mean value.

          http://jira.rhq-project.org/browse/RHQ-1061

          All three of these issues have been fixes in trunk, so if you build from source you'll be able to get around these issues until the next official release of the platform. Instructions for building are found here:

          http://support.rhq-project.org/display/RHQ/Building+RHQ

          • 2. Re: DataSource Alerts doesn't work
            rafaelcba

            Hi Marques.

            I change my condition set for this DataSource's Alert Template:

            Condition Set
            If Condition: Active Connections > 90.0% of Baseline Value
            OR If Condition: Available Connections < 10.0% of Baseline Value
            OR If Condition: Active Connections = 100.0% of Baseline Value
            Dampening Rule: Each time condition set is true
            Action Filters: Disable alert until re-enabled manually or by recovery alert : false


            Today a specific DataSource reached the top of Connections Active and 0 for Avail Connection, but the metric chart show this:
            Name Alerts O.O.B. Low Average Peak Last
            Active Connections 0 0 0.0 150.7 199.0 0.0
            Available Connections 0 0 0.0 48.3 200.0 200.0


            Any is wrong with this resource...
            May be I need clean the data for this resource?


            manually calculate the BaseLine for metric Avail Connection
            Metric Baseline & Expected Range
            Baseline: 37.6 - Change Value
            Expected Range: High Range: 200.0 - Change Value
            Low Range: 0.0 - Change Value


            And then nothing of Alert :(

            I will try the workaround suggested.

            Thanks!

            • 3. Re: DataSource Alerts doesn't work

              How long ago did you import this resource into inventory? Again, if the baselines have not been calculated yet, you may have issues.

              I see that you manually calculated the baseline for that metric, but in general you shouldn't reply on manually calculated baselines for alerting purposes. The manual baseline calculation will only *temporarily* change the baseline value. When the periodic check job comes along (governed by the baseline frequency in Administration), it will ignore your manually calculated value and replace it with the auto-calculated one. Also, a manually calculated baseline will be created by using your CURRENT metric display range (not the baseline data set in Administration), so depending on what range you were looking at when your manually calculated the baselines - last 60 mins, last 2 hours, last 8 hours, last 24 hours, etc - you might get "interesting" results.

              Aside from that, it looks like what you are trying to do is create an alert definition that triggered when the active connections spike up or down, or if the available connections get too low. I would try something like:

              If Condition: Active Connections < 75.0% of Baseline Value
              OR If Condition: Active Connections > 125.0% of Baseline Value
              OR If Condition: Available Connections < 5

              This will alert you if:

              1) the active connections swing plus/minus 25% away from the baseline value (i.e. your application is experiencing a load burst that causes the concurrent activity in your application to increase), or
              2) you're running out of connections (assuming 5 is a low value for you)

              Of course, you can adjust all of these values, but that should get your started. By the way, you should INFREQUENTLY see the need for conditions like:

              "... = 100.0% of Baseline Value"

              In this case, you would have needed the active connections to be equal to exactly "37.6", but that's impossible since that metric is collected as an integer.

              • 4. Re: DataSource Alerts doesn't work
                rafaelcba

                Hi Marques!
                My inventory has more than 40 day's metrics.

                The issue is my condition conf. It's wrong!
                I was confusing the conditions!

                I calculated the baseline for 40 days and get the 70.4 for "Available Connections" of my specific DataSource. Then I define a new set condition:

                If Condition: Active Connections > 2.5% of Baseline Value
                OR If Condition: Available Connections < 10.0% of Baseline Value


                Now my DataSource's alerts looks work :)

                According [1] on next release the baseline minimum or maximum used for condition's define will work correctly?

                Thanks for these great tips!

                [1] http://jira.rhq-project.org/browse/RHQ-1061

                • 5. Re: DataSource Alerts doesn't work
                  rafaelcba

                  Sorry!

                  This is wrong:

                  If Condition: Active Connections > 2.5% of Baseline Value
                  OR If Condition: Available Connections < 10.0% of Baseline Value


                  The correctly set condition is:

                  If Condition: Active Connections > 250.0% of Baseline Value
                  OR If Condition: Available Connections < 10.0% of Baseline Value


                  • 6. Re: DataSource Alerts doesn't work

                    Correct, the baseline-based alerts has been fixed for the next release.