9 Replies Latest reply on May 19, 2009 11:46 AM by mzeijen

    Scheduled monitoring

    mzeijen

      At my work place we have a couple of JBoss servers that get intentionally rebooted every night. We want to monitor these servers with Jopr. However that reboot could be a problem. If Jopr checks the availability during the reboot and discovers that the server is down then it will send us an Alert e-mail. Next to the availability we want to monitor the server.log file but because of a third party library there is always one error line (which doesn't really have any meaning) during the boot sequence. If we would set an alert for the server.log then we also would get an e-mail every night that we don't want to get.

      Is it possible to determine the time window when Jopr should monitor a resource? Or maybe the possibility to disable the alerting for a specific time period? Something in that direction would be great.

        • 1. Re: Scheduled monitoring
          mazz

          This is a known feature enhancement we'd like to implement some day:

          http://jira.rhq-project.org/browse/RHQ-1494

          Read that JIRA's description - describes your scenario exactly.

          As for the event log having a single ERROR - I believe you can add filters to event alerts. So you can create an alert definition to trigger an alert if an ERROR message comes in but only if that error matches a regular expression (and that regex would simply ignore the error message you want to ignore). Or, you could just go into log4j.xml and set that log4j category to FATAL to have log4j not even emit that ERROR message.

          • 2. Re: Scheduled monitoring
            mzeijen

            Thanks for the info. To bad Jopr doesn't support scheduling yet.

            • 3. Re: Scheduled monitoring

              Alternately, you can also filter what data is sent up from the agent to the server.

              If the single line that you get from this nightly restart process has a sufficiently unique token / identifier, it would also be possible to filter the message from ever getting to the server in the first place.

              When you define your event source...

              http://www.redhat.com/docs/en-US/JBoss_ON/2.2/html/Feature_Guide/sect-Feature_Guide-Events-Defining_Event_Sources.html

              ...simply use an "includes pattern" that skips over the unique token. This way, if you have multiple agents across your system, the processing required to perform the regex filter is distributed across all of them. This strategy also keeps your alert definitions snappy, since they would only need to compare the severity level of the event without any regex match.

              • 4. Re: Scheduled monitoring
                pilhuhn

                You can try to use alert dampening and only send alerts when e.g 3* availability is recorded as down within x minutes or such.

                • 5. Re: Scheduled monitoring
                  mzeijen

                   

                  "joe.marques@jboss.com" wrote:
                  Alternately, you can also filter what data is sent up from the agent to the server.

                  If the single line that you get from this nightly restart process has a sufficiently unique token / identifier, it would also be possible to filter the message from ever getting to the server in the first place.

                  When you define your event source...

                  http://www.redhat.com/docs/en-US/JBoss_ON/2.2/html/Feature_Guide/sect-Feature_Guide-Events-Defining_Event_Sources.html

                  ...simply use an "includes pattern" that skips over the unique token. This way, if you have multiple agents across your system, the processing required to perform the regex filter is distributed across all of them. This strategy also keeps your alert definitions snappy, since they would only need to compare the severity level of the event without any regex match.


                  I'll look into that. Thanks for the info.

                  • 6. Re: Scheduled monitoring
                    mazz

                    "You can try to use alert dampening and only send alerts when e.g 3* availability is recorded as down within x minutes or such. "

                    That doesn't do what you think it does.

                    The alert is only triggered on the state change - so if a resource is down for 30 minutes, you'll only get one "Going DOWN" alert. Only when the resource is seen going UP then going DOWN again will an alert get triggered.

                    • 7. Re: Scheduled monitoring
                      mzeijen

                      Some kind of scheduling system and some kind of system with which you can easily register down time is an important feature of every monitoring system. I hope these kind of features will find there way into Jopr soon.

                      • 8. Re: Scheduled monitoring
                        mazz

                        We love additional community participation and contribution. If you are willing to do so, feel free to implement this feature that you need and we can look to incorporating your patch.

                        • 9. Re: Scheduled monitoring
                          mzeijen

                          I'd love to help but I am already an active member on another open source project namely Smooks. It is very hard to combine multiple oss projects when working in my own free time.