6 Replies Latest reply on Jun 8, 2009 9:35 AM by mazz

    Acting on Resource becoming unavailable

    fbrueseke

      Hi Jopr experts.

      As I am improving my plugin I am at point where I want to react on a resource becoming unavailable. I have some state information in my ResourceComponent that needs to be updated when Jopr notices that this resource becomes unavailable. How do I do this?

      Is a ResourceComponent notified when a resource becomes unavailable?

      Kind regards
      Frank

        • 1. Re: Acting on Resource becoming unavailable
          pilhuhn

          Can't you just do what you want from inside getAvailability() before this one returning DOWN?

          Otherwise, you'd need to set up an alert that would trigger an operation to do what you want.

          • 2. Re: Acting on Resource becoming unavailable
            fbrueseke

            Sorry, for replying this late.

            Acutually using the getAvailability method only solves the problem partially. It all depends on when which getAvailability() method is called. Suppose we have this resource hierarchy:

            server S:
            - service A
            - service B
            - service C

            Now A notices it is down and updates its state accordingly. Then S notices it is down and marks all its children as down as well. Somehow S's children do not notice this. What I get is that A has valid state, but B and C still have their old, invalid state.

            So, how does this trigger work. Can you explain, or give me a pointer?

            Kind regards
            Frank

            • 3. Re: Acting on Resource becoming unavailable
              mazz

              OOoooo... that is an interesting scenario.

              I can explain how the plugin containg works today.

              Every 60 seconds, the plugin container polls the managed resources. It does this by walking the hierarchy tree - asking each resource "are you UP?". If a resource is UP, the plugin container walks the children of that resource (if it has children). If a resource is DOWN, the plugin container will not walk its children - it will assume that if the parent is down, all of its children are down as well.

              So, if service B and C were previously UP, but the next avail checking sees that server S is DOWN, service B and C's components will not get their getAvailability methods called since they will be assumed DOWN (because typically, if a parent resource is down and cannot be communicated with, the child services also will not be able to be reached - think of the typical use case where the server is a JMX Server and the services are MBeans - if I can't talk to the JMX Server to see if its up, the child MBeans can't be reached either because they are inside that JMX Server).


              You want some kind of notification mechanism for the plugin container to tell your services that they are down if their parent server is down.

              I would think that this is not the job of the plugin container. The plugin container is merely a probe here - performing some read-only "getAvailability" calls on your resource hierarchy. To me, it seems counter-intuitive for the PLUGIN CONTAINER to be the one to tell the RESOURCES they are down - it is the other way around (the resources tell the plugin container they are down).

              It would be more appropriate for your service components to register with your parent server component and have your parent server component then notify its children when they should be considered down. So, for example, your service components would get a reference to their parent component, and that parent component would have to implement some kind of registration or listener API that your child services call. Then, in your parent component's getAvailability, if its down, it should notify those registered child listeners that they should do whatever they need to do when they are determined to be down.

              • 4. Re: Acting on Resource becoming unavailable
                fbrueseke

                Hi mazz.

                Well I don't think it is so strange for the plugin-container to notify the resources they are down or seen as down. You maintain this kind of information anyways. You just don't propagate it to the resource components. You do this in the "markDown" method in the class "AvailabilityExecutor".

                However, I will implement the suggested call back solution. Thanks for the suggestion. It might be worthwhile to design the call back solution into the Plugin-API, if you do not want to add this notification feature to the plugin container.

                Kind regards
                Frank

                • 5. Re: Acting on Resource becoming unavailable
                  mazz

                   

                  You maintain this kind of information anyways. You just don't propagate it to the resource components.


                  Yes, that is correct. We do not propogate this kind of information down to the components because, at that specific point in the code, the plugins do not need to be involved - this is the code the plugin container needs to track the availabilities so it can send this data up to the server via the availability reports. But again, it is not proper for the plugin container to be telling the components what state they should be in. That kinda breaks the plugin API design. It should be the other way around - the plugins should be telling the PC what state they are in.

                  Also, keep in mind anytime the plugin container makes calls into the components, it requires some processing and some locking - we will try not to call into components unless we have to in order to reduce contention (calling into components requires submitting tasks to thread pools, performing locking and possibly spawning additional threads). So if a server that has 100 children goes down, rather than 1 call to getAvailability to that top server component (which returns DOWN), this would cause an additional 100! component calls - one to each child - just to inform them of something they already know (or should already know).


                  • 6. Re: Acting on Resource becoming unavailable
                    mazz

                    Oh, I wanted to also mention this. This is tangentially related, but I thought I'd bring it up.

                    The getAvailability() method needs to be implemented so it is very fast. It cannot take longer than 5 seconds, and really, for the average resource, it needs to be in the sub-second range.

                    Read this for more: http://jira.rhq-project.org/browse/RHQ-551

                    This is one part of the plugin API that I'm not 100% satisfied with - trying to come up with a good design or good plugin API to assist the plugin developer with this. (and trying to find time to implement it).

                    I would like to get away from the need to force the plugin developer to return availablity within 5 seconds. But haven't found a good solution. Would like to provide the plugin developer with an API to assist in being able to perform async. availability checking and have getAvailability just return the last known avail. The JIRA goes into detail about this.