6 Replies Latest reply on Jan 7, 2015 11:37 AM by Lukas Krejci

    What is a resource type, anyway?

    Lukas Krejci Apprentice

      Resource Types

       

       

      In RHQ.next we’re trying to re-think how we handle resource types. We want to make them more dynamic, CRUD-able using an API instead of being "statically" defined by a descriptor as it is in RHQ today.

      But before we can ask what is a resource type, we first have to answer the question of what actually is a resource.

       

      For the questions below, assume that we’re talking strictly about server-side features, not anything that should be defined or live on the actual monitored machines/applications.

      Only assume that somehow we receive datapoints from the machines/applications (unless additional assumptions are explicitly specified).

       

       

      Resource

      A resource generally speaking can be described as a number of measurements and configuration that logically belong together.

      While the measurements and the configuration have meaning on their own, resource gives them the primary context - they together represent one logical thing.

      But there are questions about this:

      1. Would it be useful to compose new resources ad-hoc from arbitrary measurements/config?

        E.g. out of the data that is coming from my infrastructure, I’d be able to compose a logical resource that would represent only stuff I’m really interested in. Say I knew that from time to time my webapp is causing high CPU load. If I were able to have "system load on CPU + number of requests per second on my REST endpoint" as metrics on a single resource, that resource would become a one-stop-shop for me to learn (crudely) the health of my app and the system it’s running on.

         

      2. Is a resources something set in stone or a more fluid concept? In another words, would it be useful to dynamically add and remove metrics/config from a resource?

         

        In the example at 1., I’d be able to add/remove additional metrics from my logical "health resource" over time as I learned more about the perf indicators.

         

      3. Is a measurement or configuration confined to a single resource or can it be part of multiple?

         

        Again, this is really only a useful question to ask if we consider 1. as a viable option. If it is, then the answer to this question is that obviously they would be part of multiple resources.

         

      4. Wouldn’t grouping of "whole" resources be enough to learn the same things?

         

        IMHO, the answer is "no", because the information that would be readily available at the resource itself, would be burried more deep - in the example given, I’d have a group of a CPU resource and a WAR resource and I’d have to look at the system load of the CPU and requests per second on the WAR resource (or maybe on a "sub resource" of it) to learn what I need to know.

         

      5. What if we always retained the resources as they are coming from the feeds but would add a concept of a "view" that would pick and choose data from the individual feed-backed resources?

        This is not very much different from the ad-hoc composable resource but adds a clear distinction between feed-backed resources and "synthetic" ad-hoc data view.

      Resource Type

      Resource types as they are in current RHQ are immensely useful because they provide metadata about the individual measurements/configs/operations/etc. that are applicable to resources. Thus, measurements have units and descriptions, configurations have a defined format with descriptions, too, operations have defined return types and parameter types, etc. It is crucial we keep this kind of information in some form or fashion. At the same time, we’ve known that the current way of defining the resource types is a little bit rigid and not easily understood by users that wish to create their own.

      Currently we imagine that the way out of the current "rigid" situation might be the following:

       

      1. Still allow for uploading resource type descriptors of some form using an API - the type descriptor either can remain the XML we have right now or we can try to come up with alternative formats like JSON or YAML based definitions. Still, the metadata would be described by some kind of markup.
      2. The API must allow for renaming the resource types (and their constituent parts).

        This can be done either by a special "update resource type" endpoint that would accept a "diff" of sorts or by additional metadata in the descriptors specifying the renames (like "oldNames" attribute or some such).

      3. There can be multiple versions of the resource type active in the system.
      4. The resource types can extend others (unless some important usecase is not possible, I think single inheritance should be enough).
      5. We should keep the runs-inside semantics from current RHQ because it allows for dynamic resource tree.

       

      But there is more that should be done to make the work with resource types as flexible as possible while also less cumbersome.

      Currently, the resource types are required to be first uploaded to the server and only when they are uploaded and registered there, the agents can download them and start using them.

      This is often suboptimal in situations where the feed would itself scan some kind of resource (say an MBean Server) and would like to start reporting on the found data without first needing to formally describe it using resource type.

      So, it should be possible for the feeds themselves to define resource types (this actually is kinda implicitly contained in the first bullet point above). But then there is a question of how to match 2 resource types (that may come from different feeds).

      Another suboptimal use case is when a "dumb feed" doesn’t even have a notion of a resource type and just sends name=value pairs. Even then we should be able to allow the data in and letting the server-side to then augment the data with the metadata (in the form of a resource type). In this case, the data needs to "sit" somehow on the server somewhere and wait for the server- side to assign a resource type to it.

      Also, the current practice of having the "plugins" with the resource types on the serverside is actually quite useful so as the feeds don’t have to repeat themselves and define everything that is already known to the server.

      If the feed supports RHQ plugins, it can reference the resource types used by their server-side IDs (e.g. "rhq://global/<resourceType>") instead of the details specified in the further chapters.

       

      Resource Type Descriptor

      The resource type is described by the feeds or users using a structure that resembles today’s resource type definitions in the RHQ agent’s plugin descriptors.

      The difference here is that these feeds can be sent up from the feed to the server and server then works with this information.

      Feed-side Full Resource Type Definition

      { //this is the agent-side ID of the resource type in the predefined format"
        "id" : "<agentType>://<agentName>/<resourceType>/<version>",

        "extends" : "<agentType>://<agentName>/<otherResourceType>/<otherVersion>",

        "runsInside" : [

          //list of resource type ids

        ],

       

        "configuration" : {
          //json-schema
        },
        "connection" : {
          //json-schema 
        },
        "metrics" : [
          {
            //no need for an ID of this metric, because it can be deduced from the
            //name and the resource type id
            "name" : "asdf",
            "type" : "numeric",
            "unit" : "jiffies"
          },
          ...
        ],
        "operations" : [
          {
            "name" : "asdf",
            "returnType" : {
              //json-schema 
            },
            "parameters" : [
              {
                "name" : "asdf",
                "type" : {
                  //json-schema
                }
            },
            ...
          ]
        },
        ... ]
      }

       

       

      Feed-side Datum Definition

      For feeds that don’t support resource types, a simplified format might be used.

      { 
        "id" : "<agentType>://<agentName>/<resourceType>/<version>/metric/<name>",
        "type" : "numeric",
        "unit" : "jiffies"
      }

       

       

      { 
        "id" : "<agentType>://<agentName>/<resourceType>/<version>/operation/<name>",
        "returnType" : {
        //json-schema
        },
        "parameters" : [
          {
            "name" : "asdf",
            "type" : {
              //json-schema
            }
          },
          ...
        ]
      }

       

      and similarly for configuration and connection.

       

      Server-side Datum Definition

      Notice that on the server, the datum is represented quite differently: * i18n definitions * contentHash for server-side matching of "similar" types * representedBy for linking "stuff" together (see Resource Type Linking post)

      { 
        //the dataType, name and version got merged into this, resourceType is optional
        "id" : "agentType://feedName/resourceType/version/metric/asdf",
        "type" : "numeric",
        "unit" : "jiffies",
        "i18nKey" : "metric.asdf",
        "contentHash" : "hash" //sha of "metric" + type + unit
      }

       

      or simply

       

      { 
        "id" : "agentType://feedName/resourceType/version/metric/asdf",
        "representedBy" : "otherAgentType://otherFeedName/otherVersion/otherResourceType/metric/otherName"
      }

       

       

      { 
        //the dataType and name got merged into this, resourceType is optional.
        //Yes, this means no operation overloading
        "id" : "agentType://feedName/resourceType/version/operation/asdf",
        "returnType" : {
          //json-schema
        },
        "parameters" : [
          {
            "name" : "asdf",
            "type" : {
              //json-schema
            }
          },
          ...
        ],
      "i18nKey" : "operation.asdf",
      "contentHash" : "hash" //sha of "operation" + returnType.toString() + parameters.toString()
      }

       

       

      Server-side Resource Type Definition

      Notice that on the server, the datum is represented quite differently: * i18n definitions * contentHash for server-side matching of "similar" types * representedBy for linking "stuff" together (see Resource Type Linking chapter)

      { 
      "id" : "<agentType>://<agentName>/<resourceType>/<version>",
      "metrics" : [
      //list of metric ids
      ],
      "operations" : [
      //list of operation ids
      ]
      }

       

      or simply

       

      { 
      "id" : "<agentType>://<agentName>/<resourceType>/<version>",
      "representedBy" : "<otherAgentType>://<otherAgentName>/<otherResourceType>/<otherVersion>"
      }

       

       

      Type Update

      { 
      "removed" : [
      //list of datum or resource type IDs
      ],
      "added" : [
      //list of resource type or datum definitions as above
      ],
      "changed" : [
      {
      "id" : "agentType://agentName/resourceType/...",
      "newDefinition": {
      //datum definition (NOT full resource type) as defined above
      }
      },
      ...
      ]
      }

       

        • 1. Re: What is a resource type, anyway?
          mazz Master

          I like the concept of a feed being able to define its own resource type metadata.

           

          There are some implementation details we need to consider though. When does the feed send this metadata? Everytime it sends a piece of data (like metric data)? If not, how can it be assured the server knows about the resource type to which the data belongs? Sending it everytime would probably clobber the server (and I'll get to "sending to the server" below in a second) - e.g. to send a handful of metric bytes, do I have to send tons more metadata with it? If not, somehow the feed needs to know when it does and does not have to send the metadata (I think the agent helps here; again, see below)

           

          About feeds sending data to the server - IMO, feeds should not send data to the server directly. They should use its local agent as its proxy to the server. This does a few things:

           

          1) reduces the data flow into the server - we don't want to clobber the server with tons of (potentially duplicate) metadata not to mention data. In the past, we've seen what happens when RHQ's agents all send too much data to the server - it brings it to its knees and causes all kinds of problems. I can only imagine what would happen if we opened the floodgates and allowed feeds themselves to send directly to the server (BTW: I see feeds analogous to today's RHQ plugins). We need to make sure we design the message flow so it minimizes this kind of thing.

           

          2) Gives the agent a chance to decorate feed data so it is more descriptive. In your examples above, you talk about <agentType> and <agentName> things in resource identifiers. Its this kind of data agents can decorate around feed data before sending it up to the server. This way, feeds don't have to worry about having to know agent identifiers or even where the servers are. Just like today's RHQ plugins, feeds should only have to worry about collecting data from and operating on their managed resources - it should defer to things like talking to the server to the agent. It would then be the agent's job to know if it needs to send resource type metadata to the server (so the feed wouldn't have to send it up each time it sends data).

           

          I don't know how we are going to do all this yet I just know that is how I think it should work.

          • 2. Re: What is a resource type, anyway?
            jay shaughnessy Expert

            I like the idea of being able to create ad-hoc "views" from specific metrics of other resources. But I think it's important that users don't have to ad-hoc-create every resource, meaning that I think there should be an easy to understand (tree-like) structure to all of the data initially, for navigation.


            Perhaps metric/datum definitions should not be tied to resource types but defined independently,and maybe optionally.  A datum would still be reported for a specific resource, and the the resource would still have a type.  But the type would allow for any datum. Perhaps this would reduce the need for excessive type versioning.

             

            One question I have is around how to control the resources being reported on, the data being sent for each, and the rates of collection.

             

            As Mazz points out, a central agent may be useful, but I could also see direct reporting of data.  The types to be reported would, I think, need to be registered up front and not an every send.  The type could be getting defined for the first time.

            • 3. Re: What is a resource type, anyway?
              Lukas Krejci Apprentice

              I'd imagine the definition of the known resource types would be part of the initial handshake (as it is in RHQ nowadays, too). If the feed would like to add new resource type, it'd just "shake hands" with the "server" again.

               

              The feed would send up the list of known resource type ids, the server would respond what it needs uploaded and the feed would send up what the server said it didn't know. From then on, all should be fine and the feed would not need to send up any more metadata until (potentially) the next handshake.

               

              Note that feed-originating resource types have an ID of <agent-type>://<agent-name>/...  If the feed supported plugins in the RHQ sense, the resource type could be identified as "rhq://plugins/plugin/version/resourcetype" and we could support the same workflow as we do in RHQ nowadays - such plugins could be first uploaded to the server and analyzed, then distributed somehow to the feeds and from that point on, no metadata would be flowing from the feeds to the server - it'd already know all of it.

               

              As for the proxies I fully agree with you - the batching of the server-bound traffic as well as the decrease of the number of connections to the server (if there were more than 1 feed on a machine) would definitely be a plus. At the same time, I don't think the machine-local agent should be a requirement.

               

              But your point about the local agent enabling the really "dumb feeds" is a good one - if the feed cannot provide enough metadata for us to work with it, we need to have something with enough local context that could provide it.

               

              Btw. the "server" becomes a very vague term, because for the feeds the "server" could very well be the local nextgen agent. Such agents could report to other intermediaries that would further decorate and/or preprocess the data wrt alerting, etc. Only after a number of such "hops" could the data reach the final inventory where it would finally be routed and/or persisted.

              • 4. Re: Re: What is a resource type, anyway?
                Lukas Krejci Apprentice

                jay shaughnessy wrote:

                 

                I like the idea of being able to create ad-hoc "views" from specific metrics of other resources. But I think it's important that users don't have to ad-hoc-create every resource, meaning that I think there should be an easy to understand (tree-like) structure to all of the data initially, for navigation.

                I fully agree. The more I think about it, the more I like the distinction between a view and a "real" resource. The feeds would report tree-like parent-child inventory (with possibly other relations if they are able to find them out) and only once the resources are imported into the inventory, the views could be created ad-hoc manually (by which I mean that there would be an API for that).


                Perhaps metric/datum definitions should not be tied to resource types but defined independently,and maybe optionally.  A datum would still be reported for a specific resource, and the the resource would still have a type.  But the type would allow for any datum. Perhaps this would reduce the need for excessive type versioning.

                I think I need a more detailed explanation of this. If a resource accepted any kind of data (which would have a type), how would you for example establish compatibility between resources? We need something to establish that resources have a same metric or support the same operation so that we can act on them en masse.

                 

                One question I have is around how to control the resources being reported on, the data being sent for each, and the rates of collection.

                 

                As Mazz points out, a central agent may be useful, but I could also see direct reporting of data.  The types to be reported would, I think, need to be registered up front and not an every send.  The type could be getting defined for the first time.

                Yes, I think discovery queue is a useful concept that we should keep as a "gate" (with formally allowed auto-import). Basically when a new resource arrives at the server, it only keeps the minimum data about it (id, name, ancestry, type) and drops everything else until the resource is imported. Only then the data starts flowing through the inventory further to metrics, alerts, etc.

                 

                If the types would be registered as part of the initial handshake of the feed, they would not have to be sent on every occasion.

                • 5. Re: What is a resource type, anyway?
                  jay shaughnessy Expert

                  It's funny, I spent time over the last two days coming up with a reply to the original post, only to then realize that I had already responded once prior to the winter break.  At least I'm consistent, my "new" thoughts mirrored my previous post   Anyway, here is what I came up with on this second pass of the initial post, some of which you have already commented on above, from the first pass, sorry for any confusion...

                   

                  ----------

                   

                  There is a lot to think about here.  Without going point to point, here are some thoughts going section to section:

                   

                  Resources:

                   

                  The general concept of a whole being the sum of it's parts is good, I think.  The idea that a Resource is a collection of metrics and configuration and possibly other data, provided from potentially multiple feeds is flexible, and can lead the way to ad-hoc Resources that allow users to mix and match to their heart's content.  But, it should be the case that none of that manual assembly is initially required.  Well-understood [Fixed] Resources should appear out of box.

                   

                  Ad-hoc resources should be pliable, allowing the consolidated information to be added or removed.  I don't think this is required for fixed resources, although their data could be filtered at presentation (and maybe prior to storage). I'm not sure that the reporting of data should be controlled by the resources, meaning I'm not sure that collection schedules would be a concept carried from RHQ, primarily because we may not have that kind of centralized control of feeds.  Controlling feeds is a topic to be discussed but I wonder whether we should adopt a, "store first, ask questions later" approach and leave feed configuration to the feeds themselves. Or, perhaps, as you have mentioned, be able to more finely control feeds that support that sort of feature.

                   

                  So, a measurement could be part of multiple Resources, but I think it would be 1 fixed resource and 0 or more ad-hoc resources.  Your concept of a "view" is perhaps better than actually having ad-hoc resources.  A "view" sounds basically like set of fixed resources+filters.  This ia almost like a custom dashboard.  It may make sense to keep views and resources separate as a view may not be applicable to things like group inclusion or other resource-oriented things.  We may find life easier not trying to create a special kind of "resource", and to think of views as a different thing, perhaps made easier by the more loosely coupled metric/config approach you are talking about.

                   

                   

                  Resource Types:

                   

                  I agee that renaming must be possible but I think perhaps we should have one immutable internal name that simply does not change, and allow changes to display names.  In fact I think all type meta-data should have an internal name and a display name.  This could help a lot for update, and potentially for I18N.

                   

                  I don't love the idea of supporting different type versions. Do we really need this?  It may become more obvious is we can see a use case motivating your design here.

                   

                  I also don't love the idea of allowing untyped name-value pairs being sent in.  It would make more sense to me that a metric coming in have a resource-id associated, and that the resource be for a declared resource type. Although I think the type definition could came from the same feed, prior to sending data.  I think perhaps types could be sent as you mentioned, up front, via some sort of descriptor, or perhaps even as a url to some published descriptor (not unlike a schema or dtd)?

                   

                  I'd like to see some examples of what a feed might actually be sending, for example, if I want to send some metric-value, that would be for some resource (with what sort of ID) of some type, for some metric.  The resource type would need to be known. But perhaps the metric does not. For example, a metric M comes in for resource R of type T.  It is numeric, but otherwise untyped.  Can we just add M to T with default numeric metric attributes?

                   

                  You ask, "But then there is a question of how to match 2 resource types (that may come from different feeds)". I'm not 100% sure I understand this. If it is a well-known, published type (like from one of today's plugin descriptor) the feed would just send that descriptor (or url), one time, up front.  If it's known only to the feed then the feed would define it.  Unique internal names would be required.  Anyway, again, an example would help me understand.

                  • 6. Re: What is a resource type, anyway?
                    Lukas Krejci Apprentice

                    The more I think about it, the more I like the distinction between a feed-originating resource (i.e. a sort of a traditional rhq-style resource) with a type known a priori and ad-hoc views that would be composed of individual datapoints from various other resources. As you say, this keeps the model clean.

                     

                    But I do think that the views do also need to have their type and need to be able to participate in groups. If the views should be a way of consolidating information, then I'd like to be able to act on that information, too. The grouping mechanisms in current RHQ are one of the big value-adds, IMHO, that we should strive to keep.

                     

                    If we follow that argument, though, we need to be able to match the types of such views to determine whether we are even able to act on them en masse (this is another angle to view at the need for structural matches of resource types expanded upon bleow).

                     

                    Also, I don't believe a resource type should be "well known and published" or, in another words, globally unique (across all feeds). If we had for example a simple python application that would call its resource type "metrics" and was deployed on 15 servers and we had a bash script that would send in another resource type called "metrics" for a completely different application, I think it is unreasonable to require for the two to a) accept some kind of configuration that would make them call their resource types a different name or b) to consider each and every of those resource types different (because after all, those 15 python apps are the same) or c) to consider them all the same based on their name (hence consider them "published") - after all, the bash script wouldn't match the python apps.

                     

                    To avoid all those "unreasonable approaches", I think the best way is to consider a structural match on the resource types to determine their equality (with possibly the user still having the (version 2) ability to override that decision). With a structural match of the resource types, the 15 python apps would be considered to have the same resource type (even though each of the feeds would declare it itself, there would not have to be no prior step to make the type known globally across all the feeds (i.e. the feeds would not be required to assign the same ID to that type (as opposed to the name), nor there would be a requirement for the type to have a globally unique name) and the bash script would be considered to have a different resource type based on its different structure.

                    I'd like to see some examples of what a feed might actually be sending, for example, if I want to send some metric-value, that would be for some resource (with what sort of ID) of some type, for some metric.  The resource type would need to be known. But perhaps the metric does not. For example, a metric M comes in for resource R of type T.  It is numeric, but otherwise untyped.  Can we just add M to T with default numeric metric attributes?

                    So resources with the same resource type could report different data? I'm not sure you could then call them compatible or "same".

                     

                    As for the different resource type versions - not having this ability would require all feeds to be upgraded at the same time and would complicate historical views (i.e. a year ago this app didn't have this kind of capability that is being reported on since its version 2, so hawkular (yay, I said it! ) should not pretend that the app had that metric then). Maybe the version doesn't need to be explicit (even though I think it doesn't hurt and makes it actually easier for the user to make sense of it) but we certainly do need it at least internally for tracking the historical evolution of it (i.e. a resource type with the same ID could, throughout the time, have different structure yet we'd still consider it one and be able to assert the correct set of data types for its resources when viewing their state at appropriate times in the past).

                    I also don't love the idea of allowing untyped name-value pairs being sent in.  It would make more sense to me that a metric coming in have a resource-id associated, and that the resource be for a declared resource type. Although I think the type definition could came from the same feed, prior to sending data.  I think perhaps types could be sent as you mentioned, up front, via some sort of descriptor, or perhaps even as a url to some published descriptor (not unlike a schema or dtd)?

                    I also don't particularly love that idea. I think Mazz's approach of having an intermediary feed/agent that would fill in the type is favorable. If people are only able to report untyped data, then let them report such data to a simple endpoint that will just decorate the data with some configured metadata (i.e. resource type) and send it on. We could even have some sample impls of such proxies.

                     

                    I agee that renaming must be possible but I think perhaps we should have one immutable internal name that simply does not change, and allow changes to display names.  In fact I think all type meta-data should have an internal name and a display name.  This could help a lot for update, and potentially for I18N.

                    I agree that that is a simpler approach. At the same time, we all witnessed that people and even us break this assumption (that also exists in RHQ). I think this is not a vital thing to have in v1 though. Maybe we can put it aside and wait how things turn out without it. The reason I put it in is that having such ability basically makes life easier for the feed writers. They would not have to have a number of "ifs" in their code to correctly rename stuff to the originally chosen names, because they would send us just a note about it and we'd deal with the complexity of matching it historically.