1 2 3 Previous Next 30 Replies Latest reply on Apr 9, 2010 8:17 PM by jason.greene

    To scope or not to scope (domain.xml)

    jason.greene

      In the past we talked about allowing resources to be defined, following a normal scoping mechanism, at the domain, cluster, or node level. An alternative approach would be to introduce a more limited notion of a homogenous group (1..many servers) which is the only location that things like resource definitions can occur. Creating server specific properties would require creating a new group definition. Clustering could even be viewed as a property of the homogenous group. The only issue is that some settings truly require scoping (like jvm parameters, sys properties, etc), so there would have to be special behavior there.

       

      This would give us something like:

       

      <domain>
          <group name="Production">
             <clustered-service>.....</clustered-service>
             <resource name="Production DB">
             ....
          </group>
      
          <server name="MyServer" group="Production">
                <jvm maxMemory="..." />
          </server>
      </domain>
      
        • 1. Re: To scope or not to scope (domain.xml)
          starksm64

          There is always scoping as we need to know how a property specified at the cluster or domain level applies to the server as in the example of system or jvn properties. There its straightforward; the more specific value overrides the more general.

           

          We have not talked about cluster or admin domain metadata, I mean a, so I view your example as just introducing that notion.

           

          The various domain model elements map onto a ManagedComponent in a way that need to be clearly defined. Logically I stil think about it in terms of scopes that have to be applied to a piece of deployment metadata, and each layer adds the opportunity to specify a default that can be overriden by a more specific value.

           

          So, looking at your little fragment, a clustered-service at at group (which I'm thinking of as admin-domain notion), is a specification of clustering aspects that hae to be applied to a bean or spec component. The resource element is a profile default saying that the profile that maps to the "

          Production" group should include a "Production DB" jca factory.

           

          This is our initial major challenge to moving to the domain model. We need a mapping from domain metadata namespaces onto a ManagedComponent/ManagedProperty. There can be multiple domain metadata namespaces mapping onto a given ManagedComponent/ManagedProperty.

          • 2. Re: To scope or not to scope (domain.xml)
            starksm64

            And actually its potentially a many to many mapping from a given domain metadata value to ManagedComponent/ManagedProperty when it comes to metadata that affects aspect settings like security, clustering, etc.

             

            We need to update jboss-managed and the profileservice spis to support configuration of this mapping.

            • 3. Re: To scope or not to scope (domain.xml)
              jason.greene

              I was really referring to the namespace scoping rules of the domain xml itself, not the implementation mapping, since I thought we were going to avoid talking about the jboss-managed/ps api stuff for a bit.

               

              So we could for example allow resources to be declared at every possible point in the domain.xml (domain, cluster, server, etc), which is definitely more flexible for the user. However it could also make the file more complex to understand, if you had a very large domain file. The alternative approach I mention would be less flexible (you always have to have a group definition), but it would be easier to read since you have only one spot where the majority of these values can appear.

              • 4. Re: To scope or not to scope (domain.xml)
                starksm64

                My feeling is that a given piece of metadata only has one location in the model. If resources are something we want to define globally, then we do that and allow for references from the server and other levels that can include resources.

                 

                I can also see that we want something like templates(like a datasource template) that define a number of common properties/metadata and the metadata (datasource resource) would reference the template and only specify the properties required for that particular instance.

                • 5. Re: To scope or not to scope (domain.xml)
                  brian.stansberry

                  I agree that a given piece of metadata only existing one place in the model should be the pattern. The template idea makes that relatively painless.

                   

                  An exception I could see to that would be an element that represents a more general notion of a system property. Formalize the system property substition concept with an element that represents a variable that could be referenced elsewhere.

                   

                  As to the homogeneous group notion, my thoughts on this were to have a "profile" notion that would encapsulate most of the common configuration. Then clusters, groups or standalone servers would declare what profile they run. This is copied from something I whipped up on a plane to show people in recent clustering meetings, so take it as an illustration only:

                   

                  <domain>
                     <profile name=”jee”>.... define a set of capabilities, configuration thereof</profile>
                  
                     <profile name=”datagrid”>.... define a different set of capabilities</profile>
                  
                     <profile name=”XYZ">.... define a third set of capabilities</profile>
                  
                     <cluster name=”JEECluster” mcast_addr=”234.5.6.7” default_stack=”udp” profile=”jee”/>
                  
                     <cluster name=”DataGrid” mcast_addr=”235.6.7.8” default_stack=”udp” profile=”datagrid”/>
                  
                     <server-group name="SomeGroup">
                        <server-group-property name="foo">bar</server-group-property>
                     </server-group>
                  
                     <server name=”jee1” bind_address=”192.168.0.100”>
                         <cluster-ref name=”JEECluster” cluster_address=”10.0.0.100”/>
                     </server>
                  
                     <server name=”jee2” bind_address=”192.168.0.101”>
                        <cluster-ref name=”JEECluster” cluster_address=”10.0.0.101”/>
                     </server>
                  
                  
                     <server name=”grid1” bind_address=”192.168.0.102”>
                         <cluster-ref name=”JEECluster” cluster_address=”10.0.0.102”/>
                     </server>
                  
                     <server name=”grid2” bind_address=”192.168.0.103”>
                        <cluster-ref name=”JEECluster” cluster_address=”10.0.0.103”/>
                     </server>
                  
                     <server name=”somegroupA” bind_address=”192.168.0.104”>
                        <server-group-ref name=”SomeGroup”/>
                     </server> 
                  
                     <server name=”somegroupB” bind_address=”192.168.0.105”>          <server-group-ref name=”SomeGroup”/>
                  
                     </server>
                     
                  
                     <server name=”standalone” bind_address=”192.168.0.106”>
                        <profile-ref name=”XYZ”/> <!-- Standalone server -->
                     </server>
                  
                  
                  
                  
                  </domain>

                   

                  A "cluster" element defines some clustering-related configuration. A server-group is a general group but not a cluster. Not sure exactly what that would be; but I threw it in, as Rich Sharples mentioned it had proved useful.  Then a standalone server.

                   

                  Jason, I'm not sure what a "clustered-service" is.

                  • 6. Re: To scope or not to scope (domain.xml)
                    emuckenhuber

                    Hmm wouldn't the example you posted rather be a multi-domain configuration usage?

                     

                    I mean you could not deploy a deployment to a domain - rather to a subset of servers or profiles. Although all servers share a common configuration, they might not have all services active required by the deployment. So it depends more on the server configuration, than the domain.

                    • 7. Re: To scope or not to scope (domain.xml)
                      brian.stansberry

                      Excellent question!  We need to agree on the definition of some terms.

                       

                      To me a "domain" is a set of servers meant to be managed as a unit (acknowledgment: how things are "managed as a unit" is vague/fuzzy; a key task is to clarify exactly what that means.) The servers don't need to all have a homogenous profile, as people may wish to tier things. (In example above written for the clustering meeting the "DataGrid" profile was meant to describe a bunch of servers running Infinispan to serve as large scale grid storage behind a JEE tier.)

                       

                      I see a domain as needing to allow multiple clusters, even ignoring the different tier idea. Mutliple clusters are a useful mechanism for a rolling upgrade; you subdivide your overall capacity into N smaller clusters, and then can upgrade those by taking a cluster at a time off line. This is needed instead of a rolling upgrade of 1 server at a time within the same cluster if the new version of the app can't co-exist with the old in the same cluster (e.g. incompatible state).

                       

                      <domain>
                         <profile name=”jee”>.... define a set of capabilities, configuration thereof</profile>
                      
                         <cluster name=”jee-1” mcast_addr=”234.5.6.7” default_stack=”udp” profile=”jee”/>
                      
                         <cluster name=”jee-2” mcast_addr=”235.6.7.8” default_stack=”udp” profile=”jee”/>
                      
                         <server name=”jee-1-1” bind_address=”192.168.0.100”>
                             <cluster-ref name=”jee-1” cluster_address=”10.0.0.100”/>
                         </server>
                      
                         <server name=”jee-1-2” bind_address=”192.168.0.101”>
                            <cluster-ref name=”jee-1” cluster_address=”10.0.0.101”/>
                         </server>
                      
                         <server name=”jee-2-1” bind_address=”192.168.0.102”>
                             <cluster-ref name=”jee-2” cluster_address=”10.0.0.102”/>
                         </server>
                      
                         <server name=”jee-2-1” bind_address=”192.168.0.103”>
                            <cluster-ref name=”jee-2” cluster_address=”10.0.0.103”/>
                         </server>
                      
                      </domain>

                       

                      Re: deployment, I don't see deploying to a domain. You'd deploy to a cluster, a server-group (if we end up with such a thing) or a standalone server.

                       

                      The "profile" element I showed is problematic though, since if management actions are occuring against a cluster/server-group/standalone server, how are changes written back? Perhaps what I showed becomes a "proifile-template" and then the smaller groups declare a "profile" that extends the template.

                      • 8. Re: To scope or not to scope (domain.xml)
                        jason.greene

                        Brian Stansberry wrote:

                         

                        Excellent question!  We need to agree on the definition of some terms.

                         

                        To me a "domain" is a set of servers meant to be managed as a unit (acknowledgment: how things are "managed as a unit" is vague/fuzzy; a key task is to clarify exactly what that means.) The servers don't need to all have a homogenous profile, as people may wish to tier things. (In example above written for the clustering meeting the "DataGrid" profile was meant to describe a bunch of servers running Infinispan to serve as large scale grid storage behind a JEE tier.)

                         

                        I see a domain as needing to allow multiple clusters, even ignoring the different tier idea. Mutliple clusters are a useful mechanism for a rolling upgrade; you subdivide your overall capacity into N smaller clusters, and then can upgrade those by taking a cluster at a time off line. This is needed instead of a rolling upgrade of 1 server at a time within the same cluster if the new version of the app can't co-exist with the old in the same cluster (e.g. incompatible state).

                         

                        I have a slightly simpler definition, but is essentially the same as what you are describing.  I used it in req 1:

                        "A domain is a management policy that applies to one or more servers/nodes, which may or may not be part of a cluster"

                         

                        In other words, the only thing that every member of the domain has in common is the fact that they have the same domain.xml file.  A better definition would probably be. "A domain is a management policy that applies to one or more servers/nodes, which may or may not be part of a homogenous group". When you think about it, a server-group, and a cluster are really the same thing, the only difference is that the cluster has an additional set of services that a basic group does not. This was what I meant by the cluster-service tag, a way to specify how a group would cluster a particular service (maybe not all services should be clustered). Although it was just something I through out there as an example. A better way likely exists

                        • 9. Re: To scope or not to scope (domain.xml)
                          starksm64

                          I agree with those definitions. By the way, we need someone from the JON team to chime in on some of these discussions to ensure we are in sync with them.

                           

                          I don't see that a true multi-domain notion is in scope for this effort.

                           

                          In the base usage you are deploying to the domain, otherwise I have to have a cluster even if I just have a collection of non-clustered servers. I would think we do have profiles at the domain, cluster server group and server levels.

                           

                          We need to keep a clear separation between configuration and provisioning.

                          • 10. Re: To scope or not to scope (domain.xml)
                            jason.greene

                            For another point of reference, I noticed that the Glassfish was attempting to move away from a reference based model, to more of a scoping one. This is described in a V3Specification doc available here.

                             

                            It appears however, that this was not completely adopted, no idea why.

                             

                            In this proposal they nest server definitions as part of a group or a cluster. I think this would actually make the file harder to read.

                             

                            Another unique aspect of this specification is that they made groups heterogenous and clusters homogenous. This seems strange to me since I view clustering as more of a service.

                            • 11. Re: To scope or not to scope (domain.xml)
                              starksm64

                              Right, that is why we really need some input from the JON team as we need to be in sync with the administrator perspective. It has been our tendency (I'm being kind to us ) to not properly reflect their view of dealing with server and domain configuration. I'll ping Charles on the current two threads in play.

                              • 12. Re: To scope or not to scope (domain.xml)
                                brian.stansberry

                                Scott Stark wrote:

                                 

                                I agree with those definitions. By the way, we need someone from the JON team to chime in on some of these discussions to ensure we are in sync with them.

                                 

                                I've pinged Charles to let him know about these threads.

                                I don't see that a true multi-domain notion is in scope for this effort.

                                 

                                Since we're in the mode of defining things, what is a "multi-domain notion?"  I see it as a an ability to manage within the same tool multiple sets of servers where each set has it's own management policy. The sets of servers are conceptually distinct (hence the separate management policy / domain) but, generally for operational ease of use reasons, users wish to manage multiple domains from the same tool.

                                In the base usage you are deploying to the domain, otherwise I have to have a cluster even if I just have a collection of non-clustered servers. I would think we do have profiles at the domain, cluster server group and server levels.

                                 

                                The "server-group" notion is the collection of unclustered servers. So in the simple case of a domain that's a collection of a homogenous set of unclustered servers, we'd just be requiring a bit more XML.

                                 

                                I can see having a capability to deploy to a domain as a convenience, but it can get conceptually messy.

                                We need to keep a clear separation between configuration and provisioning.

                                 

                                Agreed, but not sure what you were getting at here. Were we mixing them?

                                • 13. Re: To scope or not to scope (domain.xml)
                                  starksm64
                                  Brian Stansberry wrote:

                                  Since we're in the mode of defining things, what is a "multi-domain notion?"  I see it as a an ability to manage within the same tool multiple sets of servers where each set has it's own management policy. The sets of servers are conceptually distinct (hence the separate management policy / domain) but, generally for operational ease of use reasons, users wish to manage multiple domains from the same tool.
                                  Right, which is why I don't see this as impacting what the domain.xml looks like. This notion exists above the servers and profileservice. We are told what domain we are operating in.

                                  The "server-group" notion is the collection of unclustered servers. So in the simple case of a domain that's a collection of a homogenous set of unclustered servers, we'd just be requiring a bit more XML.

                                   

                                  I can see having a capability to deploy to a domain as a convenience, but it can get conceptually messy.

                                  We need to keep a clear separation between configuration and provisioning.

                                   

                                  Agreed, but not sure what you were getting at here. Were we mixing them?

                                  I see deploying to a domain as both a configuration of a profile at the domain level, and a runtime operation of pushing content out to all servers. I see it can be messy if the deployment impacts clustered services. If we have any hope of getting the domain.xml function in AS6, we need to separate getting a server to be driven by a domain.xml from how the provisioning would happen.

                                  • 14. Re: To scope or not to scope (domain.xml)
                                    brian.stansberry

                                    Jason Greene wrote:

                                     

                                    When you think about it, a server-group, and a cluster are really the same thing, the only difference is that the cluster has an additional set of services that a basic group does not.

                                     

                                    Yes, but isn't the set of capabilities that a server has defined by its profile? We could say that defining a server as part of a cluster means that additional capabilities are added beyond what's in the profile. But then we are dividing the definition of a profile into two places. Also, exactly what capabilities should be added if a server is in a cluster is unclear.

                                     

                                    A very simple concept of a cluster is that it is a server-group that also exposes some predefined configuration properties that need to be kept consistent throughout the group. (A server-group would not have such properties by default, although users could add custom ones.) Things such as:

                                     

                                    • Group name (current -g)
                                    • Type of intra-group communication to use by default (udp, tcp)
                                    • Default interface to use for intra-group communication
                                    • Default multicast address to use
                                    • other defaults less commonly changed

                                     

                                    A cluster might also have some management capabilities besides what is available to a server-group, although I'm not sure what. It should be possible for example to perform a (rolling) redeployment to a server-group.

                                     

                                    A separate discussion is how communication will occur between servers who are not members of a cluster. There needs to be a group communication mechanism that spans the entire domain.

                                    This was what I meant by the cluster-service tag, a way to specify how a group would cluster a particular service (maybe not all services should be clustered). Although it was just something I through out there as an example. A better way likely exists

                                     

                                    Currently how we cluster a particular service is defined in that service's configuration, it's not something applied externally. And the details of what needs to be configured are often disparate between different services, so I'm not sure those configs can profitably be externalized into some common configuration element.

                                     

                                    Ah, or are you talking about "clustered-service" as a wrapper to describe how to manage the service, e.g. how to (re)deploy across cluster?

                                    1 2 3 Previous Next