1 2 Previous Next 25 Replies Latest reply on Jun 4, 2008 1:43 PM by Scott Stark

    Controlling deployments via a barrier bean

    Brian Stansberry Master

      I've been looking at better ways to do HASingleton deployments vs the current (broken) scanner-based approach (http://jira.jboss.com/jira/browse/JBAS-5563).

      I've long thought that a better way to do this would be by adding a deployer that adds a dependency on "barrier" bean to deployments that have appropriate metadata. Similar to the JMX-based jboss.ha:service=HASingletonDeployer,type=BarrierController concept, but with an MC bean.

      An example set of deployers would be:

      <!-- For deployments that have HASingletonMetaData attached,
       establishes a dependency on a barrier bean that is only
       satisfied when this node becomes the singleton master
       -->
       <bean name="HASingletonBarrierDependencyDeployer"
       class="org.jboss.ha.singleton.deployer.HASingletonBarrierDependencyDeployer">
      
       <property name="dependencyNamesByPartition">
       <map keyClass="java.lang.String" valueClass="java.lang.String">
       <entry>
       <key>${jboss.partition.name:DefaultPartition}</key>
       <value>HASingletonBarrier</value>
       </entry>
       <!-- Add more entries if you deploy more HAPartitions -->
       </map>
       </property>
      
       </bean>
      
       <!--
       Adds HASingletonMetaData to deployments that come from the specified
       URLs. Provides support for the legacy deploy-hasingleton directory
       concept.
       -->
       <bean name="HASingletonMetadataDeployer"
       class="org.jboss.ha.singleton.deployer.VFSHASingletonMetadataDeployer">
      
       <property name="partitionName">${jboss.partition.name:DefaultPartition}</property>
      
       <property name="URIList">
       <list elementClass="java.net.URI">
       <!-- Standard HASingleton deployment directory -->
       <value>${jboss.server.home.url}deploy-hasingleton/</value>
       </list>
       </property>
       </bean>


      In deploy we'd have these beans:

       <!-- Bean on which HASingletons will depend. ControllerMode is MANUAL,
       so will not deploy until HASingletonBarrierInstaller installs it.
       -->
       <bean name="HASingletonBarrier" class="java.lang.Object" mode="MANUAL"/>
      
       <!-- Tracks cluster topology and tells the MainDeployer to bring the
       HASingletonBarrier to INSTALLED when it becomes singleton master, and
       to bring it to NOT_INSTALLED when it is no longer the master.
       -->
       <bean name="HASingletonBarrierInstaller"
       class="org.jboss.ha.singleton.deployer.HASingletonBarrierInstaller">
      
       <property name="barrierBeanName">HASingletonBarrier</property>
       <property name="mainDeployer"><inject bean="MainDeployer"/></property>
       <property name="HAPartition"><inject bean="HAPartition"/></property>
      
       </bean>


      That's the concept at least. One issue I see is on the non-singleton-master nodes the MainDeployer.checkComplete() impl is going to report an unsatisfied dependency, which will get logged at ERROR. Not good.

      Any comments on this basic approach to controlling deployments? If it seems reasonable, can we add a mechanism (perhaps a flag in DependencyItem) such that certain unresolved dependencies are not logged as an error?

      (Note it's possible this will just work w/o the logging. I'm working through other issues and haven't gotten it to work right yet. But looking at the code it seems clear there will be ERROR logging.)

        • 1. Re: Controlling deployments via a barrier bean
          Scott Stark Master

          I don't think a dependency is the way this should be handled as its not an optional dependency. Your using a dependency to try to identify the singleton bean in the cluster. Really we don't want the unelected singleton beans to be exposed as available for installation as runtime components.

          To properly handle this we would need the ability to expand the states for the component in question to allow for alternate DESCRIBE, DESCRIBE_CLUSTERED, DESCRIBE_SINGLETON states depending on the cluster metadata associated with the bean. Currently the mc state machine does not allow for this choice type of state transition. I have not worked much with the lower level state manipulation so I could also be talking out of my ass, but I think this is the proper way to do it.

          Given I don't believe there is proper mc support for this, how to do it? I'm not sure. Any idea I have would mix deployer and kernel bean state management behavior. Need some feedback from Adrian/Ales.

          • 2. Re: Controlling deployments via a barrier bean
            Adrian Brock Master

            I think as far as backwards compatibility goes Brian's approach is roughly correct,
            but it should NOT be using the MainDeployer directly.

            It should be based on a (sub-)profile.

            If you look in broad brush terms, the profile service is made up of two components

            1) A client side api that lets you define what a profile is
            2) A server side that implements that profile at runtime

            For the current default profile (hot deployment) that means three pojos that know how
            to scan three sets of directories/urls (bootstrap/deployers/deploy)
            which are the three sub-profiles that make up the profile.

            There should be a way to add extra (sub-)profiles e.g. hasingleton, jbossmq, etc.
            that could be based on normal, singleton or farm semantics.
            These are just implementation details of how the server side profile is implementated.

            In practice, these subprofiles are not implemented in such a flexible way
            with the scanners being hardwired along with the subprofiles they represent
            and a subprofile is always active.

            In terms of where we want to get to in the profile service and metadata in
            general, it should be possible using a management console to say
            I want this sub-profile, deployment, service, etc. to be a singleton in a given
            partition.

            I'd say partition is too low level since what you really want is some election policy
            which could be generalised beyond the hasingleton notion to some activation policy.
            e.g. you activate a deployment based on a timer rather than clustered failover election

            By taking a metadata approach, users can also programme this explicitly
            without having to use the management console directly.
            e.g. I add a META-INF/jboss-activation.xml to describe how/when I want
            this deployment starting/stopping

            At the service/pojo level this would probably be an annotation

            <bean name="Whatever" ...>
             <annotation>@Singleton policy="DefaultPartitionSingleton"<annotation/>
            ...
            

            Which could also be applied via the scoped metadata if you want to
            externalise it from the main descriptors in the management console/profile service case.


            Short term for now, I'd just go with extending the hot deployment profile service to allow
            some kind of barrier service like Brian describes to add/remove a sub-profile dynamically.

            I'm not sure it needs to be as complicated as Brian describes though?
            i.e. messing around with dependencies
            The hasingleton scanner should just respond to the elected/de-elected event
            by adding/removing its profile to/from the profile service.

            • 3. Re: Controlling deployments via a barrier bean
              Adrian Brock Master

               

              "adrian@jboss.org" wrote:

              There should be a way to add extra (sub-)profiles e.g. hasingleton, jbossmq, etc.
              that could be based on normal, singleton or farm semantics.
              These are just implementation details of how the server side profile is implementated.


              As discussed elsewhere, singleton and farming are really orthogonal issues.
              Even in the hot deployment profile, it should be possible to have the
              hasingleton profile take part in farming even if the current member is
              not actually deploying the deployments.



              • 4. Re: Controlling deployments via a barrier bean
                Adrian Brock Master

                OFF TOPIC, but on a related issue.

                I said a while ago that a real profile service implementation
                has a lot in common with JBoss Cache (at least in terms of aspects/technology if
                not in implementation :-).

                It is a (possibly transacted/distributed) tree like structure with an
                abitrary backing store which should be able to scale from standalone to
                clustered environments.

                Similarly it has requirements to track fine grained changes inside the tree
                (changes to the items of metadata - the fields on the managed properties)
                and applying them atomically.

                Some areas where it differs are that the (sub-)profile is versioned as a whole.
                Related to this is that the distribution of the changes has a slightly different
                semantic in that changes to the metadata would first be tried locally
                or on some staging cluster before distributing the new version across the
                production cluster.

                There are other parts that are not related to the cache structure, e.g. retrieving statistics
                or invoking managed operations like flushing a pool, but these have more
                in common with jgroups clustered invocation sematics.

                • 5. Re: Controlling deployments via a barrier bean
                  Adrian Brock Master

                   

                  "adrian@jboss.org" wrote:

                  If you look in broad brush terms, the profile service is made up of two components

                  1) A client side api that lets you define what a profile is
                  2) A server side that implements that profile at runtime

                  For the current default profile (hot deployment) that means three pojos that know how
                  to scan three sets of directories/urls (bootstrap/deployers/deploy)
                  which are the three sub-profiles that make up the profile.


                  The client side api for the default hot deployment profile is of course to
                  just modify the directory structure and the file contents. :-)

                  • 6. Re: Controlling deployments via a barrier bean
                    Ales Justin Master

                     

                    "scott.stark@jboss.org" wrote:

                    To properly handle this we would need the ability to expand the states for the component in question to allow for alternate DESCRIBE, DESCRIBE_CLUSTERED, DESCRIBE_SINGLETON states depending on the cluster metadata associated with the bean. Currently the mc state machine does not allow for this choice type of state transition. I have not worked much with the lower level state manipulation so I could also be talking out of my ass, but I think this is the proper way to do it.

                    This could be achieved.
                    OK, it would still depend on exactly what you wanted to do - no cyclic/fork states.

                    You can do this atm.
                    Add these two new states (DESCRIBE_CLUSTERED, DESCRIBE_SINGLETON) to Controller.
                    Then when you install a bean that would use any of those two states, you need to push him your own ControllerContextActions impl.
                    e.g. our alias handling
                     Map<ControllerState, ControllerContextAction> map = Collections.<ControllerState, ControllerContextAction>singletonMap(ControllerState.INSTALLED, new AliasControllerContextAction());
                     ControllerContextActions actions = new AbstractControllerContextActions(map);
                     install(new AliasControllerContext(alias, original, actions));
                    


                    • 7. Re: Controlling deployments via a barrier bean
                      Brian Stansberry Master

                      Thanks guys, for the input. :) My apologies, I opened this thread a couple hours before leaving on a trip where it turned out I had no chance for e-mail. I'll respond bit by bit as I digest.

                      "adrian@jboss.org" wrote:

                      Short term for now, I'd just go with extending the hot deployment profile service to allow
                      some kind of barrier service like Brian describes to add/remove a sub-profile dynamically.

                      I'm not sure it needs to be as complicated as Brian describes though?
                      i.e. messing around with dependencies
                      The hasingleton scanner should just respond to the elected/de-elected event
                      by adding/removing its profile to/from the profile service.


                      This is essentially what's there now. Discovered last week that something is broken with it, but since it has a few flaws I wanted to focus on a more long-term replacement rather than just a quick fix. The flaws:

                      1) A scanner that populates the profile is a behavior of the basic profile service, but not of the full repository-based profile service. In the full version it's the DeploymentRepository impl itself that scans the filesystem for content.

                      2) The distinction betweens what's in a profile and how it's reflected in the runtime. If the profile service on a non-master node isn't aware at all of a set of deployments, those deployments can't be managed (e.g. some piece of metadata changed), at least not directly on that node.

                      3) Can't handle the fine-grained metadata driven approach you describe in your '@Singleton policy="DefaultPartitionSingleton"' example.

                      Properly dealing with #2 and #3 could possibly be deferred; i.e. use a short term approach for 5.0.0. But AIUI, ability to use the full profile service is a requirement for 5.0.0 though, so I'd need to solve #1. On the farming thread Scott and I agreed last week that a clustered DeploymentRepository would need to have a clustered comm resource (JGroups Channel) available to it, so it's possible its internal scan logic could have the ability to respond to an elected/unelected event and add/remove sub-profile content. But that could end up being a fair amount of work and if the work will just be thrown away when I solve #2 and #3, I'm reluctant to do it that way.

                      I need to better understand what Scott's driving at with the discussion of DESCRIBE, DESCRIBE_CLUSTERED, DESCRIBE_SINGLETON and how that solves the issue where we "(r)eally we don't want the unelected singleton beans to be exposed as available for installation as runtime components." Thanks, Ales for the links to the alias stuff; found where that's used.

                      • 8. Re: Controlling deployments via a barrier bean
                        Adrian Brock Master

                         

                        "bstansberry@jboss.com" wrote:

                        I need to better understand what Scott's driving at with the discussion of DESCRIBE, DESCRIBE_CLUSTERED, DESCRIBE_SINGLETON and how that solves the issue where we "(r)eally we don't want the unelected singleton beans to be exposed as available for installation as runtime components."


                        I'm saying that's the wrong approach. What's required (for the first iteration) is being
                        able to activate/deactivate a sub-profile.
                        That's what your singleton activation policy does (the contract).

                        The implementation of scanning to determine what's in that profile
                        is specific to the current profile implementation.
                        Another possible implementation is to pull that profile from some reference repository
                        (which may or may not be clustered).

                        The definition and implementation of the sub-profile is an orthogonal issue to how
                        and when it gets activated.

                        • 9. Re: Controlling deployments via a barrier bean
                          Brian Stansberry Master

                           

                          "adrian@jboss.org" wrote:
                          I'm saying that's the wrong approach.


                          OK, thanks. I knew you'd said that, but since it was under discussion I figured if I had time I'd figured out what was meant. :-)

                          The definition and implementation of the sub-profile is an orthogonal issue to how and when it gets activated.


                          Yes, it's the concept of content that's available in a profile that isn't active yet that needs to be fleshed out. That's fundamentally why I want to replace the current scanning based approach; it mixes the two.

                          There would probably need to be some API change related to this, in order to allow mgmt tools to activate/inactivate sub-profiles. If we didn't want to allow external clients to do that, it could just be done via an spi that an activation policy would use.

                          AIUI, though, the detection of an @Singleton annotation or some other form of deployment-specific metadata is always done by a deployer; i.e. is not part of the profile service itself. (Waiting to be told I'm wrong ;) ) So you end up with a chicken and egg problem; you need to start deploying to find out a deployment is in an inactive sub-profile. That's fundamentally what drove me to the idea of using dependencies. I'd much prefer an approach where the deployers aren't even aware of content that isn't "activated".

                          • 10. Re: Controlling deployments via a barrier bean
                            Adrian Brock Master

                             

                            "bstansberry@jboss.com" wrote:

                            AIUI, though, the detection of an @Singleton annotation or some other form of deployment-specific metadata is always done by a deployer; i.e. is not part of the profile service itself. (Waiting to be told I'm wrong ;) ) So you end up with a chicken and egg problem; you need to start deploying to find out a deployment is in an inactive sub-profile. That's fundamentally what drove me to the idea of using dependencies. I'd much prefer an approach where the deployers aren't even aware of content that isn't "activated".


                            That's more long term (i.e. a proper implementation ;-).
                            The more fine-grained stuff involves defining some callbacks that are triggered by some policy.

                            e.g. You might have (or it could be annotation based):

                            public interface Activation
                            {
                             void activate();
                             void deactivate();
                            }
                            


                            But it will depend upon what you are doing on how the object gets registred with
                            the policy.

                            The simplest case might be something like;
                            @Activatable(policy="SomeNameInTheMC")
                            public MyPOJO
                            {
                             @Activate
                             public void method1();
                            
                             @Deactivate
                             public void method2();
                            }
                            


                            The registration can then be handled using something similar to the @JMX
                            handling. i.e. you write a lifecycle-install handler
                            that looks for the annotation/interface and does the relevant registrations.
                            (This requires another interface, i.e. ActivationPolicy or Activator)

                            More complicated would be an MDB that you want to start/stop delivery on some
                            kind of schedule.
                            That would involve registering the MDB container (or some activatable extension)
                            with the activation policy at deployment and wait for the callbacks.
                            Of course if the MDB container is just a POJO then it is the same as above.

                            The activation of the sub-profile also be the same thing.
                            The de/activation methods just to talk to the profile service.

                            @Activatable(policy="HASingletonDefaultDomain")
                            public class ProfileActivator
                            {
                             private String subProfileName;
                             @Inject
                             private ProfileService profileService;
                            
                             @Activate
                             public void activate()
                             {
                             profileService.deployProfile(subProfileName);
                             }
                            // etc.
                            

                            We could also get more complicated and with activation requiring multiple policies
                            and logic to decide whether something is active.
                            This probably ties in with what Ales was doing with the JBoss Rules integration in
                            the MC, but that's obviously in the 5% of use cases. ;-)

                            • 11. Re: Controlling deployments via a barrier bean
                              Scott Stark Master

                               

                              "bstansberry@jboss.com" wrote:

                              There would probably need to be some API change related to this, in order to allow mgmt tools to activate/inactivate sub-profiles. If we didn't want to allow external clients to do that, it could just be done via an spi that an activation policy would use.

                              AIUI, though, the detection of an @Singleton annotation or some other form of deployment-specific metadata is always done by a deployer; i.e. is not part of the profile service itself. (Waiting to be told I'm wrong ;) ) So you end up with a chicken and egg problem; you need to start deploying to find out a deployment is in an inactive sub-profile. That's fundamentally what drove me to the idea of using dependencies. I'd much prefer an approach where the deployers aren't even aware of content that isn't "activated".

                              The profile service does not look at deployment metadata, and there is no profile metadata corresponding to a jboss-activation.xml descriptor currently. I don't see how we would go from a bean/component level annotation to a subprofile membership. It seems we are driving at having an additional ActivationPolicyMetaData component for each deployment (maybe even component), but this still leaves a gap between specifying an annotation on a bean and identifying that the bean's deployment has a singleton component.

                              The profile service does not currently deal with makeup of a profile down to the component level. So what do we need for 1)? Its not completely true that:

                              1) A scanner that populates the profile is a behavior of the basic profile service, but not of the full repository-based profile service. In the full version it's the DeploymentRepository impl itself that scans the filesystem for content.

                              in that you could still have an hasingleton scanner adding content to a profile regardless of what profile service implementation is in use. There should be a flag indicating that the deployment should not be a permanent part of the profile though. Is the APPLICATION_CLUSTERED phase going to be useful? Maybe this should be APPLICATION_TRANSIENT or some such to indicate a deployment that will not be remembered across restarts.


                              • 12. Re: Controlling deployments via a barrier bean
                                Scott Stark Master

                                 

                                "adrian@jboss.org" wrote:

                                That's more long term (i.e. a proper implementation ;-).
                                The more fine-grained stuff involves defining some callbacks that are triggered by some policy.

                                e.g. You might have (or it could be annotation based):

                                public interface Activation
                                {
                                 void activate();
                                 void deactivate();
                                }
                                


                                Ok, I see, activation is a runtime behavior driven by metadata that could be talking to the profile service. That makes more sense.


                                • 13. Re: Controlling deployments via a barrier bean
                                  Adrian Brock Master

                                   

                                  "scott.stark@jboss.org" wrote:
                                  Its not completely true that:

                                  1) A scanner that populates the profile is a behavior of the basic profile service, but not of the full repository-based profile service. In the full version it's the DeploymentRepository impl itself that scans the filesystem for content.

                                  in that you could still have an hasingleton scanner adding content to a profile regardless of what profile service implementation is in use. There should be a flag indicating that the deployment should not be a permanent part of the profile though. Is the APPLICATION_CLUSTERED phase going to be useful? Maybe this should be APPLICATION_TRANSIENT or some such to indicate a deployment that will not be remembered across restarts.


                                  If that is the case then you also could say (regardless of clustering) that a hot
                                  deployment scanner could be added to any profile service implementation.

                                  Again this orthogonal.

                                  It's then a hybrid profile service implementation. Where some sub-profiles
                                  come from a versioned repository and others are unversioned in a directory.

                                  Whichever way you look at it, its still a sub-profile of the profile in operation
                                  and needs some "metadata" to say how/if/when it gets activated.

                                  Ideally this should be a part of the management console where somebody can
                                  say this sub-profile/deployment gets activated according to this policy,
                                  where the policy is a part of the profile itself.

                                  As I was trying to explain above, (but got side-tracked in implementation details :-)
                                  for fine-grained it gets harder because the link to the policy might be
                                  annotation on a pojo or a piece of metadata in jboss.xml, etc.

                                  That's why I suggested a jboss-activation.xml to try to make it easier to manage.
                                  The downside is it pushes the responsibility of this new metadata to the subsystems
                                  to read it, since only they know what it means to activate their fine-grained component.

                                  • 14. Re: Controlling deployments via a barrier bean
                                    Scott Stark Master

                                     

                                    "adrian@jboss.org" wrote:

                                    If that is the case then you also could say (regardless of clustering) that a hot deployment scanner could be added to any profile service implementation.

                                    Again this orthogonal.

                                    It's then a hybrid profile service implementation. Where some sub-profiles
                                    come from a versioned repository and others are unversioned in a directory.

                                    Right, but there is a notion of "modified deployments" in the profile spi, and the hot deployment scanner implementation uses this. If we add a APPLICATION_TRANSIENT notion, whether the profile should detect modifications needs to be defined. Currently the profile implementations only look at the APPLICATION phase deployments.

                                    "adrian@jboss.org" wrote:

                                    Whichever way you look at it, its still a sub-profile of the profile in operation
                                    and needs some "metadata" to say how/if/when it gets activated.

                                    Ideally this should be a part of the management console where somebody can say this sub-profile/deployment gets activated according to this policy,
                                    where the policy is a part of the profile itself.

                                    Ok, I'll have to look at how deployment level policies (sub-profiles) could be implemented consistently regardless of the profile service implementation.


                                    1 2 Previous Next