1 2 Previous Next 22 Replies Latest reply on Mar 3, 2009 10:41 AM by starksm64

    Clustered DeploymentRepository for 5.1

    brian.stansberry

      I'll be posting here some FYI stuff re: what I'm doing to get a clustered impl of a DeploymentRepository working, specifically for 5.1.

      There's been a lot of demand for a Farm service replacement, so my 5.1 target is to restore the equivalent functionality. In simplified terms that boils down to:

      1) In DeploymentRepository.load() reconcile the local repo with the cluster. Equivalent to what the old FarmMemberService did in startService().
      2) In DeploymentRepository.getModifiedDeployments() detect changes in the local repo and push them out to the cluster before returning. Equivalent to what the old FarmMemberService did during the periodic scan() calls.

      The former is generally useful even without the latter. E.g. imagine users wanted Jopr and the DeploymentManager to handle pushing changes out to the cluster. So #2 wouldn't be wanted. But #1 is still useful to allow a node that's been offline to sync up with the cluster when it starts.

      The fundamental thing that needs to be there for this to work is an intra-cluster communication facility available to the clustered DeploymentRepository. Some day an advanced clustered ProfileService could have a JGroups channel available to it as part of bootstrap, but for the 5.1 requirements I don't need or want to go that far. Instead I'm looking to deal with this by using subprofile dependencies. Basically a "farm" subprofile depends on the subprofile that includes "deploy". Result is that by the time load() is called on a ClusteredDeploymentRepository, the "deploy" subprofile has been installed, so the HAPartition is available in the runtime. The ClusteredDeploymentRepository then uses a service locator class to resolve the HAPartition.

      I've got that working on my trunk checkout. Following is how it works in a static setup, which I believe is what we are targetting for 5.1. The equivalent should work fine with the XML based stuff Emanuel's doing as well; just have to ensure the "farm" subprofile is installed after whatever subprofile deploys the HAPartition.

      public class StaticClusteredProfileFactory extends StaticProfileFactory
      {
      
       ....
      
       @Override
       protected void createProfileMetaData(ProfileKey rootKey, URL url) throws Exception
       {
       if(rootKey == null)
       throw new IllegalArgumentException("Null root profile key.");
      
       // Create bootstrap profile meta data
       ProfileKey bootstrapKey = new ProfileKey(BOOTSTRAP_NAME);
       ProfileMetaData bootstrap = createProfileMetaData(
       BOOTSTRAP_NAME, false, new URI[] { getBootstrapURI() }, new String[0]);
       addProfile(bootstrapKey, bootstrap);
      
       // Create deployers profile meta data
       ProfileKey deployersKey = new ProfileKey(DEPLOYERS_NAME);
       ProfileMetaData deployers = createProfileMetaData(
       DEPLOYERS_NAME, false, new URI[] { getDeployersURI() }, new String[] { BOOTSTRAP_NAME });
       addProfile(deployersKey, deployers);
      
       // Create applications profile;
       ProfileKey applicationsKey = new ProfileKey(APPLICATIONS_NAME);
       URI[] deployURIs = getApplicationURIs().toArray(new URI[getApplicationURIs().size()]);
       String[] applicationSubProfiles = new String[] { BOOTSTRAP_NAME, DEPLOYERS_NAME };
       ProfileMetaData applications = createProfileMetaData(
       APPLICATIONS_NAME, true, deployURIs, applicationSubProfiles);
       addProfile(applicationsKey, applications);
      
       ProfileMetaData farm = null;
       if (getFarmURIs() != null)
       {
       ProfileKey farmKey = new ProfileKey(FARM_NAME);
       URI[] farmURIs = getFarmURIs().toArray(new URI[getFarmURIs().size()]);
       String[] farmSubProfiles = new String[] { APPLICATIONS_NAME };
       farm = createClusteredProfileMetaData(
       FARM_NAME, true, farmURIs, farmSubProfiles, getPartitionName());
       addProfile(farmKey, farm);
       }
      
       String[] rootSubProfiles = farm == null ? new String[]{APPLICATIONS_NAME}
       : new String[] { FARM_NAME };
       ProfileMetaData root = createProfileMetaData(
       rootKey.getName(), true, new URI[0], rootSubProfiles);
       addProfile(rootKey, root);
       }
      
       protected ProfileSourceMetaData createClusteredSource(URI[] uris, boolean hotDeployment)
       {
       ClusteredProfileSourceMetaData source = null;
       if(hotDeployment)
       {
       source = new HotDeploymentClusteredProfileSourceMetaData();
       }
       else
       {
       source = new ImmutableClusteredProfileSourceMetaData();
       }
      
       source.setPartitionName(getPartitionName());
      
       List<String> sources = new ArrayList<String>();
       for(URI uri : uris)
       sources.add(uri.toString());
       source.setSources(sources);
       return source;
       }
      
       ...
      }


      Now that I know the dependency issues can be handled easily enough I'll shift back to focusing on the internal details of keeping the local repositories in sync.

        • 1. Re: Clustered DeploymentRepository for 5.1
          emuckenhuber

          Hmm is the DeploymentManager enough for you then?
          just wondering what we should do with https://jira.jboss.org/jira/browse/JBAS-5682

          IMO profile.addDeployment should not automatically deploy the deployment.
          We could consider it on modifiedDeployments, but then you don't have any control about the lifecycle.
          It will just get deployed when HDScanner runs the next time, which does not really makes sense to me.

          • 2. Re: Clustered DeploymentRepository for 5.1
            brian.stansberry

             

            "emuckenhuber" wrote:
            Hmm is the DeploymentManager enough for you then?


            Not sure about the overall scope of this question, so I'll reply separately. ;)


            just wondering what we should do with https://jira.jboss.org/jira/browse/JBAS-5682

            IMO profile.addDeployment should not automatically deploy the deployment.
            We could consider it on modifiedDeployments, but then you don't have any control about the lifecycle.
            It will just get deployed when HDScanner runs the next time, which does not really makes sense to me.


            Yes, re-reading the JBAS-5682 forum thread (http://www.jboss.org/index.html?module=bb&op=viewtopic&t=138120) I don't see why anything needs to be done. I agree that adding a deployment to the profile and telling the server to deploy it are distinct things and should be controlled by the client that calls the DeploymentManager, and not be made random by the HDScanner thread. The 5.0.x HASingletonDeploymentScanner works fine that way, by calling DeploymentManager.distribute(name, phase, url, true) and then DeploymentManager.start().

            In terms of the SPI, things are a bit confusing to me. Or perhaps I should say in terms of the implementation. Specifically, DeploymentManagerImpl.distribute(...). If you pass "true" as the copyContent param, on the server side DeployHandler.handleStream() gets called, which call DeploymentRepository.addDeploymentContent(). Simple enough. The client then calls DeploymentManager.start() which calls DeployHandler.start() which passes the deployment to the MainDeployer.

            It's more confused if you pass "false" to the copyContent param. In that case DeployHandler.distribute() gets called, which calls DeploymentRepository.addDeployment() and then passes the deployment to the MainDeployer. A call to DeploymentManager.start() then seems redundant.

            Passing a deployment to MainDeployer in two different places seems wrong.

            • 3. Re: Clustered DeploymentRepository for 5.1
              brian.stansberry

               

              "bstansberry@jboss.com" wrote:
              "emuckenhuber" wrote:
              Hmm is the DeploymentManager enough for you then?


              Not sure about the overall scope of this question, so I'll reply separately. ;)


              OK, now a bit more on the DeploymentManager in general and how it relates to the clustered deployment repository I'm working to get into 5.1. Basically my goal for 5.1 was to restore an equivalent to the old "Farm" feature where a local-filesystem based repository is aware of changes in content and replicates the changed content around the cluster. Primary target is people updating the repository content by copying a new version of a deployment into a directory.

              The DeploymentRepository.addDeploymentContent() method will also trigger replication around the cluster. So, if a DeploymentManager client loaded the "farm" subprofile and then called distribute(name, phase, url, true) the net result would be the content would be replicated to all nodes in the cluster. But, that wouldn't result in it getting deployed in all nodes, since that's a separate step. Getting it deployed in all nodes would require a clustered DeploymentManager impl, which isn't in the cards for 5.1. So, kind of half-way there.

              Bottom line, I think a tool written against the 5.1 code base would be better off independently working with each of the nodes in the cluster; i.e. let the tool coordinate getting the content distributed to and deployed on all nodes. Let the clustered deployment repository be the replacement that people have requested for the non-tooling based Farm service.

              • 4. Re: Clustered DeploymentRepository for 5.1
                emuckenhuber

                 

                "bstansberry@jboss.com" wrote:

                In terms of the SPI, things are a bit confusing to me. Or perhaps I should say in terms of the implementation. Specifically, DeploymentManagerImpl.distribute(...). If you pass "true" as the copyContent param, on the server side DeployHandler.handleStream() gets called, which call DeploymentRepository.addDeploymentContent(). Simple enough. The client then calls DeploymentManager.start() which calls DeployHandler.start() which passes the deployment to the MainDeployer.


                Yes the spi is a bit confusing, i've also posted about that here: http://www.jboss.org/index.html?module=bb&op=viewtopic&t=149569
                I'll see if we can do that for 5.1 and deprecate the old methods.

                • 5. Re: Clustered DeploymentRepository for 5.1
                  brian.stansberry

                  Thanks for the link. :) I knew there was a thread about this stuff, but yesterday the forum was taking 2 mins to reply to every click so I just gave up looking and posted.

                  • 6. Re: Clustered DeploymentRepository for 5.1
                    brian.stansberry

                    A detail in my original post that I didn't emphasize enough and that might have significant implications: there's a difference in URIs associated with the root profile metadata in my StaticClusteredProfileFactory above and the StaticProfileFactory currently in trunk.

                    In StaticProfileFactory, the profile metadata URIs point to the application URIs, e.g. the deploy/ dir:

                     protected void createProfileMetaData(ProfileKey rootKey, URL url) throws Exception
                     {
                     ......
                    
                     // Create root profile;
                     URI[] rootURIs = applicationURIs.toArray(new URI[applicationURIs.size()]);
                     String[] rootSubProfiles = new String[] { BOOTSTRAP_NAME, DEPLOYERS_NAME };
                     ProfileMetaData root = createProfileMetaData(
                     rootKey.getName(), true, rootURIs, rootSubProfiles);
                     // Add to profile map
                     addProfile(rootKey, root);
                     }
                    


                    In StaticClusteredProfileFactory above, the application URIs are associated with an APPLICATION subprofile; the root profile itself doesn't have any URIs, just subprofiles. I think this approach is more consistent with how it will work when we shift to something like XmlProfileFactory.

                    My concern is that the StaticClusteredProfileFactory approach can break DeploymentManager or ManagementView in AS 5.1, i.e. if those implementations will be counting on an implementation detail that they can load, for example, the "all" profile and then copy content to it's URIs. They would need to access the "all" profiles APPLICATION subprofile and copy content there.

                    • 7. Re: Clustered DeploymentRepository for 5.1
                      emuckenhuber

                      ManagementView is handling all active profiles, so that should be fine.

                      But i'm not really sure about the DeploymentManager. I mean it should be possible to define to which profile you want to add your content.
                      Although we might need to drop the loadProfile and releaseProfile, as this does not seem to be thread safe.
                      It might make more sense to have just:

                       deploymentMgr.distribute(String name, URL url, boolean copyContent);
                       // and
                       deploymentMgr.distribute(String name, URL url, ProfileKey key);
                      


                      Whereas the 2nd way would imply copyContent == true. As transient deployments are handled by a own profile now.

                      So we might want to have load() where it creates a mapping between a deployment and the profile. So you can stop and remove all deployments easily. And for distribute and start we would need to cache the DeploymentID with it's profile. And those stuff will get cleaned when you do release().

                      There is a sort of defaultProfileKey notion - so when you do e.g. profileService.getActiveProfile(new ProfileKey(null)); it would return the profile associated with ProfileKey($jboss.server.name}).
                      So distribute(String name, URL url, copyContent = true) would add it to the injected default profile.

                      Whereas we might also want to differentiate between a normal Profile and a MutableProfile - where you can add and remove content.

                      • 8. Re: Clustered DeploymentRepository for 5.1
                        brian.stansberry

                         

                        "emuckenhuber" wrote:

                        There is a sort of defaultProfileKey notion - so when you do e.g. profileService.getActiveProfile(new ProfileKey(null)); it would return the profile associated with ProfileKey($jboss.server.name}).
                        So distribute(String name, URL url, copyContent = true) would add it to the injected default profile.


                        So with the kind of setup I'm talking about, having the default profile be the one associated with ProfileKey($jboss.server.name}) would not work, since it will have no URIs associated with it.

                        I think the key question for 5.1 is how embedded-console/jopr are using the DeploymentManager -- what ProfileKey are the passing to loadProfile (or to your new distribute(String name, URL url, ProfileKey key) method.

                        • 9. Re: Clustered DeploymentRepository for 5.1
                          emuckenhuber

                           

                          "bstansberry@jboss.com" wrote:

                          So with the kind of setup I'm talking about, having the default profile be the one associated with ProfileKey($jboss.server.name}) would not work, since it will have no URIs associated with it.

                          I think the key question for 5.1 is how embedded-console/jopr are using the DeploymentManager -- what ProfileKey are the passing to loadProfile (or to your new distribute(String name, URL url, ProfileKey key) method.


                          Well we can create a additional application (deploy/) profile by default, as it's more likely to be similar with the XmlProfileFactory - and would match what you are doing.

                          Furthermore we could add a getProfiles() to the DeploymentManager (maybe with the notion of mutable profiles) then jopr could provide a dropdown menu to select where you actually want to upload your content to. As the profiles from ProfileService are not necessarily 'mutable' profiles.
                          And we can also make those names configurable over the profile.xml.

                          • 10. Re: Clustered DeploymentRepository for 5.1
                            brian.stansberry

                            Sounds good.

                            • 11. Re: Clustered DeploymentRepository for 5.1
                              emuckenhuber

                              I introduced a "applications" profile in trunk. But there is still one change to DeploymentManager outstanding.
                              Basically concerning loadProfile(ProfileKey, boolean allowHotDeployments).
                              I think the client should not be able to specify allowHotDeployments. Therefore the profileKey should be enough.

                              Furthermore there is now a defaultApplicationsKey - specifying the default profile where to deploy applications.
                              This would mean that in case you do loadProfile(new ProfileKey(ProfileKey.DEFAULT) - it will use this profile.

                              I've also introduced a MutableProfile, but it's basically used by default everywhere - i still need to implement that better. But this will happen most likely after beta1.

                              So what i would like to do is that only for MutableProfiles a deploymenRepository is required (because of the DeploymentManager). For an immutable profile this is therefore not needed. So we would also have some different implementations of a Profile then.
                              But i'm not really sure how this would affect your work?

                              • 12. Re: Clustered DeploymentRepository for 5.1
                                emuckenhuber

                                I've put an example (untested) of an ImmutableProfile in org.jboss.system.server.profile.repository in case you are interested...
                                So i basically created basic abstraction on top of the deploymentRepository, so that it can be used by those two.

                                • 13. Re: Clustered DeploymentRepository for 5.1
                                  brian.stansberry

                                   

                                  "emuckenhuber" wrote:
                                  I've put an example (untested) of an ImmutableProfile in org.jboss.system.server.profile.repository in case you are interested...
                                  So i basically created basic abstraction on top of the deploymentRepository, so that it can be used by those two.


                                  I'm not seeing it???

                                  So what i would like to do is that only for MutableProfiles a deploymenRepository is required (because of the DeploymentManager). For an immutable profile this is therefore not needed. So we would also have some different implementations of a Profile then.
                                  But i'm not really sure how this would affect your work?


                                  Could be trouble. What "mutable" means is still pretty unclear. But a twist a clustered repository is adding is the fact that a profile that doesn't change after the server starts can still change as part of the startup process. That is, a node that's been offline synchronizes its content with the cluster as part of startup.

                                  • 14. Re: Clustered DeploymentRepository for 5.1
                                    emuckenhuber

                                     

                                    "bstansberry@jboss.com" wrote:
                                    "emuckenhuber" wrote:
                                    I've put an example (untested) of an ImmutableProfile in org.jboss.system.server.profile.repository in case you are interested...
                                    So i basically created basic abstraction on top of the deploymentRepository, so that it can be used by those two.


                                    I'm not seeing it???

                                    https://svn.jboss.org/repos/jbossas/trunk/system/src/main/org/jboss/system/server/profile/repository/AbstractImmutableProfile.java

                                    "bstansberry@jboss.com" wrote:

                                    So what i would like to do is that only for MutableProfiles a deploymenRepository is required (because of the DeploymentManager). For an immutable profile this is therefore not needed. So we would also have some different implementations of a Profile then.
                                    But i'm not really sure how this would affect your work?


                                    Could be trouble. What "mutable" means is still pretty unclear. But a twist a clustered repository is adding is the fact that a profile that doesn't change after the server starts can still change as part of the startup process. That is, a node that's been offline synchronizes its content with the cluster as part of startup.


                                    True, mutable is not really clear. But basically it's the ability to add and remove contents.
                                    Furthermore it implies that hot-deployment could be possible.

                                    But i don't think a deploymentRepository is required for all profiles (e.g. the bootstrap profile, or in future most of the xml defined profiles).
                                    Also the root profile basically just has some dependencies.
                                    Most probably the ImmutableDeploymentRepository should be renamed to NoHotDeploymentRepository and the MutableRepository to HotDeploymentRepository :)
                                    Both should somehow support addDeploymentContent, as you might want to add a deployer - although there is no hot-deployment scanning.

                                    I mean in the end it's then depending on your implementation what it should support and if it uses a deploymentRepository.
                                    Just that the isMutable() is used by the DeploymentManager to determine which profiles are a possible target to deploy content to.
                                    But i don't think i have time to do that for 'Beta1', but most probably for CR1.


                                    1 2 Previous Next