1 2 Previous Next 16 Replies Latest reply on Jun 29, 2007 6:39 AM by Adrian Brock

    New Deployer changes - overview

    Adrian Brock Master

      This is a quick brain dump of all the deployer changes I just committed.

      As usual this is an overview thread, to collect all the information in one place.
      Create separate threads if you want to discuss indivdual topics in more detail.

      First, the changes are not yet complete.

      1) AOP - AOP has a deployer inside its project. That needs updating
      and AOP re-releasing. But since AOP is required for EJB3
      I've temporarily hacked a new version in the "system" project in
      org.jboss.aop.temp.AOPDeployer

      2) WS - Similarly webservices have their own deployers.
      I've just disabled webservices for now. I'll be working on this next
      once I've looked at some other minor loose ends.

      3) EJB3 - I've updated the EJB3 deployers to use attachment flows (see below)
      but I'm not sure if they are correct. Its working on some
      of the basic tests in the ejb3 project tests.
      I suspect there may need to be some attachment flow dependency on the
      security deployer (possibly others?), but I'm not sure where it needs to go.

      4) Security - I think this needs a lot more work to where I would like to get it.

      5) Embedded - I haven't updated this project to the new deployers.
      The version I currently have of this project doesn't build so that needs to be fixed
      before I can do this. Most of the changes should be minor, i.e. updating
      it to use the new "client side" deployment api.

      I'm going to break down these posts into the follow areas:

      * New Project Structure - separation of spi/impl and vfs, etc.
      * VFSDeployer - why this extra abstraction is important
      * Core - a quick discussion on the core project
      * Client - the new "client side" deployment api
      * Structure - the changes to the structure deployments and how the structure is built
      * MainDeployer - the changes to the main deployer and how it relates to the client side api
      * DeploymentContext - how that is nearly invisible now and what needs to be done to complete the process
      * Deployers - the new stages (real dependencies), attachment flow, changes to how components work, making the parsing deployers "more declarative"
      * TODO - other work that still needs doing like integrating the new classloader

        • 2. Deployer Project Structure
          Adrian Brock Master

          Since I making incompatible changes to the deployer api
          which means updating virtual all users of the api, I decided to do some "Dillionism"
          and restructure the project into 4 areas.

          This is really more than just "Dillionism" because it makes people think
          about where and how they implement things.

          The structure is broken down as follows:

          * Core - These are datastructures and helpers that are useful in all the projects.
          * Client - The client side api - this is intended to be used outside jboss
          * Structure - The core infrastructure for the structural deployers - it also defines the core
          datastructures used by the deployers
          * SPI/Impl - This is the deployers and MainDeployer
          * VFS - VFS extensions to the above, see the later post on why the VFS is seperated

          Some quick comments:

          * Most of the projects are seperated into spi/impl, e.g.
          deployers-client and deployers-client-spi
          The one exception is the structure project. Since the only implementation
          of the structure is via the VFS, the implementation of this spi is actually
          in the deployer-vfs project

          * The core api defines the StructureMetaData. I don't really like this being here,
          but it is used by the client side api. As such, we can't really have things
          from the server side leaking into this code, so I didn't want to put it into the
          structure project.

          • 3. VFS Deployers and projects
            Adrian Brock Master

            It was totallly ridiculous how many of the deployers had been written
            without the thought that the deployment might not be backed by a VFS file.

            One of the aims of the new deployers is to provide a "programmatic"
            api so you can deploy things without having a file system at all.

            e.g. Create a datasource (psuedo code)

            Deployment deployment = ...;
            DataSourceMetaData ds = ...;
            deployment.addAttachment(ds);
            deploy(deployment);
            


            This would break with the way those deployers were written, they were
            doing things like:

            public void deploy(DeploymentUnit unit)
            {
             // Oops NPE because getRoot() returns null!
             URL url = unit.getRoot().getURL();
            }
            


            Now there are seperate subclasses VFSDeployment and VFSDeploymentUnit
            that include the VFS based api like getRoot().
            There is not such api for the plain Deployments.

            So if your deployer needs access to the VFS api then it should subclass
            one of the VFS deployers that will automatically ignore deployments that
            aren't backed by the VFS.

            These will give you an api callback like:
            public void deploy(VFSDeploymentUnit unit)
            {
             // Your code here
            }
            


            The check for there being a VFS is already done for you!
            e.g. AbstractSimpleVFSRealDeployer
            
             @Override
             public void deploy(DeploymentUnit unit, T deployment) throws DeploymentException
             {
             if (unit instanceof VFSDeploymentUnit == false)
             return;
            
             VFSDeploymentUnit vfsDeploymentUnit = (VFSDeploymentUnit) unit;
             deploy(vfsDeploymentUnit, deployment);
             }
             /**
             * Deploy a deployment
             *
             * @param unit the unit
             * @param deployment the attachment
             * @throws DeploymentException for any error
             */
             public abstract void deploy(VFSDeploymentUnit unit, T deployment) throws DeploymentException;
            


            NOTE: All the parsing deployers (since they need the VFS to access xml files, etc.)
            have this pattern automatically as well.

            So now you no longer have to worry about the NPE described above. :-)

            • 4. Core spi
              Adrian Brock Master

              There's not much to say about this.

              This project just contains the shared api like DeploymentException, attachments
              and the StructureMetaData.

              • 5. Client spi
                Adrian Brock Master

                The client api is new.

                The aim is to remove the direct use of the DeploymentContext and MainDeployer
                from the client side (including the clients running in the same JVM like the
                profile service or embedded).

                Its made up of two main components:

                DeployerClient

                This is intended as a client side front end to the main deployer.
                The main deployer interface itself extends it to add functions that are
                purely server side (currently this is only shutdown())
                and are not relevant for client side usage.

                NOTE 1) The incomplete deployment exception processing is now a method
                on this api so we don't need to keep re-implementing in different callers
                or figure out how to get a handle on the microcontainer

                NOTE 2) There are methods to deploy a single deployment (think JSR88)
                rather than the add, add, add, process() model of the url scanner.
                However, this implementation of these is still pretty naive and needs completing,
                particularly the incomplete checking for a single deployment.

                Deployment

                The deployment is a client side model of the deployment context.
                It is NOT directly linked to the deployment context.

                It is made up of two main components:
                * predetermined attachments
                * structure metadata

                There are also deployment factories that help in making this task
                simpler.

                A typical usage would be something like:

                DeployerClient deployer = ...;
                VirtualFile vf = ...;
                
                // Create the deployment
                VFSDeploymentFactory factory = VFSDeploymentFactory.getInstance();
                VFSDeployment deployment = factory.createVFSDeployment(vf);
                
                // Add some metadata
                deployment.getPredeterminedManagedObjects().addAttachment(someMetaData);
                
                // Create a subcontext/sub deployment
                ContextInfo subcontext = factory.addContext(deployment, "sub.jar", "META-INF");
                // Add metadata to the subcontext
                subcontext.getPredeterminedManagedObjects().addAttachment(otherMetaData);
                
                // Deploy it
                deployer.addDeployment(deployment);
                deployer.process();
                deployer.checkIncomplete();
                


                The main things missing from the Deployment over the DeploymentContext are:

                1) The DeploymentState. To get this you can DeployerClient.getDeploymentState(name);

                2) The deployment types. I don't have a replacement for this yet, see below on profile service changes

                3) Deployment structure, i.e. the subdeployments and components

                • 6. Profile Service
                  Adrian Brock Master

                  I've redone the profile service to use the client side api (still need to do the same
                  for embedded).

                  I've probably been overzealous in doing this (e.g. no deployment types
                  and getChildren() on the Deployment).

                  I'll describe later how the MainDeployerImpl has an extra interface
                  MainDeployerStructure that is really the place where this information
                  can/should be obtained.
                  e.g. the EJB(3)Util classes that do the ejb-ref searches use this interface
                  to look at deployment structure.

                  • 7. StructureDeployers
                    Adrian Brock Master

                    These haven't changed all that much (at least in princple). The changes are:

                    1) Factory for DeploymentContext

                    The StructuralDeployers implementation is now a factory for the DeploymentContext
                    so it can choose the implementation

                    2) StructureMetaData

                    The StructureMetaData can now handle StructureMetaData in subcontexts
                    so you can specify the structure at subdeployment level.
                    Previously this was only available at the top level deployment level

                    3) Root file and ClassPath

                    I tried to modify the StructureMetaData such that the paths are relative
                    to the current context (rather than the root).
                    But, I haven't suceeded for the ClassPath part.
                    The issue is that manifest references are actually equivalent to

                    addClassPath("../some.jar");
                    

                    and so are not really a part of the context.

                    To make this work consistently, all classpath references are relative
                    to the root of the top level file.
                    Part of this change required passing the root/top-level file and parent file
                    in the StructureDeployer api.

                    IF the VFS was changed such
                    virtualFile.findChild("../some.jar");
                    

                    worked, then we can remove this "cognitive dissonance".
                    All classpath references could be relative to the current context
                    with manifest references translated to add ../ in front them.

                    • 8. MainDeployer changes
                      Adrian Brock Master

                      As described above, the MainDeployer is an extension to the DeployerClient
                      and also implements MainDeployerStructure (deployment navigation).

                      Here I'll describe two other significant changes.

                      1) The configuration now uses the new install callbacks in the microcontainer.
                      That is there is no need to do:

                      <install bean="MainDeployer" method="addDeployer">
                       <parameter><this/></parameter>
                      </install>
                      <uninstall bean="MainDeployer" method="removeDeployer">
                       <parameter><this/></parameter>
                      </uninstall>
                      

                      on every deployer.

                      Instead the MainDeployer (really the StructuralDeployers and DeployersImpl
                      say that they want to be called for all classes that implement the types
                      on their add/remove methods.

                       <bean name="StructuralDeployers" class="org.jboss.deployers.vfs.plugins.structure.VFSStructuralDeployersImpl">
                       <!-- Accept any implementor of structure deployer -->
                       <incallback method="addDeployer"/>
                       <uncallback method="removeDeployer"/>
                       </bean>
                      


                      2) DeployersImpl

                      Like what was done for the StructureDeployers, the real deployers
                      are now handled by a delegate class "Deployers", meaing the MainDeployer
                      actually just acts as a "Fat Interface" and delegates all its real work.

                      The new implementation DeployersImpl is also where the new deployment
                      dependencies (via the MicroContainer) and attachment flow are handled.

                      • 9. DeploymentContext
                        Adrian Brock Master

                        I've nearly got the DeploymentContext to be invisible.

                        It is still used in some places, e.g. the testsuite so I've got deprecated methods
                        that give access to it from the MainDeployer.

                        Another example is the "hack" in the bootstrap that takes the
                        classloader of the first deployed, the one for "server/xxx/conf/"
                        as the classloader to run the bootstrap with.
                        It is also still used by the ClassLoaderFactory api.

                        It should just be a matter time to work through where it is still used
                        and this class can finally be made into an implementation detail
                        that just consolidates deployment information in one place whereas
                        others provides facades over the portions relevant for what they are doing.

                        You can find methods on the [VFS]DeploymentUnit now that should give you
                        everything you need.


                        * DeploymentResource[Class]Loader

                        One part of this change was the re-introduction of what used to be
                        DeploymentInfo.localCL that was used to load resources during deployment.

                        This now has an equaivalent from VFSDeploymentUnit.getDeploymentResourceClassLoader().
                        You cannot use this for classloading but you can use it to load resources.

                        If you don't need the ClassLoader api (which is quite heavy and has
                        well known issues in the Sun JDK - e.g. locking/caching)
                        then just use VFSDeploymentUnit.getDeploymentResourceLoader()
                        api.
                        The classloader is actually implemented on top of this.

                        For simple use cases there is a convience method:
                        VirtualFile file = VFSDeploymentUnit.getFile(path);

                        • 10. Deployer Dependencies
                          Adrian Brock Master

                          The deployments can now have dependencies.

                          These are handled by the Microcontainer which calls back into the
                          MainDeployer/DeployersImpl when dependencies are satisfied.

                          The dependencies are based on stages like the old relative order:
                          PARSE - parsing deployers
                          DESCRIBE (new) - work out dependencies from metadata
                          CLASSLOADER - classloading construction
                          POST_CLASSLOADER - aop
                          REAL - others

                          While this works, there is still work to do to complete this.

                          1) It needs an api such that deployers can add dependencies (on the fly).

                          2) The actual mechanism in the DESCRIBE stage to visit metadata
                          and discover deployment dependencies does not yet exist.
                          The first prototype of this will be with the osgi bundle dependencies,
                          but this needs to be a more generic mechanism.

                          • 11. Attachment Flow
                            Adrian Brock Master

                            The next form of dependency is something I described when I first wrote the
                            deployers.

                            i.e. it should be able to work out what order to process the deployers
                            from what information you input/output.

                            Most of this can be done automatically, e.g.

                            public class MyDeployer extends AbstractSimpleRealDeployer<MyMetaData.class>
                            {
                             public MyDeployer()
                             {
                             super(MyMetaData.class);
                             }
                            }
                            


                            Will automatically know that you want MyMetaData attachments.

                            As an aside, with those classes, it will also know that you don't want to be
                            invoked for deploy/undeploy if there is no such attachment.
                            i.e. no longer any need for the boiler place
                            if (unit.getAttachment(MyMetaData.class) == null)
                             return;
                            

                            If you don't want this filtering, you can turn it off with
                             public MyDeployer()
                             {
                             super(MyMetaData.class);
                             setAllInputs(true);
                             }
                            


                            But back the attachment flow. You can also be explicit about
                            what other information you want. i.e. what deployers should be before/after you.

                            e.g.
                             public MyDeployer()
                             {
                             super(MyMetaData.class);
                             setInputs(SomeOtherMetaData1.class, SomeOtherMetaData2.class);
                             setOutputs(MyOutput.class);
                             }
                            


                            You can see this in the DEBUG where it shows the calcualted ordering
                            and input/outputs, e.g.
                            2007-06-28 14:34:27,112 DEBUG [org.jboss.deployers.plugins.deployers.DeployersImpl] Added deployer org.jboss.deployment.ClientDeployer@39471b for stage Real
                            org.jboss.deployers.vfs.deployer.kernel.KernelDeploymentDeployer@121df2a{inputs=[org.jboss.beans.metadata.spi.BeanMetaData, org.jboss.kernel.spi.deployment.KernelDeplo
                            yment] outputs=[org.jboss.beans.metadata.spi.BeanMetaData]}
                            org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer@1e46a68{inputs=[org.jboss.beans.metadata.spi.BeanMetaData] outputs=[]}
                            org.jboss.deployment.ClientDeployer@39471b{inputs=[] outputs=[org.jboss.metadata.ClientMetaData]}
                            org.jboss.deployment.security.SecurityDeployer@18b4ccb{inputs=[] outputs=[jboss.jacc]}
                            org.jboss.ejb.deployers.EjbDeployer@5e5c15{inputs=[org.jboss.metadata.ApplicationMetaData] outputs=[org.jboss.system.metadata.ServiceMetaData]}
                            org.jboss.ejb3.deployers.AppClientScanningDeployer@1743ff4{inputs=[org.jboss.ejb3.metamodel.ApplicationClientDD] outputs=[org.jboss.ejb3.metamodel.ApplicationClientDD]
                            }
                            org.jboss.ejb3.deployers.EJBRegistrationDeployer@d1afd3{inputs=[] outputs=[org.jboss.ejb3.Ejb3Deployment]}
                            org.jboss.ejb3.deployers.EJBStage2Deployer@1ee2ea8{inputs=[org.jboss.ejb3.Ejb3Deployment] outputs=[]}
                            org.jboss.ejb3.deployers.Ejb3ClientDeployer@182b9b5{inputs=[org.jboss.ejb3.metamodel.ApplicationClientDD] outputs=[org.jboss.ejb3.clientmodule.ClientENCInjectionContai
                            ner]}
                            org.jboss.resource.deployers.RARDeployer@f04dae{inputs=[org.jboss.resource.metadata.ConnectorMetaData] outputs=[org.jboss.system.metadata.ServiceMetaData]}
                            org.jboss.web.tomcat.service.deployers.TomcatDeployer@133926{inputs=[org.jboss.metadata.WebMetaData] outputs=[org.jboss.system.metadata.ServiceMetaData]}
                            org.jboss.system.deployers.ServiceDeploymentDeployer@1b7c76{inputs=[org.jboss.system.metadata.ServiceMetaData, org.jboss.system.metadata.ServiceDeployment] outputs=[or
                            g.jboss.system.metadata.ServiceMetaData]}
                            org.jboss.system.deployers.ServiceDeployer@147e668{inputs=[org.jboss.system.metadata.ServiceMetaData] outputs=[]}
                            


                            You'll notice the Service deployer is after many of the other deployers that create
                            ServiceMetaData (even though it was one of the first deployers deployed).

                            NOTE: The old relativeOrder still exists, but it is probably not as a good a way
                            to determine ordering.

                            • 12. Components
                              Adrian Brock Master

                              Components have been changed in two fundamental ways.

                              1) Deployers don't see components by default any more.

                              To get components, you need to do

                              public MyDeployer()
                              {
                               setWantComponents(true);
                              }
                              


                              2) However if you are dealing with components, you probably don't want
                              to process the main context. So you are more likely to use
                              public MyDeployer()
                              {
                               setComponentsOnly(true);
                              }
                              


                              The two current component deployers (JMX and POJO) both
                              only look at components now.

                              NOTE: On a related point you can also say that you are only interested in
                              the top level deployment, not subdeployments.

                              The other major change, (maybe you spotted it when I listed the DeployerStages or
                              the list of deployers for REAL), is that there is no longer a COMPONENT stage.
                              With the attachment flows, it is no longer necessary. The attachment ordering
                              will make sure it is invoked in the correct order.

                              • 13. Parsing Deployers
                                Adrian Brock Master

                                The parsing deployers are now almost entirely declaritive.
                                In many cases, you could consider that we don't need seperate classes
                                and the whole thing could be done with xml.

                                You no longer need to do

                                public void deploy(...)
                                {
                                 createMetaData(unit, null, "-ds.xml");
                                }
                                


                                Instead you can do
                                public MyParsingDeployer
                                {
                                 setSuffix("-ds.xml");
                                }
                                


                                The only reason for a class is if you need to do "after" processing.
                                Many deployers had overridden "random" parts of the api without
                                spotting that this callback already existed in init() and it already
                                does the "null" checking.

                                public class BeanDeployer extends SchemaResolverDeployer<KernelDeployment>
                                {
                                 /**
                                 * Create a new BeanDeployer.
                                 */
                                 public BeanDeployer()
                                 {
                                 super(KernelDeployment.class);
                                 setSuffix("-beans.xml");
                                 }
                                
                                 @Override
                                 protected void init(VFSDeploymentUnit unit, KernelDeployment metaData, VirtualFile file) throws Exception
                                 {
                                 String name = file.toURI().toString();
                                 metaData.setName(name);
                                 }
                                


                                Some other deployers were also doing other checks besides the metadata
                                file existed. IMHO most of these are HACKs that should be fixed.
                                But to make this easier, I added the accepts() callback to the parsing deployers
                                so each deployer doesn't have to put this wiring in themselves.

                                • 14. Re: New Deployer changes - overview
                                  Adrian Brock Master

                                  Finally, besides what I said above about things needing completing,
                                  the new classloader still needs to be integrated.

                                  I have some code for this, but I'm not really happy with the metadata model
                                  yet. Besides that it needs some other fixes where the RepositoryClassLoader
                                  is assumed.

                                  I decided not to commit this work yet.

                                  The other major work is in tidying up the testsuite.

                                  There are a number of failures due attachments (e.g. the ear metadata)
                                  not being serializable and also the VFS urls not being usable on the client side.

                                  1 2 Previous Next