New Deployer changes - overview

[adrian.brock]

This is a quick brain dump of all the deployer changes I just committed.

As usual this is an overview thread, to collect all the information in one place.
Create separate threads if you want to discuss indivdual topics in more detail.

First, the changes are not yet complete.

1) AOP - AOP has a deployer inside its project. That needs updating
and AOP re-releasing. But since AOP is required for EJB3
I've temporarily hacked a new version in the "system" project in
org.jboss.aop.temp.AOPDeployer

2) WS - Similarly webservices have their own deployers.
I've just disabled webservices for now. I'll be working on this next
once I've looked at some other minor loose ends.

3) EJB3 - I've updated the EJB3 deployers to use attachment flows (see below)
but I'm not sure if they are correct. Its working on some
of the basic tests in the ejb3 project tests.
I suspect there may need to be some attachment flow dependency on the
security deployer (possibly others?), but I'm not sure where it needs to go.

4) Security - I think this needs a lot more work to where I would like to get it.

5) Embedded - I haven't updated this project to the new deployers.
The version I currently have of this project doesn't build so that needs to be fixed
before I can do this. Most of the changes should be minor, i.e. updating
it to use the new "client side" deployment api.

I'm going to break down these posts into the follow areas:

* New Project Structure - separation of spi/impl and vfs, etc.
* VFSDeployer - why this extra abstraction is important
* Core - a quick discussion on the core project
* Client - the new "client side" deployment api
* Structure - the changes to the structure deployments and how the structure is built
* MainDeployer - the changes to the main deployer and how it relates to the client side api
* DeploymentContext - how that is nearly invisible now and what needs to be done to complete the process
* Deployers - the new stages (real dependencies), attachment flow, changes to how components work, making the parsing deployers "more declarative"
* TODO - other work that still needs doing like integrating the new classloader

[adrian.brock]

AOP issue thread raised here: http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4058739#4058739

[adrian.brock]

Since I making incompatible changes to the deployer api
which means updating virtual all users of the api, I decided to do some "Dillionism"
and restructure the project into 4 areas.

This is really more than just "Dillionism" because it makes people think
about where and how they implement things.

The structure is broken down as follows:

* Core - These are datastructures and helpers that are useful in all the projects.
* Client - The client side api - this is intended to be used outside jboss
* Structure - The core infrastructure for the structural deployers - it also defines the core
datastructures used by the deployers
* SPI/Impl - This is the deployers and MainDeployer
* VFS - VFS extensions to the above, see the later post on why the VFS is seperated

Some quick comments:

* Most of the projects are seperated into spi/impl, e.g.
deployers-client and deployers-client-spi
The one exception is the structure project. Since the only implementation
of the structure is via the VFS, the implementation of this spi is actually
in the deployer-vfs project

* The core api defines the StructureMetaData. I don't really like this being here,
but it is used by the client side api. As such, we can't really have things
from the server side leaking into this code, so I didn't want to put it into the
structure project.

[adrian.brock]

It was totallly ridiculous how many of the deployers had been written
without the thought that the deployment might not be backed by a VFS file.

One of the aims of the new deployers is to provide a "programmatic"
api so you can deploy things without having a file system at all.

e.g. Create a datasource (psuedo code)

Deployment deployment = ...;
DataSourceMetaData ds = ...;
deployment.addAttachment(ds);
deploy(deployment);

This would break with the way those deployers were written, they were
doing things like:

public void deploy(DeploymentUnit unit)
{
 // Oops NPE because getRoot() returns null!
 URL url = unit.getRoot().getURL();
}

Now there are seperate subclasses VFSDeployment and VFSDeploymentUnit
that include the VFS based api like getRoot().
There is not such api for the plain Deployments.

So if your deployer needs access to the VFS api then it should subclass
one of the VFS deployers that will automatically ignore deployments that
aren't backed by the VFS.

These will give you an api callback like:

public void deploy(VFSDeploymentUnit unit)
{
 // Your code here
}

The check for there being a VFS is already done for you!
e.g. AbstractSimpleVFSRealDeployer

 @Override
 public void deploy(DeploymentUnit unit, T deployment) throws DeploymentException
 {
 if (unit instanceof VFSDeploymentUnit == false)
 return;

 VFSDeploymentUnit vfsDeploymentUnit = (VFSDeploymentUnit) unit;
 deploy(vfsDeploymentUnit, deployment);
 }
 /**
 * Deploy a deployment
 *
 * @param unit the unit
 * @param deployment the attachment
 * @throws DeploymentException for any error
 */
 public abstract void deploy(VFSDeploymentUnit unit, T deployment) throws DeploymentException;

NOTE: All the parsing deployers (since they need the VFS to access xml files, etc.)
have this pattern automatically as well.

So now you no longer have to worry about the NPE described above. :-)

[adrian.brock]

There's not much to say about this.

This project just contains the shared api like DeploymentException, attachments
and the StructureMetaData.

[adrian.brock]

The client api is new.

The aim is to remove the direct use of the DeploymentContext and MainDeployer
from the client side (including the clients running in the same JVM like the
profile service or embedded).

Its made up of two main components:

DeployerClient

This is intended as a client side front end to the main deployer.
The main deployer interface itself extends it to add functions that are
purely server side (currently this is only shutdown())
and are not relevant for client side usage.

NOTE 1) The incomplete deployment exception processing is now a method
on this api so we don't need to keep re-implementing in different callers
or figure out how to get a handle on the microcontainer

NOTE 2) There are methods to deploy a single deployment (think JSR88)
rather than the add, add, add, process() model of the url scanner.
However, this implementation of these is still pretty naive and needs completing,
particularly the incomplete checking for a single deployment.

Deployment

The deployment is a client side model of the deployment context.
It is NOT directly linked to the deployment context.

It is made up of two main components:
* predetermined attachments
* structure metadata

There are also deployment factories that help in making this task
simpler.

A typical usage would be something like:

DeployerClient deployer = ...;
VirtualFile vf = ...;

// Create the deployment
VFSDeploymentFactory factory = VFSDeploymentFactory.getInstance();
VFSDeployment deployment = factory.createVFSDeployment(vf);

// Add some metadata
deployment.getPredeterminedManagedObjects().addAttachment(someMetaData);

// Create a subcontext/sub deployment
ContextInfo subcontext = factory.addContext(deployment, "sub.jar", "META-INF");
// Add metadata to the subcontext
subcontext.getPredeterminedManagedObjects().addAttachment(otherMetaData);

// Deploy it
deployer.addDeployment(deployment);
deployer.process();
deployer.checkIncomplete();

The main things missing from the Deployment over the DeploymentContext are:

1) The DeploymentState. To get this you can DeployerClient.getDeploymentState(name);

2) The deployment types. I don't have a replacement for this yet, see below on profile service changes

3) Deployment structure, i.e. the subdeployments and components

[adrian.brock]

I've redone the profile service to use the client side api (still need to do the same
for embedded).

I've probably been overzealous in doing this (e.g. no deployment types
and getChildren() on the Deployment).

I'll describe later how the MainDeployerImpl has an extra interface
MainDeployerStructure that is really the place where this information
can/should be obtained.
e.g. the EJB(3)Util classes that do the ejb-ref searches use this interface
to look at deployment structure.

[adrian.brock]

These haven't changed all that much (at least in princple). The changes are:

1) Factory for DeploymentContext

The StructuralDeployers implementation is now a factory for the DeploymentContext
so it can choose the implementation

2) StructureMetaData

The StructureMetaData can now handle StructureMetaData in subcontexts
so you can specify the structure at subdeployment level.
Previously this was only available at the top level deployment level

3) Root file and ClassPath

I tried to modify the StructureMetaData such that the paths are relative
to the current context (rather than the root).
But, I haven't suceeded for the ClassPath part.
The issue is that manifest references are actually equivalent to

addClassPath("../some.jar");

and so are not really a part of the context.

To make this work consistently, all classpath references are relative
to the root of the top level file.
Part of this change required passing the root/top-level file and parent file
in the StructureDeployer api.

IF the VFS was changed such

virtualFile.findChild("../some.jar");

worked, then we can remove this "cognitive dissonance".
All classpath references could be relative to the current context
with manifest references translated to add ../ in front them.

[adrian.brock]

As described above, the MainDeployer is an extension to the DeployerClient
and also implements MainDeployerStructure (deployment navigation).

Here I'll describe two other significant changes.

1) The configuration now uses the new install callbacks in the microcontainer.
That is there is no need to do:

<install bean="MainDeployer" method="addDeployer">
 <parameter><this/></parameter>
</install>
<uninstall bean="MainDeployer" method="removeDeployer">
 <parameter><this/></parameter>
</uninstall>

on every deployer.

Instead the MainDeployer (really the StructuralDeployers and DeployersImpl
say that they want to be called for all classes that implement the types
on their add/remove methods.

 <bean name="StructuralDeployers" class="org.jboss.deployers.vfs.plugins.structure.VFSStructuralDeployersImpl">
 <!-- Accept any implementor of structure deployer -->
 <incallback method="addDeployer"/>
 <uncallback method="removeDeployer"/>
 </bean>

2) DeployersImpl

Like what was done for the StructureDeployers, the real deployers
are now handled by a delegate class "Deployers", meaing the MainDeployer
actually just acts as a "Fat Interface" and delegates all its real work.

The new implementation DeployersImpl is also where the new deployment
dependencies (via the MicroContainer) and attachment flow are handled.

[adrian.brock]

I've nearly got the DeploymentContext to be invisible.

It is still used in some places, e.g. the testsuite so I've got deprecated methods
that give access to it from the MainDeployer.

Another example is the "hack" in the bootstrap that takes the
classloader of the first deployed, the one for "server/xxx/conf/"
as the classloader to run the bootstrap with.
It is also still used by the ClassLoaderFactory api.

It should just be a matter time to work through where it is still used
and this class can finally be made into an implementation detail
that just consolidates deployment information in one place whereas
others provides facades over the portions relevant for what they are doing.

You can find methods on the [VFS]DeploymentUnit now that should give you
everything you need.


* DeploymentResource[Class]Loader

One part of this change was the re-introduction of what used to be
DeploymentInfo.localCL that was used to load resources during deployment.

This now has an equaivalent from VFSDeploymentUnit.getDeploymentResourceClassLoader().
You cannot use this for classloading but you can use it to load resources.

If you don't need the ClassLoader api (which is quite heavy and has
well known issues in the Sun JDK - e.g. locking/caching)
then just use VFSDeploymentUnit.getDeploymentResourceLoader()
api.
The classloader is actually implemented on top of this.

For simple use cases there is a convience method:
VirtualFile file = VFSDeploymentUnit.getFile(path);

[adrian.brock]

The deployments can now have dependencies.

These are handled by the Microcontainer which calls back into the
MainDeployer/DeployersImpl when dependencies are satisfied.

The dependencies are based on stages like the old relative order:
PARSE - parsing deployers
DESCRIBE (new) - work out dependencies from metadata
CLASSLOADER - classloading construction
POST_CLASSLOADER - aop
REAL - others

While this works, there is still work to do to complete this.

1) It needs an api such that deployers can add dependencies (on the fly).

2) The actual mechanism in the DESCRIBE stage to visit metadata
and discover deployment dependencies does not yet exist.
The first prototype of this will be with the osgi bundle dependencies,
but this needs to be a more generic mechanism.

[adrian.brock]

The next form of dependency is something I described when I first wrote the
deployers.

i.e. it should be able to work out what order to process the deployers
from what information you input/output.

Most of this can be done automatically, e.g.

public class MyDeployer extends AbstractSimpleRealDeployer<MyMetaData.class>
{
 public MyDeployer()
 {
 super(MyMetaData.class);
 }
}

Will automatically know that you want MyMetaData attachments.

As an aside, with those classes, it will also know that you don't want to be
invoked for deploy/undeploy if there is no such attachment.
i.e. no longer any need for the boiler place

if (unit.getAttachment(MyMetaData.class) == null)
 return;

If you don't want this filtering, you can turn it off with

 public MyDeployer()
 {
 super(MyMetaData.class);
 setAllInputs(true);
 }

But back the attachment flow. You can also be explicit about
what other information you want. i.e. what deployers should be before/after you.

e.g.

 public MyDeployer()
 {
 super(MyMetaData.class);
 setInputs(SomeOtherMetaData1.class, SomeOtherMetaData2.class);
 setOutputs(MyOutput.class);
 }

You can see this in the DEBUG where it shows the calcualted ordering
and input/outputs, e.g.

2007-06-28 14:34:27,112 DEBUG [org.jboss.deployers.plugins.deployers.DeployersImpl] Added deployer org.jboss.deployment.ClientDeployer@39471b for stage Real
org.jboss.deployers.vfs.deployer.kernel.KernelDeploymentDeployer@121df2a{inputs=[org.jboss.beans.metadata.spi.BeanMetaData, org.jboss.kernel.spi.deployment.KernelDeplo
yment] outputs=[org.jboss.beans.metadata.spi.BeanMetaData]}
org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer@1e46a68{inputs=[org.jboss.beans.metadata.spi.BeanMetaData] outputs=[]}
org.jboss.deployment.ClientDeployer@39471b{inputs=[] outputs=[org.jboss.metadata.ClientMetaData]}
org.jboss.deployment.security.SecurityDeployer@18b4ccb{inputs=[] outputs=[jboss.jacc]}
org.jboss.ejb.deployers.EjbDeployer@5e5c15{inputs=[org.jboss.metadata.ApplicationMetaData] outputs=[org.jboss.system.metadata.ServiceMetaData]}
org.jboss.ejb3.deployers.AppClientScanningDeployer@1743ff4{inputs=[org.jboss.ejb3.metamodel.ApplicationClientDD] outputs=[org.jboss.ejb3.metamodel.ApplicationClientDD]
}
org.jboss.ejb3.deployers.EJBRegistrationDeployer@d1afd3{inputs=[] outputs=[org.jboss.ejb3.Ejb3Deployment]}
org.jboss.ejb3.deployers.EJBStage2Deployer@1ee2ea8{inputs=[org.jboss.ejb3.Ejb3Deployment] outputs=[]}
org.jboss.ejb3.deployers.Ejb3ClientDeployer@182b9b5{inputs=[org.jboss.ejb3.metamodel.ApplicationClientDD] outputs=[org.jboss.ejb3.clientmodule.ClientENCInjectionContai
ner]}
org.jboss.resource.deployers.RARDeployer@f04dae{inputs=[org.jboss.resource.metadata.ConnectorMetaData] outputs=[org.jboss.system.metadata.ServiceMetaData]}
org.jboss.web.tomcat.service.deployers.TomcatDeployer@133926{inputs=[org.jboss.metadata.WebMetaData] outputs=[org.jboss.system.metadata.ServiceMetaData]}
org.jboss.system.deployers.ServiceDeploymentDeployer@1b7c76{inputs=[org.jboss.system.metadata.ServiceMetaData, org.jboss.system.metadata.ServiceDeployment] outputs=[or
g.jboss.system.metadata.ServiceMetaData]}
org.jboss.system.deployers.ServiceDeployer@147e668{inputs=[org.jboss.system.metadata.ServiceMetaData] outputs=[]}

You'll notice the Service deployer is after many of the other deployers that create
ServiceMetaData (even though it was one of the first deployers deployed).

NOTE: The old relativeOrder still exists, but it is probably not as a good a way
to determine ordering.

[adrian.brock]

Components have been changed in two fundamental ways.

1) Deployers don't see components by default any more.

To get components, you need to do

public MyDeployer()
{
 setWantComponents(true);
}

2) However if you are dealing with components, you probably don't want
to process the main context. So you are more likely to use

public MyDeployer()
{
 setComponentsOnly(true);
}

The two current component deployers (JMX and POJO) both
only look at components now.

NOTE: On a related point you can also say that you are only interested in
the top level deployment, not subdeployments.

The other major change, (maybe you spotted it when I listed the DeployerStages or
the list of deployers for REAL), is that there is no longer a COMPONENT stage.
With the attachment flows, it is no longer necessary. The attachment ordering
will make sure it is invoked in the correct order.

[adrian.brock]

The parsing deployers are now almost entirely declaritive.
In many cases, you could consider that we don't need seperate classes
and the whole thing could be done with xml.

You no longer need to do

public void deploy(...)
{
 createMetaData(unit, null, "-ds.xml");
}

Instead you can do

public MyParsingDeployer
{
 setSuffix("-ds.xml");
}

The only reason for a class is if you need to do "after" processing.
Many deployers had overridden "random" parts of the api without
spotting that this callback already existed in init() and it already
does the "null" checking.

public class BeanDeployer extends SchemaResolverDeployer<KernelDeployment>
{
 /**
 * Create a new BeanDeployer.
 */
 public BeanDeployer()
 {
 super(KernelDeployment.class);
 setSuffix("-beans.xml");
 }

 @Override
 protected void init(VFSDeploymentUnit unit, KernelDeployment metaData, VirtualFile file) throws Exception
 {
 String name = file.toURI().toString();
 metaData.setName(name);
 }

Some other deployers were also doing other checks besides the metadata
file existed. IMHO most of these are HACKs that should be fixed.
But to make this easier, I added the accepts() callback to the parsing deployers
so each deployer doesn't have to put this wiring in themselves.

[adrian.brock]

Finally, besides what I said above about things needing completing,
the new classloader still needs to be integrated.

I have some code for this, but I'm not really happy with the metadata model
yet. Besides that it needs some other fixes where the RepositoryClassLoader
is assumed.

I decided not to commit this work yet.

The other major work is in tidying up the testsuite.

There are a number of failures due attachments (e.g. the ear metadata)
not being serializable and also the VFS urls not being usable on the client side.