VFS Aspect for deployments

[starksm64]

Regarding, http://jira.jboss.com/jira/browse/JBAS-2380 I have gotten back to prototyping this. The notion of the VFS aspect is that class loading and deployers should not be dealing with the current local file, jar unpacking, etc notions. Further, optimized local deployments should be possible for more efficient class and resource access.

In working on this I have decided jars are a bad thing when it comes to a VFS abstraction. They do not fit into a VFS notion as they do not generally have the property of a random access file system. The current java classes REQUIRE that the jar be accessible via a File object to achieve the random access semantics of the java.util.jar.JarFile api. Throw nested jars into the mix and the concept of a JarVFS with O(1) is broken even when the archive is local.

As such, the notion of a JarVFS over an arbitrary URL protocol does not make sense. So what I am thinking is that we do need to move away from the notion of ant as a deploy tool and have a canonical jboss deployment package with tools for creating such packages to with this archive/VFS mismatch. Some properties of a "canonical jboss deployment package":

* Its either an exploded directory structure or a collapsed jar. By collapsed jar I mean that all nested jars are expanded within the containing jar.
* It survives server restarts. Unlike the tmp/deploy unpacked content of the current releases, the deployment package is a persistent configuration of the server.
* It contains optimization information. The package can contain indexes for class loading, resources along with timestamp(s) for identifying whether its in synch with the traditional hot deployable source.
* It is a subset of the classes seen in the full deployment archive. A common problem with scoped deployments is users including classes that cannot be overriden. There are hacks for this in the web container class loader, but this should be handled for all containers outside of the class loader.
* Tools exist for creating the deployment package. If I'm creating a read-only CD or a netbootable image where there should be no runtime copying of the traditional deployment I need to be able to create the deployment package offline.

This needs to be integrated with the jsr88 deployment service:
http://www.jboss.com/index.html?module=bb&op=viewtopic&t=65999

A significantly different class loader than the URLClassLoader is going to be needed as unlike the traditional deployment archive where a URLClassLoader hierarchy is created using ordered lists of the URLs of the various jars/directories in the deployment, the VFSClassLoader(s) is composed of an ordered list of search contexts in the VFS. This differs from the URLClassLoader when the deployment package is a collapsed archive, as in the case of this web-sso.ear:

META-INF/application.xml
roles.properties
users.properties
sso-form-auth1.war/error.html
sso-form-auth1.war/index.html
sso-form-auth1.war/login.html
sso-form-auth1.war/META-INF/MANIFEST.MF
sso-form-auth1.war/WEB-INF/classes/org/jboss/test/web/servlets/EJBServlet.class
sso-form-auth1.war/WEB-INF/classes/org/jboss/test/web/servlets/LogoutServlet.class
sso-form-auth1.war/WEB-INF/classes/org/jboss/test/web/util/
sso-form-auth1.war/WEB-INF/classes/org/jboss/test/web/util/Util.class
sso-form-auth1.war/WEB-INF/jboss-web.xml
sso-form-auth1.war/WEB-INF/web.xml

The web-sso.ear/sso-form-auth1.war search context is not an archive. Its a prefix in the web-sso.ear deployment archive that groups the entries associated with the sso-form-auth1.war search context. When the deployment pacakge is fully unpacked then web-sso.ear/sso-form-auth1.war search context is a directory just as it in the URLClassLoader case.

[starksm64]

Note that the lifecycle of a deployment in terms of class loaders will be very different in this scheme. Basically, we will be able to create the VFSClassLoader structure immeadiately for all deployment containers since we know the complete structure and scopes. The one conflict with this could be the ENC class loaders. Managing the java:comp/env component level context based on class loader is something we have wanted to get away for a while now.

Also, the current DeploymentInfo localCL notion used to locate resources local to the deployment would nned to be replaced with a resource lookup starting at the given search context, with no delegation to parent class loaders or other search contexts visible to the VFSClassLoader.

[starksm64]

I was also reminded of an older feature request that would essentially allow a deployment to be composed of an arbitrary local directory structure. If there was a mapping definition that essentially defined the canonical deployment VFS from the local directories we could allow for this. The RFE was specific to having a war directory structure without the .war suffix do to some ide/tool limitation or feature.

[starksm64]

Another issue related to the deployment package that could be integrated via the VFS is the jarjar obfuscation of classes in a deployment which should only be an implementation detail. For example, dom4j, log4j, commons-logging xerces, hibernate etc. that could be considered implementation details which should not leak to the associated application component type system. The deployment tool could jarjar such deployment private classes, and this could be a general feature available to any deployment rather than having to integrate jarjar obfuscation into the build.

[adrian.brock]

As such, the notion of a JarVFS over an arbitrary URL protocol does not make sense. So what I am thinking is that we do need to move away from the notion of ant as a deploy tool

No. We should not move away from the most popular development tools just because
it is impossible to do every feature using them.

[adrian.brock]

Some properties of a "canonical jboss deployment package":

* Its either an exploded directory structure or a collapsed jar. By collapsed jar I mean that all nested jars are expanded within the containing jar.
* It survives server restarts. Unlike the tmp/deploy unpacked content of the current releases, the deployment package is a persistent configuration of the server.
* It contains optimization information. The package can contain indexes for class loading, resources along with timestamp(s) for identifying whether its in synch with the traditional hot deployable source.
* It is a subset of the classes seen in the full deployment archive. A common problem with scoped deployments is users including classes that cannot be overriden. There are hacks for this in the web container class loader, but this should be handled for all containers outside of the class loader.
* Tools exist for creating the deployment package. If I'm creating a read-only CD or a netbootable image where there should be no runtime copying of the traditional deployment I need to be able to create the deployment package offline.

* For oldstyle jar deployments it should still work. There should be no separate tool step (required to be done by the user - JBoss can do it though) for deploy(URL).

[adrian.brock]

The fundamental issue for me is the "V" in VFS means virtual.

The underlying physical packaging is irrelevant and can be provided by a number
of different implementations (although their feature sets and performance maybe limited).

* Jar based (old school)
* Optimized JBoss (the enhancements you are talking about)
* IDE mapped deployment - no real packaging. just references to IDE project resources and classes
etc.

We do not care what actually happens when we do findClass(), findResource()
listResources(), getSubDeployments(), etc.
as long as the rest of the deployment layer doesn't have to go around
the side of the api to achieve things.
Ok, in some crippled implementations it may have to unpack jars into tmp
or download and "cache" the deployment
or it might throw an exception to say "can't do directory listsings"
then the user can either accept it, use the workarounds
or use the more advanced VFS implementations.

[tom.elrod]

I don?t really understand everything that you are saying in this thread, but since we are going to be opening up development on classloading, I was wondering if we could also look at how we do redeployments. We currently only load a class once for any given scoped repository and then when that class is needed by other deployments within the same scope, we will re-use the class that has already been loaded.

So if have two deployments (A and B), which each have a class Foo, and A loads Foo first, when B needs Foo, it will use the one loaded from A. So if A gets redeployed, B will also need to be *manually* redeployed as well (since is maintaining what is now a stale reference to Foo, which was loaded from the original A deployment).

Would it be possible to keep track of the deployments that are using classes from another deployment, and then when recognize that the original deployment is redeployed, automatically redeploy any other deployments that need to be? So from previous example, we know that B is using Foo which was originally loaded by A, so keep track of this information and when A is redeployed detect that B needs to be redeployed as well.

[starksm64]

* For oldstyle jar deployments it should still work. There should be no separate tool step (required to be done by the user - JBoss can do it though) for deploy(URL).

Nothing changes by default in terms of what can be dropped into deploy.

[bill.burke]

The one conflict with this could be the ENC class loaders. Managing the java:comp/env component level context based on class loader is something we have wanted to get away for a while now.

I have already gotten away from classloader based ENC in EJB3. I use a ThreadLocal to push/pop the current ENC rather than relying on what the current classloader is. One of the reasons I did it this way (other than it being simpler) was that there is a bug in our current ENC implementation where you get the incorrect ENC on nested calls (or something like that). Problem is, I can't remember if this problem is real or imagined. :)

[starksm64]

Would it be possible to keep track of the deployments that are using classes from another deployment, and then when recognize that the original deployment is redeployed, automatically redeploy any other deployments that need to be? So from previous example, we know that B is using Foo which was originally loaded by A, so keep track of this information and when A is redeployed detect that B needs to be redeployed as well.

Yes, but I don't know how the dependency is created at this point.

[starksm64]

I have already gotten away from classloader based ENC in EJB3. I use a ThreadLocal to push/pop the current ENC rather than relying on what the current classloader is. One of the reasons I did it this way (other than it being simpler) was that there is a bug in our current ENC implementation where you get the incorrect ENC on nested calls (or something like that). Problem is, I can't remember if this problem is real or imagined. :)

Yes, I will look at this. There used to be such a problem a long time ago (maybe as old as 2.4?) that required the addition of traversing the class loaders to see if a parent was the one used to register the ENC.

[starksm64]

The fundamental issue for me is the "V" in VFS means virtual.

The underlying physical packaging is irrelevant and can be provided by a number of different implementations (although their feature sets and performance maybe limited).

Agreed, there is a mixture of implementation details and usecases to consider in this thread. What I am saying is that I'm not starting with the braindead JarVFS as the default implementation.

We do not care what actually happens when we do findClass(), findResource()
listResources(), getSubDeployments(), etc.
as long as the rest of the deployment layer doesn't have to go around
the side of the api to achieve things.

Correct, but I belive some of the existing class loading constructs such as TLC keyed contexts for the ENC and creating deployment local class loaders for resources needs to be changed. The deployer should be given a class loader with an api sufficient for the deployment tasks.

[adrian.brock]

Correct, but I belive some of the existing class loading constructs such as TLC keyed contexts for the ENC

Yes this should be application/deployment or component scoped "metadata".
If that uses the TLC (like it does now) that is an implementation detail.

[dimitris]

Scott, have you checked in your prototype? I had the VDFComponent prototype in system2 but I think it would be better to have a separate module for VFS as part of HEAD/common, adrian made a note about this here:

http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3902262#3902262

VDFComponent allowed navigation over a (local) filesystem, then to access a resource you just get a local URL, I guess, this is where a VFSClassloader could come, or you have a different idea (build everything on VFSClassloader)?