[mstruk]

I commited noCopy support to jar-alter-work branch. I also added two additional system property switches:

-Djboss.vfs.forceVfsJar=true

to fallback to vfsjar

and

-Djboss.vfs.forceNoReaper=true

to turn off asynchronous release of file locks


Known issues:

1) When deployment archive is removed from deploy directory an exception occurs, and the application is not undeployed - even though the file has been removed.

00:29:28,205 WARN [HDScanner] Scan failed
org.jboss.deployers.spi.DeploymentException: Exception determining structure: AbstractVFSDeployment(autogen-bloated.ear)
 at org.jboss.deployers.spi.DeploymentException.rethrowAsDeploymentException(DeploymentException.java:49)
 at org.jboss.deployers.structure.spi.helpers.AbstractStructuralDeployers.determineStructure(AbstractStructuralDeployers.java:85)
 at org.jboss.deployers.plugins.main.MainDeployerImpl.determineStructure(MainDeployerImpl.java:845)
 at org.jboss.deployers.plugins.main.MainDeployerImpl.addDeployment(MainDeployerImpl.java:303)
 at org.jboss.deployers.plugins.main.MainDeployerImpl.addDeployment(MainDeployerImpl.java:260)
 at org.jboss.system.server.profileservice.hotdeploy.HDScanner.scan(HDScanner.java:270)
 at org.jboss.system.server.profileservice.hotdeploy.HDScanner.run(HDScanner.java:221)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
 at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: Error determining structure: autogen-bloated.ear
 at org.jboss.deployment.EARStructure.determineStructure(EARStructure.java:274)
 at org.jboss.deployers.vfs.plugins.structure.StructureDeployerWrapper.determineStructure(StructureDeployerWrapper.java:72)
 at org.jboss.deployers.vfs.plugins.structure.VFSStructuralDeployersImpl.doDetermineStructure(VFSStructuralDeployersImpl.java:196)
 at org.jboss.deployers.vfs.plugins.structure.VFSStructuralDeployersImpl.determineStructure(VFSStructuralDeployersImpl.java:220)
 at org.jboss.deployers.structure.spi.helpers.AbstractStructuralDeployers.determineStructure(AbstractStructuralDeployers.java:77)
 ... 14 more
Caused by: java.lang.RuntimeException: /autogen-bloated.jar module listed in application.xml does not exist within .ear autogen-bloated.ear
 at org.jboss.deployment.EARStructure.determineStructure(EARStructure.java:253)
 ... 18 more

I'm not sure this is VFS's problem.


2) As applications are redeployed, memory consumption keeps increasing. It needs to be investigated.


Tomorrow I'm merging the code into trunk :)

Cheers,

- marko

[alesj]

"mstruk" wrote:
I commited noCopy support to jar-alter-work branch. I also added two additional system property switches:

-Djboss.vfs.forceVfsJar=true

to fallback to vfsjar

and

-Djboss.vfs.forceNoReaper=true

to turn off asynchronous release of file locks

Nice. ;-)

"mstruk" wrote:

Known issues:

1) When deployment archive is removed from deploy directory an exception occurs, and the application is not undeployed - even though the file has been removed.

00:29:28,205 WARN [HDScanner] Scan failed
org.jboss.deployers.spi.DeploymentException: Exception determining structure: AbstractVFSDeployment(autogen-bloated.ear)
 at org.jboss.deployers.spi.DeploymentException.rethrowAsDeploymentException(DeploymentException.java:49)
 at org.jboss.deployers.structure.spi.helpers.AbstractStructuralDeployers.determineStructure(AbstractStructuralDeployers.java:85)
 at org.jboss.deployers.plugins.main.MainDeployerImpl.determineStructure(MainDeployerImpl.java:845)
 at org.jboss.deployers.plugins.main.MainDeployerImpl.addDeployment(MainDeployerImpl.java:303)
 at org.jboss.deployers.plugins.main.MainDeployerImpl.addDeployment(MainDeployerImpl.java:260)
 at org.jboss.system.server.profileservice.hotdeploy.HDScanner.scan(HDScanner.java:270)
 at org.jboss.system.server.profileservice.hotdeploy.HDScanner.run(HDScanner.java:221)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
 at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.RuntimeException: Error determining structure: autogen-bloated.ear
 at org.jboss.deployment.EARStructure.determineStructure(EARStructure.java:274)
 at org.jboss.deployers.vfs.plugins.structure.StructureDeployerWrapper.determineStructure(StructureDeployerWrapper.java:72)
 at org.jboss.deployers.vfs.plugins.structure.VFSStructuralDeployersImpl.doDetermineStructure(VFSStructuralDeployersImpl.java:196)
 at org.jboss.deployers.vfs.plugins.structure.VFSStructuralDeployersImpl.determineStructure(VFSStructuralDeployersImpl.java:220)
 at org.jboss.deployers.structure.spi.helpers.AbstractStructuralDeployers.determineStructure(AbstractStructuralDeployers.java:77)
 ... 14 more
Caused by: java.lang.RuntimeException: /autogen-bloated.jar module listed in application.xml does not exist within .ear autogen-bloated.ear
 at org.jboss.deployment.EARStructure.determineStructure(EARStructure.java:253)
 ... 18 more

I'm not sure this is VFS's problem.

Better question is why is this ear being treated as new deployment.
Since you can see that it's being added as deployment.

1) Why is it added / who still sees it?
2) What's the real cause module jar cannot be found?

"mstruk" wrote:

2) As applications are redeployed, memory consumption keeps increasing. It needs to be investigated.

That would be great.

"mstruk" wrote:

Tomorrow I'm merging the code into trunk :)

Like Dimitris said, little commits as possible - making revert simple. :-)

[mstruk]

I fixed the 'Scan failed' issue from my previous post. It was a VFS's problem - in a way that existing tests couldn't detect it. I added a test for it.

'jar-alter-work' branch is now merged in the trunk - did it in one commit :)

Give it a try, and if something isn't working, tell me about it.

- marko

[dimitris]

So I've updated AS trunk to the latest VFS 2.0.0.Beta12 that Ales produced that contains the new alternative implementation.

The problem that showed up immediately with the testsuite is that the jboss-minimal-tests fail. This is done by starting the jboss minimal config, then deploy a shutdown.sar and remove immediately. At undeploy time, shutdown.sar issues an exit and the server stop.

Now, on Windows, if you try to remove the file before the reaper has run, the file cannot be removed, it is still locked.

Then, if the reaper runs, the file gets unlocked so you can remove it, but on both windows and unix you get a:

2008-06-02 21:12:55,140 WARN [org.jboss.test.jmx.shutdown.ExitOnShutdown] (HDScanner) Stopping failed jboss.test:name=ExitOnShutdown
java.lang.NoClassDefFoundError: org/jboss/test/jmx/shutdown/ExitOnShutdown$1
 at org.jboss.test.jmx.shutdown.ExitOnShutdown.stopService(ExitOnShutdown.java:48)
 at org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:328)
 at org.jboss.system.ServiceMBeanSupport.stop(ServiceMBeanSupport.java:206)

The inner class org/jboss/test/jmx/shutdown/ExitOnShutdown$1 cannot be loaded anymore!

After going in cyrcles all this time with the file locking issue, can we not replicate in VFS the functionality of good old jboss 4.x that it is known to work (i.e. point the classloader to copies in tmp, but at the same time don't do excessive file copying/unziping)

[mstruk]

The first problem you encounter and it's a reaper issue :)

As far as VFS is concerned -DforceNoReaper=true should do the trick here.

Regarding NoClassDefFoundError, this looks to be an issue beyond VFS, I may not totally understand the issue but some ideas - off the bat ...

So the essence of the problem is that when a deployed archive is removed jboss learns about it after the fact, there can be no beforeRemove event where components could be undeployed, and lifecycle events processed while classes are still available. But if required classes have already been loaded then there should be no issue.

For well known deployment modules that are part of wider jbossAS infrastructure we can know in advance what classes are still required after file removal. Deployer could have a preload feature - where it reads some configuration that tells it - 'when you load this class, also load these classes'.

It's a relatively high-maintenance solution (having to determine all the dependencies and keeping them up-to-date), but it might be a reasonable best-of-both-worlds.

For unknown third party components - like user applications - I don't see any other way but to either (1) use jboss-4 style tmp copies as suggested or (2) preload all the classes in the archive - this one could be used for small archives. Or (3) if we can determine in advance that after file removal there will be no need for any additional classes (depending on component type maybe) then we don't have to do anything, but I'm not sure it's possible to reliably predict that.

Now, I have no idea how difficult this would be to implement :)

Cheers,

- marko

[dimitris]

I did some local testing with the latest VFS Beta13 and no-matter what combination of flags I use, I cannot get to the point of getting the test with shutdown.sar behave correctly.

The other thing is I don't see noticable differences in the performance of the various implementations. Probably the reaper impl is a little bit faster.

My conclusion is we should really go for the good old tested mode of copying jars to tmp and loading them from there.

Ales is telling me this is not only a VFS issue; the deployers will have to be changed as well. I don't know what exactly is required for that, but if we don't go that direction I feel we are just going cyrcles with this problem.

[alesj]

Ales is telling me this is not only a VFS issue; the deployers will have to be changed as well. I don't know what exactly is required for that, but if we don't go that direction I feel we are just going cyrcles with this problem.

No, deployers already have this. ;-)
I was just trying to explain it to you, that I've made a little fix to handle this the right way - adding new VFSUtils api and a new ModificationType enum in deployers.
I now diff between unpack (meant for nested archives only) and explode (mostly meant for top level archives, but could be used on nested as well == then it's the same as unpack).

Why I want to diff the two - we should already be able to handle (read, scan, ...) top levels, so I don't really see the need to unpack them, except if you explicitly say so, and that is what I call explode.

[dimitris]

How you treat a case of .e.g. a simple ejb .jar or a .sar without nested libraries. It's not nested and not exploded either. We just want to have a copy of that in tmp to be able to remove the original from deploy.

[alesj]

We just want to have a copy of that in tmp to be able to remove the original from deploy.

You mean like this

2008-06-05 17:27:02,828 DEBUG [org.jboss.deployers.plugins.deployers.DeployersImpl] (HDScanner) Undeploying vfszip:/C:/projects/jboss5/trunk/build/output/jboss-5.0.0.CR1/server/minimal/deploy/shutdown.sar
2008-06-05 17:27:02,828 DEBUG [org.jboss.system.ServiceController] (HDScanner) stopping service: jboss.test:name=ExitOnShutdown
2008-06-05 17:27:02,828 DEBUG [org.jboss.test.jmx.shutdown.ExitOnShutdown] (HDScanner) Stopping jboss.test:name=ExitOnShutdown
2008-06-05 17:27:02,828 DEBUG [org.jboss.test.jmx.shutdown.ExitOnShutdown] (HDScanner) Stopped jboss.test:name=ExitOnShutdown
2008-06-05 17:27:02,828 DEBUG [org.jboss.system.ServiceController] (HDScanner) destroying service: jboss.test:name=ExitOnShutdown
2008-06-05 17:27:02,828 DEBUG [org.jboss.system.ServiceController] (HDScanner) removing service: jboss.test:name=ExitOnShutdown

You can call it magic. ;-)