[dlmiles]

It looks like you are leaving the work of the actual copy to WTP's PublishUtil class.

My understand of how this works inside WTP is that it does a file copy of each file to the directory $HOME/workspace/.metadata/.plugins/org.eclipse.wst.server.core/tmp12345.class and then does a rename to the final deployment location (which WTP is expecting to be in a directory like $HOME/workspace/.metadata/.plugins/org.eclipse.wst.server.core/tmp1/

If you check your directory $HOME/workspace/.metadata/.plugins/org.eclipse.wst.server.core/ both before and after your a publish with JBossTools-AS where your runtime is on another file system you should see left over files appear and the number of files in the directory grow. It does not seem WTP deletes the old temporary after the renameTo() failed.

This works great for Tomcat and other such containers that can treat the "tmp1" directory above as a deployment directory but not do useful if the runtime it somewhere else.

Since all the built in WTP runtime tools I have seen use the "tmp1/" directory in the workspace area as the deployment area you can't expect these tools to work for deployment outside of the workspace area.


My suggestion would be for JBoss to open up an API request with the server tooling to allow the PublishUtil tool to be instanced and configured an exact path to be used as the temporary location to copy to.

File tmpDir = new File("/opt/jboss-4.2.1.GA/server/default/tmp/jbosside-as");
if(!tmpDir.getParent().exist() == false)
 throw new RuntimeException("Directory " + tmpDir.getParent().getAbsolutePath() + " does not exist");
tmpDir.mkdirs();
PublishUtil publishUtilInstance = new PublishUtil(tmpDir);

// Then in your code instead of using the static methods use the instanced ones
publishUtilInstance.callSomeMethod(someArgs);

So it looks like to me you need a way to pass down a temporary path that WTP standard methods can use as the temporary directory to perform the initial copy from the "project build area". This is runtime specific. Its also concerning that the rename must have failed and that no error is indicated in the UI so there is another area to look at in that.



What does this idiom "copied = copied &&" do ?

http://viewvc.jboss.org/cgi-bin/viewvc.cgi/jbosstools/trunk/as/plugins/org.jboss.ide.eclipse.as.core/jbosscore/org/jboss/ide/eclipse/as/core/util/FileUtil.java?view=log

#fileSafeCopy(File src, File dest, IFileUtilListener listener) {
 copied = copied && fileSafeCopy(subFiles, newDest, listener);
 }

 // I would think that propagation of an error would but more correct here :

 if(fileSafeCopy(subFiles, newDest, listener) == false) {
 copied = false;
}

[rob.stryker]

Thanks for all your help in this. I really wish I could have dedicated more to it before you did, but hey, thats hte benefit of community right?

I will definitely try to have this fixed for CR1.

>> What does this idiom "copied = copied &&" do ?

That's my attempt to make sure *all* files were copied correctly. If any single file is not copied, the end value of copied would be false.

I realize I probably don't do anything with that value in the end, but it's there for me to then use in the Event Log. I'll need to make sure that shows up in the event log, or find a better way to report it.

Again, thanks definitely for all the help here. Your tips are going to be very helpful.

[maxandersen]

I never seen anything in that tmp file location you mention - it just contains a profile.dat file which I think just represents the last list of resources that have been copied ?

[dlmiles]

Thanks for all your help in this. I really wish I could have dedicated more to it before you did, but hey, thats hte benefit of community right?

>> What does this idiom "copied = copied &&" do ?

That's my attempt to make sure *all* files were copied correctly. If any single file is not copied, the end value of copied would be false.

Yes sure on the community comment.

I still don't understand the "copied = copied &&" I can't see how it works (even if you presume an optimizing JVM does not remove the operation due to redundancy). Its a single "=" character, if it were a compare operator then that would fail fast (as oppose to try to copy everything with best effort) even though copied is not set to anything other than 'true' AFAIS.

I never seen anything in that tmp file location you mention - it just contains a profile.dat file which I think just represents the last list of resources that have been copied ?

Can you be more specific? Are you windows or linux ?

Have you ever create a Tomcat runtime and WAR/DWP deployed to it ? Have you even created JBoss runtime using WTP driver and deployed to it ? If no and no then I wouldn't expect to see much in the directory if you've never had a publish failure.

What is the specific path on your computer you are looking at ?

Can you confirm the absolute path to your JBoss runtime, I have not verified if the problem I see is re-creatable on windows by having JBoss runtime on a different drive letter. I'm not sure if windows has rename magic (which turns out to be a file copy) if you rename a file across drive letters.

[maxandersen]

"dlmiles" wrote:

I never seen anything in that tmp file location you mention - it just contains a profile.dat file which I think just represents the last list of resources that have been copied ?

Can you be more specific? Are you windows or linux ?

both.

Have you ever create a Tomcat runtime and WAR/DWP deployed to it ?

sure - but now we are talking about fixing our jboss adapter deployment....

Have you even created JBoss runtime using WTP driver and deployed to it ? If no and no then I wouldn't expect to see much in the directory if you've never had a publish failure.

I stepped through the code of jstpublisher andi don't see any temp file copying being performed when we use our adapter...I do realize that tomcat and jboss adapter does this of different reasons (tomcat - has it as a "deploy" location and jboss wtp as a tmp storage before zipping it up)

What is the specific path on your computer you are looking at ?

the one you gave me. .metadata/org.eclipse...etc.

Can you confirm the absolute path to your JBoss runtime, I have not verified if the problem I see is re-creatable on windows by having JBoss runtime on a different drive letter. I'm not sure if windows has rename magic (which turns out to be a file copy) if you rename a file across drive letters.

I'm not saying we don't have issues across disk boundaries; i'm just saying i'm not seeing any temporary files being created...

[rob.stryker]

>> I still don't understand the "copied = copied && fileSafeCopy(etc)"

assume: fileSafeCopy returns true or false;
prime copied: boolean copied = true;

in a loop:
copied = copied && fileSafeCopy(next file)

which means
new value of copied = old value of coppied && the result of the function

Therefore:
if fileSafeCopy returns true:
newCopied = oldCopied && true
newCopied = true && true
newCopied = true

if fileSafeCopy returns false:
newCopied = oldCopied && false
newCopied = true && false
newCopied = false

The only difference is that we're assigning the result to the same variable that we're checking. It's very similar to saying: i = i + 1, except instead of the + operator, we're using the && operator.

x = x && true
x = x && false
i = i + 1
i = i - 1

It's the same idea.

[dlmiles]

I stepped through the code of jstpublisher andi don't see any temp file copying being performed when we use our adapter...I do realize that tomcat and jboss adapter does this of different reasons (tomcat - has it as a "deploy" location and jboss wtp as a tmp storage before zipping it up)

I'm not saying we don't have issues across disk boundaries; i'm just saying i'm not seeing any temporary files being created...

My comment earlier was that I guess you delegate the task of the actual file copy to PublishUtil and as it stands this API provided by WTP is only good if you intend to use the ".../tmp1/" publishing area (due to this cross file system concern). This is why I think you need to open an API change request with WTP (or at least have them knock you back and fork your own implementation of PublishUtil). I can see any reason why you can't have the API improved as it should be possible to maintain 100% backward compatibility.

The copy of PublishUtil I have from 1.5 still has the bug that is uses a static variable for the file copy buffer and this class is shared by many drivers, and as far as I know having two runtimes active at the same time is allowed and updates maybe done concurrently. So instancing this class with a private 'byte[] buffer' and a configurable 'File tempDir' makes sense to me.


Anyway to spot the temp file usage try something like this:

Set a breakpoint in this class org.eclipse.jst.server.core.PublishUtil for all methods, or rather org.eclipse.wst.server.core.PublishUtil (since it appears to have moved between 1.5.x and 2.0). I only have WTP 1.5 tree checked out, I don't know if there is a viewcvs HTTP resource to cite more information so I can't see what the current WTP 2.0 PublishUtil looks like.

Set a breakpoint for java.io.File#createTempFile(String, File), modify a .jave file and save, do a publish with th JbossTools-AS driver.

If you open the source to the PublishUtil class and look for "temp".

[maxandersen]

I can see the temp creation, but we can't use any changerequest to WTP 2.x at this point since that won't be out until a few months. (we can of course report it)

But before that we need to figure out if we actually *have* an issue here....but if we have an issue all other deployers should have this issue....

Darryl - are you saying all WTP deployers that can deploy to servers deploy dir are broken when it deploys across filesystems ?

[dlmiles]

I can see the temp creation, but we can't use any changerequest to WTP 2.x at this point since that won't be out until a few months. (we can of course report it)

But before that we need to figure out if we actually *have* an issue here....but if we have an issue all other deployers should have this issue....

I don't really understand the point you're making or what you are asking (did you read or understand my other comments in this thread ?)


The purpose of this thread was to query the problem with users of JBossTool-AS.

I think sufficient detail has been given in the thread so far to easily allow a 3rd party to re-create the issue. Its been cited a few times that you've been unable to observe the issue or are unable to see any temporary files in use. I've tried to be helpful and establish what maybe different by asking questions about your observation methods and configuration but got given vague answers. Unfortunately you've not provided sufficient detail to allow me to assist you to observe the problem (if you are still unable to see it).

On the subject are other things broken, probably who cares about them ? I'm not trying to save the world here. I'm not in a position to be able to make any claims for all WTP deployers, but I have already reported in this thread that the WTP provided JBoss deployer appears (to me) to deploy into the "tmp0/" directory using a single file .EAR from what I have observed. That's all I can provide you with.

"Darryl - are you saying all WTP deployers that can deploy to servers deploy dir are broken when it deploys across filesystems ?"

Which servers deploy dir ? Across what file systems where ?


There are 3 filesystems in play here, the file system your workspace is in, the file system your project is in (thats right, your project does not have to be inside your workspace tree) and the file system your runtime is in.

All 3 file systems can be different. When you publish using a WTP runtime driver to Tomcat (for example) the following scenario occurs:

* You save a change to a .java file, this gets saved in your project area lets say this was JavaSource/domain/MyClass.java
* Eclipse this auto-compiles the class for this, and this ends inside your project areas lets say as bin/domain/MyClass.class
* Eclipse has an active runtime with auto-publish enabled, so events fire off due to the .class change.
* The runtime driver processes the resource change event and if it uses the WST PublishUtil class for doing the work it creates an empty temporary file in your workspace area as .metadata/.plugins/org.eclipse.wst.server.core/tmp12345.tmp
* Then a file copy is performed from project area file bin/domain/MyClass.class into workspace area file .metadata/.plugins/org.eclipse.wst.server.core/tmp12345.tmp
* Then the tmp12345.tmp file is closed, and the modification stamp fixed up (to whatever you want it to be)
* Then the file tmp12345.tmp is renamed to another location still within the workspace area but also in the deployment area of the runtime (operating from within the workspace) lets call this path .metadata/.plugins/org.eclipse.wst.server.core/tmp0/webapps/MyProject/WEB-INF/classes/domain/MyClass.class

As you can see according to my rules there is nothing wrong with this, a file copy was performed between project build area and workspace runtime instance deployment area and the only rename that occured stayed inside the same area (that being the workspace area).

Now the problem occurs when you use an external runtime area and that external directory is not on the same filesystem as the workspace area. This is because if you use PublishUtil from WTP it will still perform the file copy into the workspace tmp12345.tmp file.

So my suggestion is to allow the temporary directory where that file copy is performed to be a configurable thing, this way JBossTools-AS can still use PublishUtil class and can instruct it to use a directory like /opt/jboss-4.2.1.GA/server/default/tmp/jbosstools-as/ this making the temporary path ./opt/jboss-4.2.1.GA/server/default/tmp/jbosstools-as/tmp12345.tmp in the above hypothetical situation.


When this happens maybe tone down the 64Kb static buffer from PublishUtil to something >= 512 and <= 8192 there are plenty of white papers detailing write performance va write size.

[maxandersen]

I don't really understand the point you're making or what you are asking (did you read or understand my other comments in this thread ?)

Yes - but i'm simply trying to figure out why I don't see this issue and how many other deployers will have this issue.

I fully understand what you are saying about how Tomcat (and as far as I understand other WTP deployers apparently perform).

I just can't reproduce this error and thus I'm asking you if

a) this occurs with other deployers too

b) did you see leftover artifacts when using our jboss deployer

If a and b then this is most likely limited only to possible issues in jstpublisher, if a and not b or not and b then it is some issue specific to our tooling.

So my suggestion is to allow the temporary directory where that file copy is performed to be a configurable thing, this way JBossTools-AS can still use PublishUtil class and can instruct it to use a directory like /opt/jboss-4.2.1.GA/server/default/tmp/jbosstools-as/ this making the temporary path ./opt/jboss-4.2.1.GA/server/default/tmp/jbosstools-as/tmp12345.tmp in the above hypothetical situation.

And we can't perform that change since it won't be ready in a WTP release in time; so we probably need to fork PublishUtil to make it work.

[dlmiles]

I just can't reproduce this error and thus I'm asking you if

a) this occurs with other deployers too

b) did you see leftover artifacts when using our jboss deployer

If a and b then this is most likely limited only to possible issues in jstpublisher, if a and not b or not and b then it is some issue specific to our tooling.

a) I can't comment on 'a' since I don't have any other deployers setup, just Tomcat and JBoss. As I understand the limitations with Java/JVM and Unix system semantics I'm happy to claim that all users of WST PublishUtil are affected while the "File tempDir" member has a hardwired value. It's important to phrase the question 'b' correctly "this occurs with other deployers that deploy to runtimes outside of the workspace filesystem that also use PublishUtil class provided by WST in WTP to place the files into the deployment directory". Maybe you can pick a deployer and container you'd like to me to spare a minute to test it with ?

b) Yes I do. Here are the left over files (from $HOME/workspace/.metadata/.plugins/org.eclipse.wst.server.core/ to my example EAREclipseManager.ear project from deploying a moment ago:

tmp40855.xml tmp40856.MF tmp40857.xml tmp40858.class tmp40859.class tmp40860.class tmp40861.class

The modification to the JBoss runtime deploy directory has all the directories created that I'd expect to see for the project, but the 7 missing files are the MANIFEST.MF, the ejb-jar.xml, the application.xml and 4 x Implementation .class files. Examining the contents of those 7 tmpXXXXX.???? files clearly indicate they are those files, left over from failed renameTo() calls.


I'd like to help you observe the issue, this would be best on a linux system (as that is what I use day-to-day). Please confirm your file system layout from 'df', the absolute location of your workspace, the absolute location of your project, the absolute location of your jboss runtime.

I can then confirm if I believe your configuration should be able to observe the problem and provide steps.

And we can't perform that change since it won't be ready in a WTP release in time; so we probably need to fork PublishUtil to make it work.

There is always the option of listing the (cross filesystem boundary) as a known issue for the CR1 release. Since it is not that difficult to work around for a developer to ensure his runtime is on the same file system as his workspace. Open an API request ASAP and creating a backwards compatible implementation of the static PublishUtil and a new instancable delegate.

But... I do think the lack of UI feedback over publish failure is a MUST FIX item for any major release. When the renameTo fails the user must know (no silent failures).

[maxandersen]

Ok - my df shows me this:

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
234381968 63291276 158992748 29% /
/dev/sda1 101086 30344 65523 32% /boot
tmpfs 2016112 0 2016112 0% /dev/shm

so I don't think this could be tested easily on my setup; well I guess I could mount a usb disk and deploy to that ;)

Have you reported these issues against bugs.eclipse.org ? If not I suggest you to do so and let me know about the bugid and I'll add whatever we do to fix this to that bug.

[dlmiles]

Ok - my df shows me this:
<snip>
so I don't think this could be tested easily on my setup; well I guess I could mount a usb disk and deploy to that ;)

Have you reported these issues against bugs.eclipse.org ? If not I suggest you to do so and let me know about the bugid and I'll add whatever we do to fix this to that bug.

If you don't want to use USB, dig out my post dated "Posted: Tue Oct 9, 2007 01:43 AM" and execute the commands to create yourself a 160Mb loopback file system. This is where Linux mounts a file as a fileing system. The file can exist on your "/" file system as /tmp/bigfile.ext2


Yes I reported the issues for the Tomcat driver with it too did not consider the different file system problems. This was over a year ago and these things were mostly fixed during WTP 0.7/1.0 time frame.

But I'm not sure there is a bug with the Eclipse WTP provided JBoss driver, since it publishes to the "tmp0/" directory in exploded format (without using the .ear directory extension), once that pass is complete it then creates an EAR file in the JBoss runtime deployment directory (out of the contents it just put into the "tmp0/" directory). So in effect converting from exploded format to single file EAR format implicitly means Eclipse ends up doing a file copy. I am completely guessing this is what it does from my observations.

[maxandersen]

Ok - we'll look into reproducing fixing this.

With respect to the bugids did you report those about buffer size not being good enough etc. ?

Would be good to have the bugids ?

And yes, the adapter needs to do exploded deployment to trigger this error.

[dlmiles]

Ok - we'll look into reproducing fixing this.

With respect to the bugids did you report those about buffer size not being good enough etc. ?

Would be good to have the bugids ?

And yes, the adapter needs to do exploded deployment to trigger this error.

Short:

I have never lodged bugids about either buffer size or threading issues, I never cared for them enough, more concerned with creating bugids for things that actually affect me rather than a thorny code review.


Long:

I would guess a windows developer mistakenly thought that using a large buffer 64kb will mean that file writes will handle 64Kb atomically, since most files being published are < 64Kb and this skips over most problems of the runtime being a part built file, <head_shake>NO, NO, NO, NO, NO!</head_shake>. I can only guess this is the illogical reason for this large size. I use the term "windows developer" in the derogatory sense, I hope I offend, ROFL :)

The threading issue I commented in another comment to a different bugid, knowing the maintainer for that file may read it. For all I know all server publish actions are serialized and since server drivers are the only things expected to use PublishUtil class there maybe no real problem here, even if I think I see one. It would also be very rare occurrence needing concurrent usage of PublishUtil to trigger, which presumes the user has 2 active runtimes at once, all these factors somewhat reduce its importance to "can't be bothered to write a bug entry for it" status.

On the subject of size; I care more that the buffer is shared by all users (is 'static') than about the fact its 64kb, but if you start instancing that class then making it 1Kb should get you within 97% performance, and 4Kb within 99% performance of 64Kb (if you ever through there was a performance issue in the first place) or just allocate a byte[] on demand. Those white papers indicate you get diminishing returns for performance.