12 Replies Latest reply on Apr 22, 2008 1:26 PM by Scott Stark

    File locking

    Ales Justin Master

      Moving file locking issue to the forum.

      I tried using the feature I recently added - explicit unpacking of archives - with top level .ear file.

      Adding this jboss-structure.xml to ear's META-INF directory does the trick of unpacking ear into tmp directory, but still doesn't allow the original ear file to be deleted.

      <structure>
       <context modification="Unpack">
       <path name=""/>
       <metaDataPath>
       <path name="META-INF"/>
       </metaDataPath>
       </context>
       <context>
       <path name="inner.war"/>
       <metaDataPath>
       <path name="WEB-INF"/>
       </metaDataPath>
       </context>
      </structure>
      


      I guess when you do any kind of reading inside file, in this case reading the contents of jboss-structure.xml, the file gets locked.

      I don't see how we can get past this w/o rewriting how URL(Connection)s are handled.

      Can someone try how we stand with this on Linux?

        • 1. Re: File locking
          Ales Justin Master

           

          "adrian@jboss.org" wrote:

          "alesj" wrote:

          Can someone try how we stand with this on Linux?

          There's no locking issue on Linux.
          It will however manifest itself as a "leaked"
          open file.

          I guess it would need something to explictly close()
          the URLConnection or JarFile created in the JARHandler constructor
          when it is unpacked and replaced?

          i.e. My guess is something is keeping a reference previous
          packed VirtualFile and hence to the JarFile which means the
          connection/file handle is kept open.



          • 2. Re: File locking
            Ales Justin Master

             

            "adrian@jboss.org" wrote:

            I guess it would need something to explictly close()
            the URLConnection or JarFile created in the JARHandler constructor
            when it is unpacked and replaced?

            How are you gonna do that? :-)
            Since URLConnection doesn't have anything useful to do a close.
            See below for more info.

            "adrian@jboss.org" wrote:

            i.e. My guess is something is keeping a reference previous
            packed VirtualFile and hence to the JarFile which means the
            connection/file handle is kept open.

            When we do lastModified, for HDScanning purposes, we mostly use URLConnection::getLastModified, which is probably the cause for open handle.
            See URLExistsUnitTestCase in JBossVFS.

            In the case of non-jar file, I was able to do this hacky thing:
             InputStream in = conn.getInputStream();
             long lastModified;
             try
             {
             lastModified = conn.getLastModified();
             System.out.println("lastModified, "+lastModified);
             assertNotSame("lastModified", 0, lastModified);
             }
             finally
             {
             in.close();
             }
             assertTrue(tmp.getAbsolutePath()+" deleted", tmp.delete());
            

            Open/close input stream, and was then allowed to delete the file.
            W/o this, you cannot delete the file.

            But in the case of jar file, see 2nd test in URLExistsUnitTestCase, I cannot do the same hack, since it complains about not specifying the entry:
            java.io.IOException: no entry name specified
             at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:129)
             at org.jboss.test.virtual.test.URLExistsUnitTestCase.testJarURLs(URLExistsUnitTestCase.java:102)
            


            • 3. Re: File locking
              Adrian Brock Master

               

              "alesj" wrote:

              But in the case of jar file, see 2nd test in URLExistsUnitTestCase, I cannot do the same hack, since it complains about not specifying the entry:
              java.io.IOException: no entry name specified
               at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:129)
               at org.jboss.test.virtual.test.URLExistsUnitTestCase.testJarURLs(URLExistsUnitTestCase.java:102)
              


              For the top level jar file you should be using the file: url not the jar: url
              A top level jar is a file :-)

              There used to be a HACK in the VFS JAR code that worked around this issue,
              i.e. it reworked the URL based on the parent but I don't see it in the current codebase?
              It probably got removed when the url handling was reworked?

              Also you need to be careful with JAR files and their URL connections.

              The JDK caches them internally and can even mmap them on some operating systems.
              So testing a jar entry has changed isn't the same thing as checking the top level file
              changed.
              Additionally, closing the top jar connection will close the cached version
              which can lead to strange errors.

              The internal caching is the source of the problems (e.g. locking on windows)
              and something Sun have consistently refused to fix. :-(

              • 4. Re: File locking
                Adrian Brock Master

                 

                "alesj" wrote:

                See URLExistsUnitTestCase in JBossVFS.


                What is the purpose of this test?
                It is testing the JDK rather than the VFS so why is it in the VFS testsuite?

                Surely this code:
                 conn = tmpURL.openConnection();
                 lastModified = conn.getLastModified();
                

                should be changed to use the VFS to check the getLastModified works
                and assert the file can still be deleted?

                • 5. Re: File locking
                  Ales Justin Master

                   

                  "adrian@jboss.org" wrote:
                  "alesj" wrote:

                  See URLExistsUnitTestCase in JBossVFS.


                  What is the purpose of this test?
                  It is testing the JDK rather than the VFS so why is it in the VFS testsuite?

                  Surely this code:
                   conn = tmpURL.openConnection();
                   lastModified = conn.getLastModified();
                  

                  should be changed to use the VFS to check the getLastModified works
                  and assert the file can still be deleted?

                  It's Scott's test.
                  But I think the purpose is to verify that URLConnection::lastModified can also tell you that the file has been deleted.
                  See comments on the test code:
                   /**
                   * Test file deletion can be detected via URLConnection.getLastModified == 0.
                  

                  Since like it was already said, we use this for lastModified check on URL based VFS handlers.

                  • 6. Re: File locking
                    Sacha Labourey Master

                    What about using the same class from the Apache Java implementation? I doubt they have the same bug.

                    • 7. Re: File locking
                      Ales Justin Master

                       

                      "sacha.labourey@jboss.com" wrote:
                      What about using the same class from the Apache Java implementation? I doubt they have the same bug.

                      Which class?
                      And how would you force any JDK to use it?

                      • 8. Re: File locking
                        Ales Justin Master

                        You mean this one, that you already proposed. :-)
                        - http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4134400

                        • 9. Re: File locking
                          Adrian Brock Master

                           

                          "sacha.labourey@jboss.com" wrote:
                          What about using the same class from the Apache Java implementation? I doubt they have the same bug.


                          Because it uses native code.

                          But more importantly, even if we use
                          org.apache.xxx.JarFile or org.jboss.xxx.JarFile
                          that doesn't mean everybody will.

                          e.g. Hibernate opens jar files to scan for annotations.

                          Of course if this causes locking problems then we can use Ales's new unpack option
                          to give them a temporary copy.
                          But ideally we want to be minimizing the number of cases where this is required.
                          i.e. by finding other workarounds that don't require the unpacking.

                          • 10. Re: File locking
                            Scott Stark Master

                             

                            "alesj" wrote:

                            It's Scott's test.
                            But I think the purpose is to verify that URLConnection::lastModified can also tell you that the file has been deleted.
                            See comments on the test code:
                             /**
                             * Test file deletion can be detected via URLConnection.getLastModified == 0.
                            

                            Since like it was already said, we use this for lastModified check on URL based VFS handlers.

                            Correct, its testing whether if the URLConnection api can be used to detect file deletion.


                            • 11. Re: File locking
                              Adrian Brock Master

                               

                              "scott.stark@jboss.org" wrote:

                              Correct, its testing whether if the URLConnection api can be used to detect file deletion.


                              So we'll know when the JDK doesn't support it, but not when it
                              doesn't work in the VFS. :-)

                              Do you think it is a good idea to rely on implementation details of a particular JDK
                              rather than what it says in the spec? (rhetorical question :-)

                              • 12. Re: File locking
                                Scott Stark Master

                                Right, we only know we are relying on an unstable implementation detail when this test fails. Taking over the handler implementation details to not rely a jdk version was the point of the urlconn-work branch of the vfs:
                                https://svn.jboss.org/repos/jbossas/projects/vfs/branches/urlconn-work/