1 2 Previous Next 23 Replies Latest reply on Jun 5, 2008 11:32 AM by Ales Justin

    Alternative vfs jar implementation

    Marko Strukelj Master

      I wrote an alternative implementation of jar handling - from scratch in order to simplify and reduce the amount of code, and specifically to address the problems I kept running into when trying to fix JBVFS-4.

      I currently have what appears to be a fully working jar handling implementation, that exists in parallel to vfsjar - I named it vfszip. It works with jboss trunk, it does away with issues I was working on in vfsjar, and has no problems with file locks on windows - undeployment and redeployment work as they should. I don't think it's production ready but it may be getting close. Some things still need to be finalized - temp files for inner jars, serialization, priviliged blocks, and maybe other things I don't see yet. Performance-wise it is comparable to vfsjar, I did not do any memory profiling yet. Also archive file closing / reopening is a little bit too aggressive at the moment, so performance can still be improved by a few percent.

      Some details about my implementation ...

      I made no change to VFS API - I did make some additions and changes in abstract context and handler classes that don't affect other vfs context implementations. I changed FileSystemContext to use vfszip impl instead of JarHandler - it could be made configurable through system properties.

      I centralized all the logic in one place - in ZipEntryVFSContext, so it's easy to control access to resources (files). This way handlers are little more than proxies to a context. I tightly coupled ZipEntryHandler to ZipEntryVFSContext. I think it's not such a good idea to pass FileSystemContext to a JarHandler for example - significantly limiting how the two can interact. My ZipEntryHandler presupposes it is created through ZipEntryContext, so it works with ZipEntryContext directly and can interface with it in implementation specific ways. I remodeled the code in such a way that the same code can be used to handle inner jars and outer jars by mounting ZipEntryVFSContext into another ZipEntryVFSContext - I introduced a mounting mechanism using ReferenceHandler and a little extra code in abstract classes. This way a jar file inside FileSystemContext is handled by mounting a ZipEntryVFSContext. The code is more modular and more contained.

      JBVFS-4 is a natural non-issue in this alternative implementation. Handling of inner jars of arbitrary depth is automatic. When file locking issue came to my attention I was able to fix it in a few hours with this implementation. Due to centralized access to jar files the issue was trivial to fix.

      My implementation makes no use of jar:file: URLs. My handler does not extend AbstractURLHandler - I don't use URL for getLastModified(), getSize(), openStream(). My implementation controls access to JarFile centrally inside context and opens and closes it as necessary. For URLs, vfsURLs I always generate vfszip: url schema so any URL access goes through VirtualFileURLConnection and through my context again.

      The code is in svn:
      https://svn.jboss.org/repos/jbossas/projects/vfs/branches/jar-alter-work

      If anyone is interested, give it a try, take a look at the code, tell me what you think, what's missing, what's not working properly ... I'll be glad to answer any questions.

      Cheers :)

      - marko

        • 1. Re: Alternative vfs jar implementation
          Scott Stark Master

          Fantastic if it works! I'll pull it down into my jbossas workspace and try it out tonight.

          • 2. Re: Alternative vfs jar implementation
            Scott Stark Master

            Just pulling the branch down and running mvn package on it shows a number of failures. Are those expected at this point?

            Running org.jboss.test.virtual.test.JAREntryTestCase
            Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.136 sec
            [INFO] Building jar: /tmp/surefirebooter129.jar
            Running org.jboss.test.virtual.test.CopyFileVFSUnitTestCase
            Tests run: 42, Failures: 0, Errors: 11, Skipped: 0, Time elapsed: 0.692 sec <<< FAILURE!
            [INFO] Building jar: /tmp/surefirebooter132.jar
            Running org.jboss.test.virtual.test.VirtualFileUnitTestCase
            Tests run: 82, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.236 sec
            [INFO] Building jar: /tmp/surefirebooter135.jar
            Running org.jboss.test.virtual.test.JARVirtualFileHandlerUnitTestCase
            Tests run: 44, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.22 sec
            [INFO] Building jar: /tmp/surefirebooter138.jar
            Running org.jboss.test.virtual.test.VFSUnitTestCase
            Tests run: 84, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.213 sec
            [INFO] Building jar: /tmp/surefirebooter141.jar
            Running org.jboss.test.virtual.test.JARSerializationUnitTestCase
            Tests run: 4, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 0.182 sec <<< FAILURE!
            [INFO] Building jar: /tmp/surefirebooter144.jar
            Running org.jboss.test.virtual.test.SundryVFSUnitTestCase
            Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.105 sec
            [INFO] Building jar: /tmp/surefirebooter147.jar
            Running org.jboss.test.virtual.test.VisitorUnitTestCase
            Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.14 sec
            [INFO] Building jar: /tmp/surefirebooter150.jar
            Running org.jboss.test.virtual.test.CopyJAREntryTestCase
            Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.082 sec
            [INFO] Building jar: /tmp/surefirebooter153.jar
            Running org.jboss.test.virtual.test.FileVirtualFileHandlerUnitTestCase
            Tests run: 44, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.681 sec
            [INFO] Building jar: /tmp/surefirebooter156.jar
            Running org.jboss.test.virtual.test.ZipEntryHandlerUnitTestCase
            Tests run: 44, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.19 sec <<< FAILURE!
            [INFO] Building jar: /tmp/surefirebooter159.jar
            Running org.jboss.test.virtual.test.ZipEntryVFSContextUnitTestCase
            Tests run: 23, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.166 sec <<< FAILURE!
            [INFO] Building jar: /tmp/surefirebooter162.jar
            Running org.jboss.test.virtual.test.UnpackTestCase
            Tests run: 5, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 0.133 sec <<< FAILURE!
            [INFO] Building jar: /tmp/surefirebooter165.jar
            Running org.jboss.test.virtual.test.FileVFSUnitTestCase
            Tests run: 42, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 0.499 sec <<< FAILURE!
            [INFO] Building jar: /tmp/surefirebooter168.jar
            Running org.jboss.test.virtual.test.PathQueryTestCase
            Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.084 sec
            [INFO] Building jar: /tmp/surefirebooter171.jar
            Running org.jboss.test.virtual.test.URLConnectionUnitTestCase
            Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.046 sec
            [INFO] Building jar: /tmp/surefirebooter174.jar
            Running org.jboss.test.virtual.test.CopyJARCacheUnitTestCase
            Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.08 sec
            [INFO] Building jar: /tmp/surefirebooter177.jar
            Running org.jboss.test.virtual.test.FileVFSContextUnitTestCase
            Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.139 sec
            [INFO] Building jar: /tmp/surefirebooter180.jar
            Running org.jboss.test.virtual.test.URLResolutionUnitTestCase
            Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.05 sec
            [INFO] Building jar: /tmp/surefirebooter183.jar
            Running org.jboss.test.virtual.test.MemoryTestCase
            Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.181 sec
            [INFO] Building jar: /tmp/surefirebooter186.jar
            Running org.jboss.test.virtual.test.AssembledContextTestCase
            Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.247 sec
            [INFO] Building jar: /tmp/surefirebooter189.jar
            Running org.jboss.test.virtual.test.JARVFSContextUnitTestCase
            Tests run: 23, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.195 sec
            [INFO] Building jar: /tmp/surefirebooter192.jar
            Running org.jboss.test.virtual.test.CopyJARSerializationUnitTestCase
            Tests run: 4, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 0.17 sec <<< FAILURE!
            [INFO] Building jar: /tmp/surefirebooter195.jar
            Running org.jboss.test.virtual.test.JARCacheUnitTestCase
            Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.123 sec
            [INFO] Building jar: /tmp/surefirebooter198.jar
            Running org.jboss.test.virtual.test.URLExistsUnitTestCase
            Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.069 sec
            
            Results :
            
            Tests in error:
            
            Tests run: 504, Failures: 0, Errors: 28, Skipped: 0
            



            • 3. Re: Alternative vfs jar implementation
              Scott Stark Master

              These must not be expected since mvn won't create the artifacts due to the test failures.

              • 4. Re: Alternative vfs jar implementation
                Marko Strukelj Master


                Yes, test failures are expected at the moment. I'm still working on those :)

                In the mean time I use:

                mvn -Dmaven.test.skip=true package

                • 5. Re: Alternative vfs jar implementation
                  Sacha Labourey Master

                  Great if it works! you are on fire Marko!

                  • 6. Re: Alternative vfs jar implementation
                    Marko Strukelj Master


                    He he he :)

                    I'm pretty sure it 'works'. The real question is if it works enough. I could only test it so much - with seam apps still not deploying on jboss trunk ...

                    Anyway I think that whatever won't exactly work, can be fixed or modified rather quickly.

                    Cheers,

                    - marko

                    • 7. Re: Alternative vfs jar implementation
                      Marko Strukelj Master

                      Ok ... I fixed most of the unit test failures (not yet commited). There is one remaining category of failing tests that I need an opinion about.

                      My ZipEntryContext rewrites a URL used to create it to always have vfszip: schema in order to ensure proper handling when accessing via URL. There are now 7 tests failing with variations of this issue:

                      ERROR: junit.framework.AssertionFailedError: expected:<vfsfile:/C:/Users/Devel/svnroot/jboss-vfs-alter/some.jar> but was:<vfszip:/C:/Users/Devel/svnroot/jboss-vfs-alter/some.jar>

                      A schema mismatch.

                      What happens here is that vfs context is created on directory. Then findChild() is called to get a jar child. It's expected that child's URL is directory URL + '/' + childName. But in this case child is a mount point of ZipEntryContext into FileSystemContext and has a different URL schema.

                      The question is if this expectation is part of the contract, and if it has some specific use-case behind it.

                      Seems to me that VirtualFileHandler.toURL() should be free to return whatever as long as the URL leads to a correct virtual file.

                      I would go and fix these tests to compare URL file part, but not schema. Does that sound ok?

                      cheers,

                      - marko

                      • 8. Re: Alternative vfs jar implementation
                        Scott Stark Master

                        Not sure if the directory url syntax is at issue here, but that a directory URL ends in '/' is out of the URL rfc/spec as far as I know. The problem is that resolution of relative paths against a directory URL vs non-directory URL are different, see for example this faq:
                        http://www.iusmentis.com/technology/www/relativeurls/

                        If the issue is just about the expectation of a {vfsfile:/.../}.findChild also returning a vfsfile url, that really is not part of the vfs contract. I would drop the schema portion and compare the remainder as you suggest.

                        • 9. Re: Alternative vfs jar implementation
                          Marko Strukelj Master


                          Yes, the issue is just about findChild returning vfsfile url.

                          I'll fix the tests then. Tnx.

                          - marko

                          • 10. Re: Alternative vfs jar implementation
                            Marko Strukelj Master

                            jboss-vfs-alter branch now passes all the unit tests.

                            In AbstractVFSContextTest and FileVFSUnitTestCase I changed assertions that test URL equality. When ZipEntryContext is used URL schema changes, and those tests fail - I changed them so that only URL path part is compared.

                            In UnpackTestCase I changed some assertions since the previous ones test some implementation details that are different when it comes to ZipEntryContext. Ales, have a look, maybe we can put some more sensible tests there.

                            I also commited an optimization of the way file handles are released. I mentioned that it was too aggressive and causing a slight performance penalty. The slow down was not big, but was still noticable so I added an automatic asynchronous mechanism that releases files when they haven't been used for a few seconds. Currently this period is 5 seconds - hardcoded at the moment - can be made configurable. This fixes maybe about 80% of performance degradation which now looks to be ~1-2% jboss startup time relative to current vfs trunk.

                            I'm starting working on noCopy support now :)

                            Cheers,

                            - marko

                            • 11. Re: Alternative vfs jar implementation
                              Dimitris Andreadis Master

                               

                              This fixes maybe about 80% of performance degradation which now looks to be ~1-2% jboss startup time relative to current vfs trunk.

                              Optimizations of this type often introduce side effects, so this better be configurable in case we want to turn this off.

                              Nice work, btw.

                              • 12. Re: Alternative vfs jar implementation
                                Marko Strukelj Master

                                 

                                Optimizations of this type often introduce side effects, so this better be configurable in case we want to turn this off.


                                True, I'll make it configurable.

                                Nice work, btw.


                                Tnx :)

                                • 13. Re: Alternative vfs jar implementation
                                  Ales Justin Master

                                   

                                  "mstruk" wrote:

                                  In UnpackTestCase I changed some assertions since the previous ones test some implementation details that are different when it comes to ZipEntryContext. Ales, have a look, maybe we can put some more sensible tests there.

                                  OK, I'll have a look once full merge is done, e.g. NoCopy is included, ...
                                  I suspect it's an issue with trunk having NoCopy by default, and when we read the nested jars, your impl already copies the file, where in our trunk case we don't - and that's what the test expects - hence assertEqual.

                                  "mstruk" wrote:

                                  I also commited an optimization of the way file handles are released. I mentioned that it was too aggressive and causing a slight performance penalty. The slow down was not big, but was still noticable so I added an automatic asynchronous mechanism that releases files when they haven't been used for a few seconds. Currently this period is 5 seconds - hardcoded at the moment - can be made configurable. This fixes maybe about 80% of performance degradation which now looks to be ~1-2% jboss startup time relative to current vfs trunk.

                                  I don't see why this asynch behavior is necessary.
                                  OK, I guess there must be a good reason for it, otherwise you wouldn't include it.
                                  Just couldn't find any good explanation of it in this tread. ;-)

                                  • 14. Re: Alternative vfs jar implementation
                                    Marko Strukelj Master

                                     

                                    I don't see why this asynch behavior is necessary.


                                    Maybe I should explain a little :)

                                    There is a tradeoff between performance and releasing of file locks when it comes to zip files. Every opening of a zip file comes with an overhead. So - to maximize performance, you open a zip file (acquire a lock) and never release it.

                                    That's how current trunk works, and we don't like it for obvious reasons.

                                    So I implemented the other extreme. For every file that was read from a zip archive a zip file was opened, file was read, and then a zip file was closed right away. This resolved the locking problem but came with a performance penalty - reopening a zip file for every entry.

                                    To have the best of both worlds the obvious solution is to not close a zip file immediately after every use, so that the next time an already open file can be reused, but still close it after a few seconds so that locks are being released.

                                    Now, you have no guarantee after any call to VFS that another call will occur within a reasonable time. If the last call left the zip file open the file may remain open for a long time.

                                    There are only two alternatives that I see here - use a third thread, or mandate at the level of VFS API that clients need to do polling - either through existing method - like getLastModified() - or through special method.

                                    The third-thread-option seems less nasty to me than relying on API client for proper operation. Although I suppose MC's VFS deployer does a regular polling and maybe could be relied upon in this matter? Also consider that release period will then be defined by MC's polling period in that it can only be a period of that, and can't be shorter than that.

                                    My implementation on the other hand is completely autonomous. It uses a daemon timer thread, that autodetects lack of activity and cancels itself out - getting reinitialized when activity reoccurs. So a third thread only exists for a brief period of time - at start up and at redeployment, other times it's not even there.

                                    - marko


                                    1 2 Previous Next