1 2 Previous Next 15 Replies Latest reply on Dec 10, 2006 5:53 PM by starksm64

    vfs* based URLs

    bill.burke

      About a month ago I did a prototype on my laptop of changing VirtualFile.toURL() to return a VFS based URL. I got it up in working, but ran into a lot of problems when using VFS based URLs with a URLClassLoader.

      The way I had it working was that a URL Handler returned a VirtualFileURLConnection object. This class delegated to a VirtualFile to openStreams and such. VirtualFile.toURL() would return a "vfs" base URL. While VirtualFileHandlers remained untouched and returned their real underlying URL.

      For Jars I ran into a lot of Streaming problems where there would be pre-mature end-of-file errors when loading classes. This was solved by creating ByteArrayInputStreams around any loading of jar entries. I also found that the URL Classloader implementation worked differently depending on whether the URL ended in "/" or not.

      What scared me the most about this process was that I had to hack Sun specific by decompiling it to understand what the problems were as most of the behaviors I had to debug were not documented.

      I'm doing a second iteration of this code right now. One thing I'm going to try to do is to not return a VirtualFileURLConnection from the VFS protocol Handler, but instead return VirtualFileHandler.toURL().openConnection(). I think this may solve a lot of the problems that were encountered initially.

        • 1. Re: vfs* based URLs
          starksm64

          My concern with this is that we are still subject to the behavior of the underlying protocol handler and its implementation details. I'm still trying to drill into the current readBytes error coming from the FileInputStream native method on win32 and not having access to the actual error is tedious. The same type of issue exists for JarFile where a lot of detail is in native methods.

          The first thing we need is a java.util.jar.JarFile abstraction so we can replace its usage.

          • 2. Re: vfs* based URLs
            bill.burke

            I personally don't like the idea of rewriting and maintaining stuff that already exists, tested and used by millions of people within the JDK. I think if we move to VFS based ClassLoaders a lot of the problems I saw at least will become moot.

            • 3. Re: vfs* based URLs
              starksm64

              Are you adding the JarFile abstraction or not?

              • 4. Re: vfs* based URLs
                bill.burke

                Yes, I can do that.

                • 5. Re: vfs* based URLs
                  starksm64

                  Also, there is no native toURL().openConnection() for a nested jar. It cannot be a requirement that we unpack nested jars as is currently being done. How are you going to handle this type of vf?

                  • 6. Re: vfs* based URLs
                    bill.burke

                    I can look into that too. One option I guess is to load the jar into a byte array and have pointers into that array for the entries, returning BAIS, when needed. This seems like a memory hog though as you have to have the contents of the Jar in memory. I wonder if JarFile and the URLClassLoaders that use them are efficient in that regard? Using a stream interface so that they don't have to keep the raw bytes in memory. If not, then the bytearray approach probably works great.

                    I think thats sorta what you're already doing with the current implementation that is there.

                    • 7. Re: vfs* based URLs
                      starksm64

                      Yes, that is the current no copy nested jar impl. The total compressed memory of all jars is not that much for a server class machine, so having an in memory ram jar file implementation is something we should be able to support.

                      • 8. Re: vfs* based URLs
                        bill.burke

                        almost done with this... I'm running into other non-related problems that prevent me from doing adequet testing

                        • 9. Re: vfs* based URLs
                          bill.burke

                          I have finished "vfs" URL based support. I have tested it with the virtual testsuite and a snapshot of JBossas head. I need to test it with the latest head before I make it official.

                          Code is committed in microcontainer SVN.

                          • 10. Re: vfs* based URLs
                            bill.burke

                            For "vfs" based urls to be able to work efficiently in things like URLClassLoaders, I need to cache VFSs and the children of VirtualFiles. If you don't do this, then jar files are continuously being reprocessed and things grind to a hault.

                            The protocol Handler for vfs based URLs returns a VirtualFileURLConnection. If the setUseCaches is true, it caches the base VFS of the URL you are looking up. If not, the cache is not used and a new VFS is done.

                            VirtualFile.getChildren, findChildren, etc... FileHandler keeps a cache of children. Whenver find/get is called, the FileHandler looks in its cache but will also make sure that the file has not changed on disk since it was cached. If it has, it will remove that item from the cache and create a new handler.

                            This requires me to cache the timestamp of the file when the virtualfilehandler is created so that it can be compared with the latest timestamp of the file.

                            If we dont' do this timestamp comparison, then the cache will have old items when things are redeployed.

                            • 11. Re: vfs* based URLs
                              starksm64

                              I still need to get to the bottom of the ZipFile corruption issue on redeployment. Have you run into this working on the vfs urls?

                              • 12. Re: vfs* based URLs
                                bill.burke

                                I thought removing finalize from VirtualFile fixed this problem? I have not yet run into ZipFile corruption issues.

                                • 13. Re: vfs* based URLs
                                  starksm64

                                  I would doubt it as it was a reproducible issue that depended on modifying the contents of a test deployment and redeploying it. I'll look into this weekend.

                                  • 14. Re: vfs* based URLs
                                    starksm64

                                    There are 9 testcase failures in the mc deployers due to the vfsurl changes. These tests need to be run when updating the vfs as this is the major consumer of the vfs.

                                    1 2 Previous Next