1 2 3 4 Previous Next 58 Replies Latest reply on Apr 10, 2013 6:16 AM by stuartdjames

    -Djboss.vfs.forceNoCopy=true not working AS 5.0.0.GA

    rnicholson10

      Running on x86/linux, setting the above flag does not seem to stop jars filling up the "jboss/server/default/tmp/vfs-nested.tmp/" directory. We hit a point where we had 72GB of jar files in this directory.

      Does anyone have a fix or workaround for this problem?

      Thanks

        • 1. Re: -Djboss.vfs.forceNoCopy=true not working AS 5.0.0.GA
          rnicholson10

          Some more information on this problem.

          It would appear that this directory maintains a constant size for a time. Then all of a sudden it will start reproducing the same jars once every couple of seconds. Once this starts happening the size of the directory increases hugely in size. So much so that even running the "ls" command in the directory will fail.

          I am now aware that the flag "jboss.vfs.forceNoCopy" should be enabled by default in AS5.0.0.GA. And instead we have "jboss.vfs.forceCopy" option to switch it on again.

          Short of writing a script to constantly empty this directory every few minutes I don't see how we can use this release in a production environment.

          • 2. Re: -Djboss.vfs.forceNoCopy=true not working AS 5.0.0.GA
            mstruk


            At the time you're seeing uncontrolled proliferation of jars, are you also seeing your application being redeployed?

            - marko

            • 3. Re: -Djboss.vfs.forceNoCopy=true not working AS 5.0.0.GA
              rnicholson10

              No, this is a production system, and so there are no redeployments.

              Interestingly though, we also have a dev system and after several deployments the same thing happens.


              If I set the flag -Djboss.vfs.forceCopy=false, then the tmp directory is not used, and our problem goes away.

              I think that the forceNoCopy option is not set by default as it should be. Now I'm thinking that maybe the -Djboss.vfs.forceNoReaper=false could be used instead in case the reaper was never started to clean jars out of the tmp directory.

              As some defaults are wrong maybe others are too!

              Any other ideas would be welcome.

              • 4. Re: -Djboss.vfs.forceNoCopy=true not working AS 5.0.0.GA
                rnicholson10

                 

                "mstruk" wrote:

                At the time you're seeing uncontrolled proliferation of jars, are you also seeing your application being redeployed?

                - marko


                Actually, it would appear that we were seeing an uncontrolled proliferation of jars. Just talking to lads in charge of the production systems and they say that every couple of mins new jars would be deployed by JBoss itself. This problem may be bigger than we thought.

                Is there a reason why JBoss would redeploy jars? Surely once they are deployed once that should be enough?

                • 5. Re: -Djboss.vfs.forceNoCopy=true not working AS 5.0.0.GA
                  alesj

                  You're seeing the vfs hits due to cache expire.
                  In this case perhaps change the VFS cache impl to LRU cache.

                  See vfs.xml in conf/bootstrap.
                  e.g.

                   <bean name="VFSCache">
                   <constructor factoryClass="org.jboss.virtual.spi.cache.VFSCacheFactory" factoryMethod="getInstance">
                   <!-- Use the IterableTimedVFSCache implementation -->
                   <parameter>org.jboss.virtual.plugins.cache.LRUVFSCache</parameter>
                   <parameter>
                   <map keyClass="java.lang.String" valueClass="java.lang.Object">
                   <entry>
                   <key>jboss.vfs.cache.LRUPolicyCaching.min</key>
                   <value>2</value>
                   </entry>
                   <entry>
                   <key>jboss.vfs.cache.LRUPolicyCaching.max</key>
                   <value>10</value>
                   </entry>
                   </map>
                   </parameter>
                   </constructor>
                   <start ignored="true"/>
                   </bean>
                  



                  • 6. Re: -Djboss.vfs.forceNoCopy=true not working AS 5.0.0.GA
                    rnicholson10

                    Does this mean I shouldn't use

                    -Djboss.vfs.forceNoReaper=false

                    or
                    -Djboss.vfs.forceCopy=false


                    and just change the VFS Cahe impl instead?

                    • 7. Re: -Djboss.vfs.forceNoCopy=true not working AS 5.0.0.GA
                      dimitris

                       

                      "alesj" wrote:
                      You're seeing the vfs hits due to cache expire.
                      In this case perhaps change the VFS cache impl to LRU cache
                      ....


                      Can you elaborate?

                      • 8. Re: -Djboss.vfs.forceNoCopy=true not working AS 5.0.0.GA
                        alesj

                         

                        "dimitris@jboss.org" wrote:
                        "alesj" wrote:
                        You're seeing the vfs hits due to cache expire.
                        In this case perhaps change the VFS cache impl to LRU cache
                        ....


                        Can you elaborate?


                        The timed cache expires in 30min.
                        After that each new URL --> (VFS)URLConnection hit creates new context since there is no matching cache entry.

                        But LRU doesn't expire, it's just limited.
                        Currently we have 5-6 context entries, which mostly cover all the possible roots.
                        My suggestion would be to change the default cache impl to LRU.

                        • 9. Re: -Djboss.vfs.forceNoCopy=true not working AS 5.0.0.GA
                          rnicholson10

                          alesj,

                          Would recommend I leave out the startup flags and use LRU cache instead?

                          • 10. Re: -Djboss.vfs.forceNoCopy=true not working AS 5.0.0.GA
                            alesj

                             

                            "rnicholson10" wrote:

                            Would recommend I leave out the startup flags and use LRU cache instead?

                            You can try. :-)
                            And let us know if it makes a difference.

                            The other flags should be OK,
                            but t's up to you on how you handle your env.

                            NoCopy is more about performance and memory -
                            files and direct access vs. nested entries and cached bytes.
                            Reaper==false makes sense if you're gonna do quick changes,
                            hence no delayed file locking should be used.

                            • 11. Re: -Djboss.vfs.forceNoCopy=true not working AS 5.0.0.GA
                              rnicholson10

                              We are now using just the -Djboss.vfs.forceNoReaper=false flag to make sure the reaper is running. We had to remove -Djboss.vfs.forceCopy=false flag as it has serious performance issues.

                              We will try the LRU Cache for VFS next and see how this helps. Using the reaper flag new jars are still created every 30 minutes and the old jars are never removed. So theoretically this would eventually fill up the disk.

                              I'll let you know how things go with LRU Cache.

                              • 12. Re: -Djboss.vfs.forceNoCopy=true not working AS 5.0.0.GA
                                bdean42

                                We are having the same problem. Setting the flag -Djboss.vfs.forceCopy=false works as far as keeping our disks from filling up. Without that, using all the defaults for JBoss 5.0.0.GA, the disks on our servers fill up every day or so. Note that these are development servers so our CI server redeploys fairly often. Nevertheless, we didn't have any problems like this with our previous JBoss version (4.2.2).

                                A application server that we have to keep an eye on and restart every day or so isn't a very viable solution.

                                Also, I really have no idea what performance implications there are to turning this VFS stuff on or off. Why is the default setting in 5.0.0.GA something that will in sort order fill up a disk? That certainly can't be the intended behavior can it?

                                • 13. Re: -Djboss.vfs.forceNoCopy=true not working AS 5.0.0.GA
                                  bdean42

                                  Also, I have noticed that during the past day since setting the -Djboss.vfs.forceCopy=false flag, JBoss 5.0.0.GA has crashed a couple times with an OutOfMemoryException for running out of heap space.

                                  Does turning off the vfs forceCopy make vfs do ... whatever it does, in memory?

                                  Also, I think it would be beneficial to me and maybe others if there was some explanation of what this vfs copy stuff is doing and why. This all seemed to work very well in JBoss 4 and now in 5 it doesn't seem to be stable at all.

                                  • 14. Re: -Djboss.vfs.forceNoCopy=true not working AS 5.0.0.GA
                                    alesj

                                     

                                    "bdean42" wrote:

                                    Does turning off the vfs forceCopy make vfs do ... whatever it does, in memory?

                                    "vfs force copy == false" handles all nested jars/zips in memory --> very memory consuming (if done wrong).

                                    This alone doesn't do what you want ( = less copies).
                                    Why?
                                    Since after the cache entries expire (~ 30min),
                                    every "URL --> VirtualFile" hit for nested resources/entries will create a new VFS root + create an instance whole hierarchy representation.

                                    In in-memory case there will be plenty of duplicated instances,
                                    in force-copy case there will be plenty of temp files.

                                    Both is bad. :-(

                                    I've got the fix locally, just need to create some proper long running tests.
                                    But this will be definitely included in the up-coming 5.0.1.

                                    "bdean42" wrote:

                                    Also, I think it would be beneficial to me and maybe others if there was some explanation of what this vfs copy stuff is doing and why.

                                    VFS copy just makes a temp copy of nested jar/zip.
                                    It's easy to handle direct jars/zips, where nested stuff is hard to do with plain JDK.

                                    "bdean42" wrote:

                                    This all seemed to work very well in JBoss 4 and now in 5 it doesn't seem to be stable at all.

                                    In JBoss4 everything was a temp copy.
                                    With JBoss5 we went with VFS, to make dev life easier - an unified view of the file system.
                                    With all sort of useful tweaks - caching, re-reading, case-sensitivity ... - it was just unfortunate that we missed on this issue.

                                    The proper solution is
                                    - have long running tests
                                    - combined cache (pre-init is permanent, other new entries are timed or lru)

                                    I'll commit the stuff asap, and let you know how to use it:
                                    - updating VFS
                                    - changing vfs.xml

                                    HTH?


                                    1 2 3 4 Previous Next