1 2 Previous Next 17 Replies Latest reply on Nov 24, 2008 6:30 PM by alesj

    JBAS-5953; jar override issue

    alesj

      Moving this discussion here:
      - https://jira.jboss.org/jira/browse/JBAS-5953

      I guess this is similar to what we've already seen in some VFS failures:
      - http://www.jboss.com/index.html?module=bb&op=viewtopic&t=134177
      - http://www.jboss.com/index.html?module=bb&op=viewtopic&t=137839
      - http://www.jboss.com/index.html?module=bb&op=viewtopic&t=138343

      Taken from one of these threads:

      "adrian@jboss.org" wrote:

      Also you need to be careful with JAR files and their URL connections.

      The JDK caches them internally and can even mmap them on some operating systems.
      So testing a jar entry has changed isn't the same thing as checking the top level file
      changed.
      Additionally, closing the top jar connection will close the cached version
      which can lead to strange errors.

      The internal caching is the source of the problems (e.g. locking on windows)
      and something Sun have consistently refused to fix. :-(


      Shelly, what exactly do you mean here: "but wanted to check if you have concerns on org.jboss.vfs logic (refer to attached stack trace)"?

      Since jar handling is quite broken in JDK,
      we definitely have some ideas of doing this on our own:
      - https://jira.jboss.org/jira/browse/JBVFS-53

        • 1. Re: JBAS-5953; jar override issue

          I don't see the crash problem that Shelley is seeing,
          but I do see the test failing.

          If I put a sleep between the copy of the file and redeploy,
          then the test passes.

          [ejort@warjort testsuite]$ svn diff
          Index: src/main/org/jboss/test/jmx/test/UndeployBrokenPackageUnitTestCase.java
          ===================================================================
          --- src/main/org/jboss/test/jmx/test/UndeployBrokenPackageUnitTestCase.java (revision 78789)
          +++ src/main/org/jboss/test/jmx/test/UndeployBrokenPackageUnitTestCase.java (working copy)
          @@ -152,6 +152,8 @@
          
           Files.copy(goodjar, thejar);
           getLog().info("Redeploying testPackage: " + testPackage);
          +
          +Thread.sleep(10000);
          
           redeploy(testPackage);
           Object home = getInitialContext().lookup("EntityA");
          


          So this is clearly either a caching issue inside the VFS/JDK
          or there is some issue with the Files.copy() not flushing properly to the file system
          so the server is seeing a corrupt file.

          • 2. Re: JBAS-5953; jar override issue
            alesj

             

            "adrian@jboss.org" wrote:

            So this is clearly either a caching issue inside the VFS ...

            Could be an issue with the way VFS reads
            and then closes things - via delay / reaper thread:
            - http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4153696#4153696

            This was definitely an issue with ProfileService tests,
            where we eventually needed VFS::delete impl:
            - http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4160591#4160591

            But since there are many other similar issues:
            - http://www.jboss.com/index.html?module=bb&op=viewtopic&t=142095
            Looks like we really need some custom jar/zip impl.

            • 3. Re: JBAS-5953; jar override issue
              shelly.mcgowan

              Running locally I've seen this test pass, fail with EntityA not found or crash. These different results can be also seen in the full AS 5 testsuite runs depending on which report you view. I was considering a test code change to avoid the Sun JDK issue but didn't want to overlook an issue with the VFS implementation handling the same-name jar redeployment. Previously looking at org.jboss.virtual.plugins.context.zip.ZipEntryContext.java whether checkIfModified() and initEntries() were correct and, alternatively, as Adrian's points out above, possible issue with Files.copy(). Do you recommend to change the test code to avoid the inconsistent result?

              • 4. Re: JBAS-5953; jar override issue

                 

                "smcgowan@redhat.com" wrote:
                Do you recommend to change the test code to avoid the inconsistent result?


                Not until we understand the cause of the inconsistency. :-)

                This looks more like a server side problem rather than a broken test.
                I looked at Files.copy() and it does flush and close the file.

                If Ales can't find a problem in the VFS, then I'd suggest trying to reproduce it
                with Iced Tea where we can debug the native zip code.

                Or failing that, producing a simple cutdown version of the test to give to Sun
                so they can fix it. You should definitely give them core dump if you can reproduce
                it with the latest JDK6 release.

                • 5. Re: JBAS-5953; jar override issue
                  shelly.mcgowan

                  Failure and JVM crash reproducable with OpenJDK 6. I'll file a bug and include testcase and core dump to Sun.

                  • 6. Re: JBAS-5953; jar override issue
                    mstruk

                    If the archive isn't deleted before the new version is copied over, I'd try that using VirtualFile.delete().

                    I was dealing with a similar issue:

                    - http://www.jboss.com/index.html?module=bb&op=viewtopic&t=138343


                    - marko

                    • 7. Re: JBAS-5953; jar override issue
                      alesj

                       

                      "adrian@jboss.org" wrote:

                      If Ales can't find a problem in the VFS

                      I've added assertTrue(thejar.delete()) before every Files::copy.
                      If I run AS with -Djboss.vfs.noReaper=true test passes every time,
                      w/o that flag it fails on newly added assert before 2nd Files::copy.

                      I already have VFSUtils::enableNoReaper(VirtualFile),
                      but looks like it's kicking in too late.
                      I'm checking if VFSUtils::enableNoReaper(VFS) can do the trick.
                      (need to add that method first :-)

                      • 8. Re: JBAS-5953; jar override issue
                        slaboure

                        If we find no good solution with the JDK layer, why not checking at boot time whether a suitable JBoss Native library has been installed and use that instead?

                        • 9. Re: JBAS-5953; jar override issue
                          dimitris

                          First, independent of the particular failure we should not let one failing test block the execution of a whole testsuite. So the test should be conditionally skipped for this particular jvm version/vendor combination and another simpler test created to expose the underlying problem either in AS or VFS (but again, not holding the whole testsuite).

                          Obviously the usecase of re-deploying a jar is quite common and needs to be addressed. What if we jboss.vfs.forceNoReaper=true by default?

                          Trying to remember what forceNoRepear is supposed to do by lookint at VFSUtils.java I must say I got lost. We have zero documentation on VFS options and their number keeps increasing.

                          http://anonsvn.jboss.org/repos/jbossas/projects/vfs/trunk/src/main/java/org/jboss/virtual/VFSUtils.java

                          • 10. Re: JBAS-5953; jar override issue
                            alesj

                             

                            "dimitris@jboss.org" wrote:

                            Obviously the usecase of re-deploying a jar is quite common and needs to be addressed. What if we jboss.vfs.forceNoReaper=true by default?

                            I would still leave it at false.

                            I was able to do a legit workaround the issue,
                            while global VFS setting is still at NoReaper==false.

                            I add the following piece to UndeployBrokenPackageUnitTestCase:
                             assertTrue(thejar.delete()); // TODO - this should work
                             Files.copy(goodjar, thejar);
                             getLog().info("Redeploying testPackage: " + testPackage);
                            

                            so that assert should complain before if things cannot me mounted together.

                            If I add the following code to old MainDeployer::deploy
                             VFS vfs = VFS.getVFS(url);
                             VFSUtils.enableNoReaper(vfs);
                             VirtualFile file = vfs.getRoot();
                             VFSDeployment deployment = deploymentFactory.createVFSDeployment(file);
                            

                            the test passes w/o any problems.

                            In this case it's actually the file openness that prevents it from being overridden.
                            Reaper is holding it open for 5sec to get better read performance,
                            but the client == test tries to override it in less than 5sec.

                            I think users don't really intend to re-deploy things in less than 5sec,
                            so we should be fine. :-)

                            If this becomes a more serious issue for users,
                            we can always add some option to ProfileImpl,
                            to either use VFSUtils + VFS to enable/disable certain options.

                            • 11. Re: JBAS-5953; jar override issue
                              dimitris

                              Every we create a VFS optimization are we going to change the MainDeployer interface?

                              Adding extended deploy method to old MainDeployer:
                              
                               /**
                               * The <code>deploy</code> method deploys a package identified by a URL
                               *
                               * @param url an <code>URL</code> value
                               * @param noReaper enable no reaper
                               * @param forceCopy enable copy
                               * @param caseSensitive enable case sensitive
                               * @throws DeploymentException for any error
                               */
                               void deploy(URL url, boolean noReaper, boolean forceCopy, boolean caseSensitive) throws DeploymentException;
                              
                              Invoking deploy with noReaper==true allows testjar file deletion.
                              


                              • 12. Re: JBAS-5953; jar override issue
                                alesj

                                 

                                "dimitris@jboss.org" wrote:
                                Every we create a VFS optimization are we going to change the MainDeployer interface?

                                How does Map work with remote JMX call?
                                e.g.
                                void deploy(URL url, Map<String, String> options);
                                


                                • 13. Re: JBAS-5953; jar override issue
                                  dimitris

                                  Ok, I'll say it differently. The real problem is letting internal details, like VFS options, leak to the client.

                                  The client only cares about deploying a URL. He doesn't care how this should be done or what is VFS and especially not what VFS options would be needed to workaround a particular problem.

                                  If we need to workaround the problem for this particular testcase, that's easy, just introduce a small delay and a note pointing to a JIRA with future work on this.

                                  Extending the MainDeployer api with implentation details doesn't offer any real value, it just complicates things.

                                  And BTW, I think all VFS options should be visible in bootstrap/vfs.xml with the user able to override them in the command line, or by modifying the file.

                                  • 14. Re: JBAS-5953; jar override issue
                                    starksm64

                                    It should be the deployment that contains the metadata to setup the vfs behavior as needed, not the deployer call.

                                    1 2 Previous Next