1 2 Previous Next 22 Replies Latest reply on May 23, 2006 10:54 AM by smarlow

    Random OutOfMemoryError on the initial deploy of conf/jboss-

    dimitris

      (Moving here from private discussion)

      This is relatively easy to reproduce by using any out of-the-box jboss v4.0.2/4.0.3SP1 on a linux/sun 64bit 1.5.0_05 jdk (e.g. dev17).

      One out of ten times, or so, we get an OOM Error just when the UCL associated with conf/jboss-service.xml and all the server/libs is trying to open tmp/deploy/tmpNNNNNjboss-service.xml with java.util.zip.ZipFile(), looking for the very first mbean class referenced in conf/jboss.service.xml.

      The whole thing starts with MainDeployer.deploy(file: ... /conf/jboss-service.xml)

      conf/jboss-service.xml (or any other deployment be it a packaged one or a simple xml descriptor) is copied by default due to the MainDeployer copyFiles==true, but this is not really the problem: I tried with copyFiles==false and still get the OOM Error.

      Now, either the original (conf/jboss-service.xml) or the copy
      (tmp/deploy/tmp31494jboss-service.xml) get on the classpath
      because we wrap the deployed URL with a DeploymentInfo and
      then we do a:

      // initialize the unified classloaders for this deployment
      deployment.createClassLoaders();

      That creates the UCL pointing to the xml descriptor:

      // the classes are passed to a UCL that will share the classes with the whole base
      Object[] args = {localUrl, origUrl, Boolean.TRUE};
      String[] sig = {"java.net.URL", "java.net.URL", "boolean"};
      ucl = (RepositoryClassLoader)
      server.invoke(repositoryConfig.repositoryName,
      "newClassLoader",args, sig);

      Onto which the referenced lib/*.jars are added:

      // Add any library jars seen before the UCL was created
      if( classpath.size() > 0 )
      {
      Iterator jars = classpath.iterator();
      while( jars.hasNext() )
      {
      URL jar = (URL) jars.next();
      ucl.addURL(jar);
      }
      }

      The bootstrap descriptor is not different from any other deployment.

      I'm not sure if a solution would be to pass a *null* URL to
      "newClassLoader", if the DeploymentInfo points to an .xml
      descriptor, to avoid having the base URLClassLoader trying to
      ZipFile.open() it, since we know there won't be any classes/resources in there.

        • 1. Re: Random OutOfMemoryError on the initial deploy of conf/jb
          dimitris

          Scott:


          I would think we need to pass in the container directory of the standalone descriptor as the localURL for the case of a standalone, unpackaged sar descriptor. For the unpacked case we are passing in the sar directory correct?


          • 2. Re: Random OutOfMemoryError on the initial deploy of conf/jb
            dimitris

            Correct. But, if I understand, it doesn't seem appropriate to pass in ./conf for conf/jboss-service.xml or ./deploy for deploy/ejb-deployer.xml.

            The OOM error comes from the UCL loader, not when using the URLClassLoader on the DeploymentInfo.localUrl.

            • 3. Re: Random OutOfMemoryError on the initial deploy of conf/jb
              starksm64

              The UCL does not open any resources so I don't know what you mean by by the OOME coming from the UCL. All actual class/resource file opens are done by the parent URLClassLoader so I would say its the non-jar file classpath elements that lead to the problem. I would agree that just putting the descriptor containing directory on the classpath will likely lead to undesirable results.

              • 4. Re: Random OutOfMemoryError on the initial deploy of conf/jb
                dimitris

                Ok, I meant the parent URLClassLoader.

                I was wondering if a URLClassLoader initialized to an empty URL[] is valid, so we can use this for plain xml descriptor deployments (like conf/jboss-service.xml). I guess I can try it.

                • 5. Re: Random OutOfMemoryError on the initial deploy of conf/jb
                  starksm64

                  The first question I have is, does this cause the OOME:

                  File someXML = new File(".../some.xml");
                  URL[] cp = {someXML.toURL()};
                  URLClassLoader ucl = new URLClassLoader(cp);
                  ucl.findResource(...) or ucl.loadClass(...)
                  


                  If it does then pursuing the removal xml descriptor is worth the time. If it does not, I don't think it is.


                  • 6. Re: Random OutOfMemoryError on the initial deploy of conf/jb
                    dimitris

                    Got it! This following test produces OOM quite often (30%) at the very first class lookup.

                    I run it by pointing to the server/all/lib directory.

                    The OOM happens when I drop conf/jboss-service.xml in there.

                    import java.io.File;
                    import java.net.MalformedURLException;
                    import java.net.URL;
                    import java.net.URLClassLoader;
                    import java.util.Enumeration;
                    import java.util.Properties;
                    
                    public class TestOOM
                    {
                     public static void main(String[] args)
                     {
                     outputSystemProperties();
                    
                     File dir = new File(args[0]);
                     File[] files = dir.listFiles();
                     URL[] cp = new URL[files.length];
                     for (int i = 0; i < files.length; i++)
                     {
                     try
                     {
                     cp = files.toURL();
                     System.out.println("#" + i + " " + cp);
                     }
                     catch (MalformedURLException ignore) { }
                     }
                     URLClassLoader ucl = new URLClassLoader(cp);
                    
                     long l = 0;
                     while(true)
                     {
                     try
                     {
                     ucl.findResource("foo.bar.Class" + l);
                     }
                     catch (Throwable t)
                     {
                     // got it!
                     t.printStackTrace();
                     System.out.println("l = " + l);
                     break;
                     }
                     if (l++ % 1000 == 0)
                     System.out.print(".");
                     }
                     }
                    
                     public static void outputSystemProperties()
                     {
                     System.out.println("+++ SYSTEM PROPERTIES BEGIN +++");
                     Properties props = System.getProperties();
                     Enumeration names = props.propertyNames();
                     while (names.hasMoreElements())
                     {
                     String key = (String) names.nextElement();
                     String value = props.getProperty(key);
                     System.out.println(key + "=" + value);
                     }
                     System.out.println("+++ SYSTEM PROPERTIES END +++");
                     }
                     }
                    


                    • 7. Re: Random OutOfMemoryError on the initial deploy of conf/jb
                      starksm64

                      You have to include the other jars in there to cause this to happen?

                      • 8. Re: Random OutOfMemoryError on the initial deploy of conf/jb
                        dimitris

                        Yes, it happens when I point to the server/all/lib jars, plus any other (a) non empty (b) non jar file that (c) appears last in the URL[] list!

                        Weird

                        • 9. Re: Random OutOfMemoryError on the initial deploy of conf/jb
                          dimitris

                          By passing 'null', as the first argument to "newClassLoader", when pointing to an .xml descriptor (in DeploymentInfo createClassLoaders() and setRepositoryInfo()) as shown below:

                           Object[] args = { isXML ? null : localUrl, origUrl, Boolean.TRUE };
                           String[] sig = { "java.net.URL", "java.net.URL", "boolean" };
                           ucl = (RepositoryClassLoader) server.invoke(repositoryConfig.repositoryName,
                           "newClassLoader",args, sig);
                          


                          ... the OOM Error seems to go away in the initial boot phase, or at least, it didn't appear again in my tests.

                          Is there any other implication for not doing this change?


                          • 10. Re: Random OutOfMemoryError on the initial deploy of conf/jb
                            starksm64

                            The main concern will be what is the CodeSource for such a class loader in terms of how security policies are defined. Have you run the testsuite tests-security-manager to validate the behavior at a minimum?

                            • 11. Re: Random OutOfMemoryError on the initial deploy of conf/jb
                              dimitris

                              I tried tests-security-manager and it passes.

                              I am not familiar with CodeSource but I'd think if the UCL is associated with only an .xml descriptor (e.g. ejb-deployer.xml), no classes will ever be loaded from that one.

                              In the case of conf/jboss-service.xml, the URLs for the lib/* jars are added at a next step (ucl.addURL(jar)), so there shouldn't be a problem there.

                              Debugging some more, the above change doesn't eliminate ZipFile.open() on .xml descriptors. I saw ZipFiles created on the content of deploy-hasingleton/jms directory, as a result of an InitialContext() creation in HTTPServerILService.bindJNDIReferences (!). However, I didn't get any OOM Errors for that.

                              So, we can't possibly catch all execution paths to ZipFile.open on non-jar files but if this workaround doesn't do harm maybe we could try the change in HEAD and see who complaints.

                              I'll try reporting the issue to Sun, although I wouldn't expect a fix anytime soon.

                              • 12. Re: Random OutOfMemoryError on the initial deploy of conf/jb
                                smarlow

                                We are seeing the OutOfMemory here at Novell on a AMD 64 bit processor machine. JBoss 4.0.3SP1, Suse (SLES9) OS, "Java 64-Bit Server VM build 1.5.0_06-b05"

                                I couldn't recreate the issue with the standalone TestOOM program (I copied server/all/conf/jboss-service.xml to server/all/lib + entered command: "java TestOOM jboss/server/all/lib".

                                I'll also try to report to Sun and see if there is a fix. If you have a Sun bug number that I can cross reference, please let me know the number.

                                It also doesn't matter whether I set Min/Max Java memory to 512M or use the defaults Java memory settings.

                                • 13. Re: Random OutOfMemoryError on the initial deploy of conf/jb
                                  dimitris

                                  The bug is reported here: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6373059

                                  To get the error with the TestOOM program, you also need to specify (as a last parameter, if I remember) any non-empty, non-jar file (e.g. a test.xml)

                                  java TestOOM jboss-4.0.2/server/all/lib/ test.xml

                                  • 14. Re: Random OutOfMemoryError on the initial deploy of conf/jb
                                    smarlow

                                    I followed your suggestion and could reproduce with the Java 1.5.0_96-b05 (Server VM) build.

                                    I'll cross reference the bug id that you gave me when I report it to Sun.

                                    Thanks,
                                    Scott

                                    1 2 Previous Next