Random OutOfMemoryError on the initial deploy of conf/jboss-

[dimitris]

(Moving here from private discussion)

This is relatively easy to reproduce by using any out of-the-box jboss v4.0.2/4.0.3SP1 on a linux/sun 64bit 1.5.0_05 jdk (e.g. dev17).

One out of ten times, or so, we get an OOM Error just when the UCL associated with conf/jboss-service.xml and all the server/libs is trying to open tmp/deploy/tmpNNNNNjboss-service.xml with java.util.zip.ZipFile(), looking for the very first mbean class referenced in conf/jboss.service.xml.

The whole thing starts with MainDeployer.deploy(file: ... /conf/jboss-service.xml)

conf/jboss-service.xml (or any other deployment be it a packaged one or a simple xml descriptor) is copied by default due to the MainDeployer copyFiles==true, but this is not really the problem: I tried with copyFiles==false and still get the OOM Error.

Now, either the original (conf/jboss-service.xml) or the copy
(tmp/deploy/tmp31494jboss-service.xml) get on the classpath
because we wrap the deployed URL with a DeploymentInfo and
then we do a:

// initialize the unified classloaders for this deployment
deployment.createClassLoaders();

That creates the UCL pointing to the xml descriptor:

// the classes are passed to a UCL that will share the classes with the whole base
Object[] args = {localUrl, origUrl, Boolean.TRUE};
String[] sig = {"java.net.URL", "java.net.URL", "boolean"};
ucl = (RepositoryClassLoader)
server.invoke(repositoryConfig.repositoryName,
"newClassLoader",args, sig);

Onto which the referenced lib/*.jars are added:

// Add any library jars seen before the UCL was created
if( classpath.size() > 0 )
{
Iterator jars = classpath.iterator();
while( jars.hasNext() )
{
URL jar = (URL) jars.next();
ucl.addURL(jar);
}
}

The bootstrap descriptor is not different from any other deployment.

I'm not sure if a solution would be to pass a *null* URL to
"newClassLoader", if the DeploymentInfo points to an .xml
descriptor, to avoid having the base URLClassLoader trying to
ZipFile.open() it, since we know there won't be any classes/resources in there.

[dimitris]

Scott:

I would think we need to pass in the container directory of the standalone descriptor as the localURL for the case of a standalone, unpackaged sar descriptor. For the unpacked case we are passing in the sar directory correct?

[dimitris]

Correct. But, if I understand, it doesn't seem appropriate to pass in ./conf for conf/jboss-service.xml or ./deploy for deploy/ejb-deployer.xml.

The OOM error comes from the UCL loader, not when using the URLClassLoader on the DeploymentInfo.localUrl.

[starksm64]

The UCL does not open any resources so I don't know what you mean by by the OOME coming from the UCL. All actual class/resource file opens are done by the parent URLClassLoader so I would say its the non-jar file classpath elements that lead to the problem. I would agree that just putting the descriptor containing directory on the classpath will likely lead to undesirable results.

[dimitris]

Ok, I meant the parent URLClassLoader.

I was wondering if a URLClassLoader initialized to an empty URL[] is valid, so we can use this for plain xml descriptor deployments (like conf/jboss-service.xml). I guess I can try it.

[starksm64]

The first question I have is, does this cause the OOME:

File someXML = new File(".../some.xml");
URL[] cp = {someXML.toURL()};
URLClassLoader ucl = new URLClassLoader(cp);
ucl.findResource(...) or ucl.loadClass(...)

If it does then pursuing the removal xml descriptor is worth the time. If it does not, I don't think it is.

[dimitris]

Got it! This following test produces OOM quite often (30%) at the very first class lookup.

I run it by pointing to the server/all/lib directory.

The OOM happens when I drop conf/jboss-service.xml in there.

import java.io.File;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.Enumeration;
import java.util.Properties;

public class TestOOM
{
 public static void main(String[] args)
 {
 outputSystemProperties();

 File dir = new File(args[0]);
 File[] files = dir.listFiles();
 URL[] cp = new URL[files.length];
 for (int i = 0; i < files.length; i++)
 {
 try
 {
 cp = files.toURL();
 System.out.println("#" + i + " " + cp);
 }
 catch (MalformedURLException ignore) { }
 }
 URLClassLoader ucl = new URLClassLoader(cp);

 long l = 0;
 while(true)
 {
 try
 {
 ucl.findResource("foo.bar.Class" + l);
 }
 catch (Throwable t)
 {
 // got it!
 t.printStackTrace();
 System.out.println("l = " + l);
 break;
 }
 if (l++ % 1000 == 0)
 System.out.print(".");
 }
 }

 public static void outputSystemProperties()
 {
 System.out.println("+++ SYSTEM PROPERTIES BEGIN +++");
 Properties props = System.getProperties();
 Enumeration names = props.propertyNames();
 while (names.hasMoreElements())
 {
 String key = (String) names.nextElement();
 String value = props.getProperty(key);
 System.out.println(key + "=" + value);
 }
 System.out.println("+++ SYSTEM PROPERTIES END +++");
 }
 }

[starksm64]

You have to include the other jars in there to cause this to happen?

[dimitris]

Yes, it happens when I point to the server/all/lib jars, plus any other (a) non empty (b) non jar file that (c) appears last in the URL[] list!

Weird

[dimitris]

By passing 'null', as the first argument to "newClassLoader", when pointing to an .xml descriptor (in DeploymentInfo createClassLoaders() and setRepositoryInfo()) as shown below:

 Object[] args = { isXML ? null : localUrl, origUrl, Boolean.TRUE };
 String[] sig = { "java.net.URL", "java.net.URL", "boolean" };
 ucl = (RepositoryClassLoader) server.invoke(repositoryConfig.repositoryName,
 "newClassLoader",args, sig);

... the OOM Error seems to go away in the initial boot phase, or at least, it didn't appear again in my tests.

Is there any other implication for not doing this change?

[starksm64]

The main concern will be what is the CodeSource for such a class loader in terms of how security policies are defined. Have you run the testsuite tests-security-manager to validate the behavior at a minimum?

[dimitris]

I tried tests-security-manager and it passes.

I am not familiar with CodeSource but I'd think if the UCL is associated with only an .xml descriptor (e.g. ejb-deployer.xml), no classes will ever be loaded from that one.

In the case of conf/jboss-service.xml, the URLs for the lib/* jars are added at a next step (ucl.addURL(jar)), so there shouldn't be a problem there.

Debugging some more, the above change doesn't eliminate ZipFile.open() on .xml descriptors. I saw ZipFiles created on the content of deploy-hasingleton/jms directory, as a result of an InitialContext() creation in HTTPServerILService.bindJNDIReferences (!). However, I didn't get any OOM Errors for that.

So, we can't possibly catch all execution paths to ZipFile.open on non-jar files but if this workaround doesn't do harm maybe we could try the change in HEAD and see who complaints.

I'll try reporting the issue to Sun, although I wouldn't expect a fix anytime soon.

[smarlow]

We are seeing the OutOfMemory here at Novell on a AMD 64 bit processor machine. JBoss 4.0.3SP1, Suse (SLES9) OS, "Java 64-Bit Server VM build 1.5.0_06-b05"

I couldn't recreate the issue with the standalone TestOOM program (I copied server/all/conf/jboss-service.xml to server/all/lib + entered command: "java TestOOM jboss/server/all/lib".

I'll also try to report to Sun and see if there is a fix. If you have a Sun bug number that I can cross reference, please let me know the number.

It also doesn't matter whether I set Min/Max Java memory to 512M or use the defaults Java memory settings.

[dimitris]

The bug is reported here: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6373059

To get the error with the TestOOM program, you also need to specify (as a last parameter, if I remember) any non-empty, non-jar file (e.g. a test.xml)

java TestOOM jboss-4.0.2/server/all/lib/ test.xml

[smarlow]

I followed your suggestion and could reproduce with the Java 1.5.0_96-b05 (Server VM) build.

I'll cross reference the bug id that you gave me when I report it to Sun.

Thanks,
Scott