startup fails almost immediately with an OutOfMemory

[eiman]

We are running jboss 3.2.5, jdk (build 1.5.0-b64) on a 64-bit opteron running linux.

On occasion we get the following OutOfMemory when starting jboss. The error is sporadic and in all cases when we restart the server the problem goes away.

================================================================================

JBoss Bootstrap Environment

JBOSS_HOME: /export/jboss/3.2.5

JAVA: /usr/java/jdk1.5.0/bin/java

JAVA_OPTS: -Djava.net.preferIPv4Stack=true -Djava.util.prefs.userRoot=/tmp/.java -Djava.util.prefs.systemRoot=/tmp/.java -Dpositionservice.ignorenodes=legalentity,ram=trading -Dpositionservice.pricepreloadstartdate=12/31/2003 -Drcm.valuation=remote -Drcm.valuationservice.explicit=true -Dpositionservice.preloadprices=true -server -Dpositionservice.preloadtrades=true -Dpositionservice.precalculatepositions=true -Dpositionservice.otcworkers=12 -XX:+DisableExplicitGC -DRCM_ENV=production -verbose:gc -XX:NewSize=256m -Xmx4500m -Xms4500m -Dprogram.name=jboss_run.sh

CLASSPATH: /export/jboss/3.2.5/bin/run.jar:/usr/java/jdk1.5.0/lib/tools.jar

================================================================================

00:30:11,978 INFO [Server] Starting JBoss (MX MicroKernel)...
00:30:11,979 INFO [Server] Release ID: JBoss [WonderLand] 3.2.5 (build: CVSTag=JBoss_3_2_5 date=200408121639)
00:30:11,979 INFO [Server] Home Dir: /export/jboss/3.2.5
00:30:11,980 INFO [Server] Home URL: file:/export/jboss/3.2.5/
00:30:11,980 INFO [Server] Library URL: file:/export/jboss/3.2.5/lib/
00:30:11,980 INFO [Server] Patch URL: null
00:30:11,980 INFO [Server] Server Name: position_service_1
00:30:11,981 INFO [Server] Server Home Dir: /export/jboss/3.2.5/server/position_service_1
00:30:11,982 INFO [Server] Server Home URL: file:/export/jboss/3.2.5/server/position_service_1/
00:30:11,983 INFO [Server] Server Data Dir: /export/jboss/3.2.5/server/position_service_1/data
00:30:11,983 INFO [Server] Server Temp Dir: /export/jboss/3.2.5/server/position_service_1/tmp
00:30:11,983 INFO [Server] Server Config URL: file:/export/jboss/3.2.5/server/position_service_1/conf/
00:30:11,983 INFO [Server] Server Library URL: file:/export/jboss/3.2.5/server/position_service_1/lib/
00:30:11,983 INFO [Server] Root Deployment Filename: jboss-service.xml
00:30:11,986 INFO [Server] Starting General Purpose Architecture (GPA)...
00:30:12,191 INFO [ServerInfo] Java version: 1.5.0,Sun Microsystems Inc.
00:30:12,191 INFO [ServerInfo] Java VM: Java HotSpot(TM) 64-Bit Server VM 1.5.0-b64,Sun Microsystems Inc.
00:30:12,192 INFO [ServerInfo] OS-System: Linux 2.6.8-1.521smp,amd64
00:30:12,403 INFO [Server] Core system initialized
00:30:14,789 WARN [NestedThrowable] Duplicate throwable nesting of same base type: class org.jboss.deployment.DeploymentException is assignable from: class org.jboss.deployment.DeploymentException
00:30:14,789 ERROR [MainDeployer] could not create deployment: file:/export/jboss/3.2.5/server/position_service_1/conf/jboss-service.xml
org.jboss.deployment.DeploymentException: create operation failed for package file:/export/jboss/3.2.5/server/position_service_1/conf/jboss-service.xml; - nested throwable: (org.jboss.deployment.DeploymentException: - nested throwable: (java.lang.reflect.UndeclaredThrowableException))
at org.jboss.deployment.SARDeployer.create(SARDeployer.java:227)
at org.jboss.deployment.MainDeployer.create(MainDeployer.java:790)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:644)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:608)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:592)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:61)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:53)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:185)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
at $Proxy5.deploy(Unknown Source)
at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:407)
at org.jboss.system.server.ServerImpl.start(ServerImpl.java:311)
at org.jboss.Main.boot(Main.java:145)
at org.jboss.Main$1.run(Main.java:399)
at java.lang.Thread.run(Thread.java:595)
Caused by: org.jboss.deployment.DeploymentException: - nested throwable: (java.lang.reflect.UndeclaredThrowableException)
at org.jboss.system.ServiceConfigurator.install(ServiceConfigurator.java:139)
at org.jboss.system.ServiceController.install(ServiceController.java:198)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:61)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:53)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:185)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
at $Proxy4.install(Unknown Source)
at org.jboss.deployment.SARDeployer.create(SARDeployer.java:204)
... 21 more
Caused by: java.lang.reflect.UndeclaredThrowableException
at org.jboss.system.ServiceCreator.install(ServiceCreator.java:175)
at org.jboss.system.ServiceConfigurator.internalInstall(ServiceConfigurator.java:149)
at org.jboss.system.ServiceConfigurator.install(ServiceConfigurator.java:114)
... 35 more
Caused by: java.lang.OutOfMemoryError
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:204)
at java.util.jar.JarFile.(JarFile.java:132)
at java.util.jar.JarFile.(JarFile.java:70)
at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:576)
at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:543)
at sun.misc.URLClassPath$3.run(URLClassPath.java:321)
at java.security.AccessController.doPrivileged(Native Method)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:310)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:287)
at sun.misc.URLClassPath.findResource(URLClassPath.java:138)
at java.net.URLClassLoader$2.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findResource(URLClassLoader.java:359)
at java.lang.ClassLoader.getResource(ClassLoader.java:977)
at org.jboss.mx.loading.UnifiedClassLoader.getResourceLocally(UnifiedClassLoader.java:327)
at org.jboss.mx.loading.LoadMgr3.beginLoadTask(LoadMgr3.java:234)
at org.jboss.mx.loading.UnifiedClassLoader3.loadClassImpl(UnifiedClassLoader3.java:178)
at org.jboss.mx.loading.UnifiedClassLoader3.loadClass(UnifiedClassLoader3.java:132)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at org.jboss.mx.server.MBeanServerImpl.instantiate(MBeanServerImpl.java:843)
at org.jboss.mx.server.MBeanServerImpl.instantiate(MBeanServerImpl.java:290)
at org.jboss.mx.server.MBeanServerImpl.createMBean(MBeanServerImpl.java:317)
at org.jboss.system.ServiceCreator.install(ServiceCreator.java:98)
... 37 more
Failed to boot JBoss:
org.jboss.deployment.DeploymentException: create operation failed for package file:/export/jboss/3.2.5/server/position_service_1/conf/jboss-service.xml; - nested throwable: (org.jboss.deployment.DeploymentException: - nested throwable: (java.lang.reflect.UndeclaredThrowableException))
at org.jboss.deployment.SARDeployer.create(SARDeployer.java:227)
at org.jboss.deployment.MainDeployer.create(MainDeployer.java:790)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:644)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:608)
at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:592)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:61)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:53)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:185)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
at $Proxy5.deploy(Unknown Source)
at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:407)
at org.jboss.system.server.ServerImpl.start(ServerImpl.java:311)
at org.jboss.Main.boot(Main.java:145)
at org.jboss.Main$1.run(Main.java:399)
at java.lang.Thread.run(Thread.java:595)
Caused by: org.jboss.deployment.DeploymentException: - nested throwable: (java.lang.reflect.UndeclaredThrowableException)
at org.jboss.system.ServiceConfigurator.install(ServiceConfigurator.java:139)
at org.jboss.system.ServiceController.install(ServiceController.java:198)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.jboss.mx.server.ReflectedDispatcher.dispatch(ReflectedDispatcher.java:60)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:61)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:53)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:185)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:473)
at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
at $Proxy4.install(Unknown Source)
at org.jboss.deployment.SARDeployer.create(SARDeployer.java:204)
... 21 more
Caused by: java.lang.reflect.UndeclaredThrowableException
at org.jboss.system.ServiceCreator.install(ServiceCreator.java:175)
at org.jboss.system.ServiceConfigurator.internalInstall(ServiceConfigurator.java:149)
at org.jboss.system.ServiceConfigurator.install(ServiceConfigurator.java:114)
... 35 more
Caused by: java.lang.OutOfMemoryError
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:204)
at java.util.jar.JarFile.(JarFile.java:132)
at java.util.jar.JarFile.(JarFile.java:70)
at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:576)
at sun.misc.URLClassPath$JarLoader.(URLClassPath.java:543)
at sun.misc.URLClassPath$3.run(URLClassPath.java:321)
at java.security.AccessController.doPrivileged(Native Method)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:310)
at sun.misc.URLClassPath.getLoader(URLClassPath.java:287)
at sun.misc.URLClassPath.findResource(URLClassPath.java:138)
at java.net.URLClassLoader$2.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findResource(URLClassLoader.java:359)
at java.lang.ClassLoader.getResource(ClassLoader.java:977)
at org.jboss.mx.loading.UnifiedClassLoader.getResourceLocally(UnifiedClassLoader.java:327)
at org.jboss.mx.loading.LoadMgr3.beginLoadTask(LoadMgr3.java:234)
at org.jboss.mx.loading.UnifiedClassLoader3.loadClassImpl(UnifiedClassLoader3.java:178)
at org.jboss.mx.loading.UnifiedClassLoader3.loadClass(UnifiedClassLoader3.java:132)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at org.jboss.mx.server.MBeanServerImpl.instantiate(MBeanServerImpl.java:843)
at org.jboss.mx.server.MBeanServerImpl.instantiate(MBeanServerImpl.java:290)
at org.jboss.mx.server.MBeanServerImpl.createMBean(MBeanServerImpl.java:317)
at org.jboss.system.ServiceCreator.install(ServiceCreator.java:98)
... 37 more
00:30:14,798 INFO [Server] JBoss SHUTDOWN: Undeploying all packages
Shutting down
Shutdown complete
Halting VM

[starksm64]

Clearly the state of 64 bit vms on this platform are pathetic then.

[eiman]

Some more info regarding this from the java bug database (bug #5077277 ):

FULL PRODUCT VERSION :
Tested with 1.4.2_03, 1.4.2_04 and 1.3.1_06

A DESCRIPTION OF THE PROBLEM :
As described in Bug 4548840, the ZipFile class maps the zip file into memory in current Java releases.

The problem here is that it appears to allocate the memory for the file outside of the JVM heap. If the space outside of that heap is not big enough (even if there would be enough free space within the heap) an OutOfMemoryError is thrown.

This would cause a user to think that the heap is configured too small instead of, as it is, too big. This is needlessly confusing, since the API description of OutOfMemoryError speaks only of "memory" but also mentions the garbage collector operating on said "memory", making one believe that this might refer to the heap only.

I believe this is not so much an implementation issue as one with the available documentation.

I would suggest adding a note to the Javadoc for the ZipFile class that an OutOfMemoryError thrown by its constructor might not signify a lack of JVM heap space, or mentioning such a fact with the documentation for OutOfMemoryError itself.

[thammoud]

This is a bug in JBOSS and not JDK1.5. See http://jira.jboss.com/jira/browse/JBAS-1837 for more details. We encountered the same behavior.