growableArray.cpp. Out of swap space?

[mdpatil]

Exception java.lang.OutOfMemoryError: requested 131072000 bytes for GrET* in D:/
BUILD_AREA/jdk142-update/ws/fcs/hotspot\src\share\vm\utilities\growableArray.cpp
. Out of swap space

Our server application is running RedHad Linux on IBM machine with 12gb ram.

OS-Redhat4
DB:oracle
jboss 4.0.2
jdk1.4

run.conf

JAVA_OPTS="-server -Xms512m -Xmx2000m"


Can you pls.. help me out.

MD

[peterj]

According to the error message, the app is allocating a 130MB object. How many of these huge objects do you have around?

[mdpatil]

hi peter,

our application is real time application(core banking application) ,so can't say how many object are at run time, after few minutes(45m) jboss trimented. .

if you know then tell me how to see run time objects in heap.

if any Monitor tool for jboss??

thanks,
MD

[peterj]

See http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html, especially the -verbose:gc and -XX:PrintHeapAtGC options.

[peterj]

Oh, you should also run a profiling tool, such a JProbe, on your application.

[jcreynol]

Was a resolution ever found for this problem? Different JVM options?

What are the odds that we would see the exact same error with the exact same size?

Exception java.lang.OutOfMemoryError: requested 131072000 bytes for GrET* in /BUILD_AREA/jdk1.5.0_14/hotspot/src/share/vm/utilities/growableArray.cpp. Out of swap space?

Any chance this is a JBoss leak?

[peterj]

Looks like the JVM is attempting to allocate another 128MB of memory (probably to expand the heap) and the OS is saying that it can't allocate the memory because it has run out - even the swap space is full.

How much RAM do you have? What is your swap space size? What is the non-JBossAS memory usage? What do you have the JVM heap sizes set to? I am guessing that -Xms and -Xmx are not the same.

You can usually detect a memory leak by gathering gc statistics using -verbose:gc and noting the amount of heap space in use after a major collection. After about a dozen or so major collections, assuming that the work load is consistent, the heap in use after major gc should be about the same. If it goes up, there is a possible memory leak.

[jcreynol]

Thanks, Peter. Definitely looks like we're leaking someplace.

Just found out from admins that the startup params are:

-Xms2048m -Xmx2048m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000

There's 8GB on the box and two instances running with these settings. Not sure why the GCInterval was set this high and will look to tighten that up... could that cause the memory bloat over time?

Your note did get us to look at the other usage on the box, as well, and so we may be able to find something there. (we were assuming that 8GB should be plenty to support 2 JBoss instances with Xmx2048m)

Thanks!
John

[jcreynol]

One additional answer. Swap space is 2GB.

[peterj]

Leave the gcinterval set as is, or better yet, make it larger. This is the amount of time (1 hour in this case) the jvm will wait before firing up a full garbage collection to collect any leftover objects lying around from rmi usage. Doing this is a useless exercise within an app server but the jvm insists on it.

See also this discussion: http://www.jboss.com/index.html?module=bb&op=viewtopic&t=130504

[smarlow]

The GrowableArray is used to implement many of the internal Java virtual machine functionality (in the native code portions).

In your application, one of the internal GrowableArrays was 65m in size and an attempt to allocate a 130m block failed. Even if the operation succeeded in allocating the 130m block, a later attempt to allocate a 260m would probably fail.

Please vote for this Sun Java bug to be fixed as I think it would help a lot: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6423256

It is easy and free to register with Sun to be able to vote.

If you have a Java support agreement with Sun, also request a fix for this bug for the version of the jvm that you use (it would be nice to see this fixed in a Java 5 or at least Java 6 release.)

Good luck,
Scott