startup jvm memory config suggestions

[jschneider]

Version: 4.0.5 GA
OS: Linux Red Hat Enterprise

We started getting PermGen errors on one of our servers. Before digging in to understand what this meant, my knee-jerk reaction was to just add more memory. We increased from 4GB to 8GB.

After digging in to PermGen more (reading until my eyes glossed over), I found the PermSize/MaxPermSize. After upgrading the memory, I tried changing the jvm config options to take advantage of the new memory. JBoss is about all that is running on that server so I'd like to give it as much memory as possible. I figured I could up -Xmx to 6GB and then up MaxPermSize to 256m, but I got startup errors when I did that. "not enough heap space" (and some other one that I can't remember). Here are the java-opts that I have now:

-Dprogram.name=run.sh -server
-Xms2560m -Xmx2560m -Dsun.rmi.dgc.client.gcInterval=1800000 -Dsun.rmi.dgc.server.gcInterval=1800000 -XX:MaxPermSize=128m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC

While I'm not sure it's related, our users were complaining of performance issues today. With the garbage collection enabled as above, we got this (first entry and most recent):

================================================================================
{Heap before gc invocations=1:
PSYoungGen total 254912K, used 56837K [0x9f9f0000, 0xb1660000, 0xb1660000)
eden space 218560K, 26% used [0x9f9f0000,0xa31717d8,0xacf60000)
from space 36352K, 0% used [0xaf2e0000,0xaf2e0000,0xb1660000)
to space 36352K, 0% used [0xacf60000,0xacf60000,0xaf2e0000)
PSOldGen total 2330176K, used 0K [0x11660000, 0x9f9f0000, 0x9f9f0000)
object space 2330176K, 0% used [0x11660000,0x11660000,0x9f9f0000)
PSPermGen total 16384K, used 8080K [0x09660000, 0x0a660000, 0x11660000)
object space 16384K, 49% used [0x09660000,0x09e44068,0x0a660000)
2.687: [GC [PSYoungGen: 56837K->3028K(254912K)] 56837K->3028K(2585088K), 0.01156
90 secs]
Heap after gc invocations=1:
PSYoungGen total 254912K, used 3028K [0x9f9f0000, 0xb1660000, 0xb1660000)
eden space 218560K, 0% used [0x9f9f0000,0x9f9f0000,0xacf60000)
from space 36352K, 8% used [0xacf60000,0xad2553c8,0xaf2e0000)
to space 36352K, 0% used [0xaf2e0000,0xaf2e0000,0xb1660000)
PSOldGen total 2330176K, used 0K [0x11660000, 0x9f9f0000, 0x9f9f0000)
object space 2330176K, 0% used [0x11660000,0x11660000,0x9f9f0000)
PSPermGen total 16384K, used 8080K [0x09660000, 0x0a660000, 0x11660000)
object space 16384K, 49% used [0x09660000,0x09e44068,0x0a660000)
}

================================================================================
{Heap before gc invocations=559:
PSYoungGen total 179072K, used 175247K [0x9f9f0000, 0xaf100000, 0xb1660000
)
eden space 174784K, 100% used [0x9f9f0000,0xaa4a0000,0xaa4a0000)
from space 4288K, 10% used [0xaecd0000,0xaed43d20,0xaf100000)
to space 39104K, 0% used [0xaa4a0000,0xaa4a0000,0xacad0000)
PSOldGen total 2330176K, used 214697K [0x11660000, 0x9f9f0000, 0x9f9f000
0)
object space 2330176K, 9% used [0x11660000,0x1e80a770,0x9f9f0000)
PSPermGen total 68736K, used 64198K [0x09660000, 0x0d980000, 0x11660000)
object space 68736K, 93% used [0x09660000,0x0d5118c0,0x0d980000)
39479.614: [GC [PSYoungGen: 175247K->352K(174592K)] 389945K->215333K(2504768K),
0.0041540 secs]
Heap after gc invocations=559:
PSYoungGen total 174592K, used 352K [0x9f9f0000, 0xaed70000, 0xb1660000)
eden space 174208K, 0% used [0x9f9f0000,0x9f9f0000,0xaa410000)
from space 384K, 91% used [0xaa4a0000,0xaa4f8020,0xaa500000)
to space 37568K, 0% used [0xac8c0000,0xac8c0000,0xaed70000)
PSOldGen total 2330176K, used 214981K [0x11660000, 0x9f9f0000, 0x9f9f000
0)
object space 2330176K, 9% used [0x11660000,0x1e8514b0,0x9f9f0000)
PSPermGen total 68736K, used 64198K [0x09660000, 0x0d980000, 0x11660000)
object space 68736K, 93% used [0x09660000,0x0d5118c0,0x0d980000)
}


If anyone has any suggestions for memory and garbage collection settings, I'd appreciate your thoughts.

tia,

jim

[peterj]

Your max perm gen setting is 128m, but only 16m has been allocated and only 8m is in use. Do you redeploy applications often? That is what usually causes the need for a larger perm gen. If so, leave it at 256m, it is good insurance.

Looking at the heap usage, ignore the first gc - it is one that jbossas forces during startup. Actually, about the first dozen gcs can be ignored because they are all startup related. Looking at the last gc, almost all of the objects in the young generation were collected (less that 1mb of the objects survived the collection). I suspect that the young generation, currently set at 179m, is too big. And the overall heap size of 2.6gb is way too big - you are using all of 200mb of it.

Replace the -Xms and -Xmx with these settings:

-Xms600m -Xmx600m -XX:MaxNewSize=100m -XX:NewSize=100m

If that causes the old generation to fill up too quickly, increase everything:

-Xms800m -Xmx800m -XX:MaxNewSize=150m -XX:NewSize=150m

Also, gather the gc data using only -verbose:gc and plot the data. Looking at the plot of the graph will give you a better idea of how to adjust the heap settings.

See also http://www.jboss.com/index.html?module=bb&op=viewtopic&t=130504

By the way, are you running 64-bit Red Hat with a 64-bit JVM?

[jschneider]

Thanks, Peter. To answer some of your questions...

No, we do not redeploy very often, but we were running into PermGen errors daily with the default setting. Curiously, it was only 1 of the 2 servers. The other is/was configured exactly the same way. The load balancer is supposed to alternate sessions between servers, but I don't know how well that is working.

I am curious as to the -ms/-mx settings given the amount of memory available in our system, but I obviously don't know how this really works. I will look at the link and the presentation included within.

I do not believe it is the 64-bit OS or JVM.