Servers swapping memory

[suchitkhobarekar]

Hi,

We have configured Liferay servers on JBoss AS 7.1.1. We have two environments Stage and Prod which are pretty much identical in terms of the hardware and software. However we are seeing some heavy swapping happening on the Prod servers. About a few hours after the servers are started they start swapping memory and this is how it read (as below):

Memory              Used          Total      Percentage

============================

Physical           4079M       7872M          51%

Actual                3630M       7872M          46%

Swap                    878M   2047M          42%

This is not a good sign for us. The JAVA_OPTS setting in standalone.conf is as below

JAVA_OPTS="$JAVA_OPTS -Dfile.encoding=UTF-8 -Djava.net.preferIPv4Stack=true -Duser.timezone=GMT -server -XX:NewSize=700m -XX:MaxNewSize=700m -Xms2048m -Xmx4096m -XX:MaxPermSize=1536m -XX:SurvivorRatio=6 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=15 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:ParallelGCThreads=16 -XX:+CMSCompactWhenClearAllSoftRefs -XX:CMSInitiatingOccupancyFraction=85 -XX:+CMSScavengeBeforeRemark -XX:+CMSConcurrentMTEnabled -XX:ParallelCMSThreads=2"

There are 2 nodes with 8 GB of RAM each

I found a thread which mentioned that the -Xms and -Xmx should ideally have the same values. This is the thread https://community.jboss.org/message/221207

Any help on this is really appreciated. I

[wdfink]

Hi,

it seems that you give the JVM's more memory than the system has.

Could you give more details how the system is, do you mean you have two boxes with 8GB each?

How much JBoss instances do you start.

Also check the memory consumption of the JVM, you might use 'jstat -gc <JVM pid>' the output show how the memory is occupied. See Oracle page for jstat details, it is part of the JVM.

Also you hace 1.5GB PermSize does your application is that big? Maybe jstat show that the PermSize is too much and you can reduce it.

[suchitkhobarekar]

Thank you so much for you response.

So we have two VM's with 8GB RAM each. And we start only JBoss instance per VM. The other applications running on the VM's are openoffice and imagemagick. But they dont seem to be hogging to much memory.

We will try out the jstat and post the results soon.

[rhusar]

You have totally incorrectly configured JVMs!

The total memory usage of each JVM is 4 GB heap + 1.5 GB max perm (The -Xmx parameter refers to heap memory, whereas the permanent generation space is a separate memory allocation!) + other memory for threads, memory allocated through JNI, shared JVM stuff, etc. So each JVM is about 6 GB but system total memory is only 8 GB!

You need to fix the allocations for the system to be stable.

[wdfink]

As I mentioned before, I suppose that your -XX:MaxPermSize=1536m setting can be reduced as this area is only used to store classes.

Please check with jstat.

Also I suppose your JVM setting memory and GC is not a 'tuned' one, do you have grabbed an example somewhere.

As these settings are very individual you should drop the most of the settings (i.e. CMS settings)