High Swap Utilization, lot's of RAM free

[polaren]

Hey guys,

Alright, so I have a server I am monitoring 8 Opteron dual cores, 28GB of ram, and a decent amount of SCSI HD space, not in a raid configuration though.
Essentially it uses Jboss application server with a program that writes SQL statements commits them.etc.etc to the database(Sybase). The database is not being maxed at all. Although this server is for some reason utilizing tons of swap space. So we say okay it is maxing out 28GB of RAM? It's not as it is utilizing around 14GB , with another 14GB committed. I have no idea why it is swapping so much, it doesn't make much sense when it has 14GB of memory free. It's almost almost as though it doesn't know how to utilize the remaining memory, yet somedays it will use up to around 20GB. When it get's into heavy load it will not use the extra memory it has, if we lower the available memory to it (as it is not using it) we get memory exception errors. 28GB committed has been the only way it is working properly.

I am not a Jboss expert in anyway, I monitor the performance and utilization of the servers. Any advice or input would be appreciated. If you need more info, I'd be happy to provide it.

[peterj]

What operating system? Windows tends to swap things out to disk even when it doesn't need to.

What are your JVM heap settings?

Also, you might get better performance if you limit the app server to running on 2, 3 or 4 of the Opterons(sockets). We have found that after 4 sockets that either you get no additional performance benefit or performance drops. And that some deployed apps run better on 2, others on 3, and others on 4 sockets.

[polaren]

"PeterJ" wrote:
What operating system? Windows tends to swap things out to disk even when it doesn't need to.

What are your JVM heap settings?

Also, you might get better performance if you limit the app server to running on 2, 3 or 4 of the Opterons(sockets). We have found that after 4 sockets that either you get no additional performance benefit or performance drops. And that some deployed apps run better on 2, others on 3, and others on 4 sockets.

Operating System is Windows Server 2k3, sorry forgot to specify that.

Where can I grab the heap settings from? Sorry I haven't needed to get that information before.

More info on the swapping is that as the amount of users sending their info to the database increases, the server uses more of the swap space. It's almost like it is storing client information, yet the server logically has nobody logged into it.

I'll look into the CPU performance problems.
Just looking for a solution to make it so that the server can handle more information as our user base is becoming larger. I am assuming right now that JBoss and that server are not functioning properly, as it is quite powerful for what it is doing.

[polaren]

This may give you a better indication as to what I am looking at over a 24hr period

[url=http://img524.imageshack.us/my.php?image=swap24hryk6.jpg][img=http://img524.imageshack.us/img524/1281/swap24hryk6.th.jpg][/url]

[peterj]

How are you starting the app server? By manually executing run.bat or have you installed the app server as a service? If you installed the app server as a service, which utility did you use to do this? The utilities I know of include JBoss Native, JavaService, Java Service Wrapper and each configures JVM options differently.

If you are using run.bat of JBoss Native the JVM heap options are set in JAVA_OPTS in run.bat.

Have you used -verbose:gc (or other heap-monitoring options) to determine JVM heap usage?

[polaren]

"PeterJ" wrote:
How are you starting the app server? By manually executing run.bat or have you installed the app server as a service? If you installed the app server as a service, which utility did you use to do this? The utilities I know of include JBoss Native, JavaService, Java Service Wrapper and each configures JVM options differently.

Manually executing run.bat

If you are using run.bat of JBoss Native the JVM heap options are set in JAVA_OPTS in run.bat.

rem Sun JVM memory allocation pool parameters. Modify as appropriate.

set JAVA_OPTS=%JAVA_OPTS% -Xms160m -Xmx28672m -XX:NewSize=4m -XX:PermSize=256m -XX:SurvivorRatio=2 -XX:TargetSurvivorRatio=80

Have you used -verbose:gc (or other heap-monitoring options) to determine JVM heap usage?

No, Sir

If you need more info, let me know :)
Thanks for your help so far

[peterj]

How did you go about selecting the NewSize and survivor ratios? With the initial ratio of 2, each survivor space is 1m and the eden 2m. You will fill up the 2m eden in no time (literally in milliseconds). And the 1m survivor space will be mostly empty because only a small fraction of objects survive a collection.

For production, you should set Xmx and Xms to the same size (prevents the JVM form constantly requesting more memory, or freeing up memory, and from deciding if it should do either. I wonder if that has a bearing on the swap utilization you are seeing, not really sure, it could also be the heap sizes you are using).

Do not set the heap to 28G like you have it. You will not be happy if you fill it up and the garbage collector collects that heap - it will take forever (1/2 hour, 1 hour, more???).

A good starting point for the heap is 1200M, unless you can show that you need more memory, or can get away with less. Based on your system RAM, 1200M is nothing.

If you set both Xmx and Xms to 1200M, you should set NewSize to around 300m (NewSize should be 1/3 to 1/4 of the heap size) .

Do not set any survivor ratio arguments unless you have analyzed the data from PrintHeapAtGC and determined that you need to fine tune the survivor ratio.

The resulting JAVA_OPS are thus:

set JAVA_OPTS=%JAVA_OPTS% -Xms1200m -Xmx1200m -XX:NewSize=300m -XX:MaxNewSize=300m -XX:PermSize=256m

[polaren]

Thanks again Peter,

I have passed the information along, we are going to test the resulting JAVA_OPS and see how it performs compared to the current. Which I am sure will be somewhat better.
Most peers have questioned mainly why such a low amount of RAM is needed, when on some days we are hitting 20GB, but normally 14GB. Now dropping to 1200M?
Any insight or reasoning would be quite helpful.

Also regarding the NewSize and survivor ratios, do you believe these will have to be fined tuned from your experience? or will the current setting supply the need.

[peterj]

RAM is not the same as Java heap. Looking at RAM usage typically does not mean much. Unless you analyze the Java heap usage, it is difficult to say how much memory the app is really using, or how much it really needs. JBoss AS by itself does not need much (64M is probably more than enough), and unless you have a very unusual application I doubt that your application would need 14G of heap.

And yes the settings I gave are starting points. You need to analyze the heap usage and adjust those accordingly.

I thought occurred to me that the JVM, if given the opportunity (such as offered by your initial heap settings) might grab a significant chunk of memory if it is available. If I have some time, I might wrestle one of our Windows boxes from the performance team and try it out, or maybe one of them knows off-hand.