JBoss (web service app) huge memory usage

[xinhua]

hi, all

i have a ejb3 based webservice (jbossWS) on JbossAS 4.2.2GA which generate responses for about 10.000 requests everyday. Physical memory is 16G, 32Cpu, SunOS 5.10 (sparcv9) 64bit.

My jvm opt is -Xms1024m -Xms1024m -Xss256k -d64

prstat shows that there are fixed 178 lwps but the avg. memory usage is 5.5G, it is huge!

The service has runed more than one week, there is no outOfMemory crash, memory usage max. reached 5.8G.

i tested my App with jProfiler on local server (identical server config), i didnot see any memory leak. So I am very confused :(

can anyone help? Thanks in advance

[xinhua]

here is my memory pool:

Total Memory Pools: 5

 Pool: Code Cache (Non-heap memory)

 Peak Usage : init:2097152, used:19078080, committed:19136512, max:50331648
 Current Usage : init:2097152, used:19070336, committed:19136512, max:50331648


 |-------------------------| committed:18,25Mb
 +---------------------------------------------------------------------+
 |/////////////////////////| | max:48Mb
 +---------------------------------------------------------------------+
 |-------------------------| used:18,19Mb


 Pool: PS Eden Space (Heap memory)

 Peak Usage : init:270532608, used:328335360, committed:328335360, max:349634560
 Current Usage : init:270532608, used:121320968, committed:297140224, max:328335360


 |--------------------------------------------------------------| committed:283,38Mb
 +---------------------------------------------------------------------+
 |///////////////////////// | | max:313,12Mb
 +---------------------------------------------------------------------+
 |------------------------| used:115,7Mb


 Pool: PS Survivor Space (Heap memory)

 Peak Usage : init:45088768, used:45088424, committed:61734912, max:61734912
 Current Usage : init:45088768, used:14708968, committed:14745600, max:14745600


 |---------------------------------------------------------------------| committed:14,06Mb
 +---------------------------------------------------------------------+
 |/////////////////////////////////////////////////////////////////////| max:14,06Mb
 +---------------------------------------------------------------------+
 |--------------------------------------------------------------------| used:14,03Mb


 Pool: PS Old Gen (Heap memory)

 Peak Usage : init:717225984, used:364008224, committed:717225984, max:717225984
 Current Usage : init:717225984, used:217202688, committed:717225984, max:717225984


 |---------------------------------------------------------------------| committed:684Mb
 +---------------------------------------------------------------------+
 |///////////////////// | max:684Mb
 +---------------------------------------------------------------------+
 |--------------------| used:207,14Mb


 Pool: PS Perm Gen (Non-heap memory)

 Peak Usage : init:25165824, used:75780424, committed:88080384, max:88080384
 Current Usage : init:25165824, used:75610008, committed:83886080, max:88080384


 |-----------------------------------------------------------------| committed:80Mb
 +---------------------------------------------------------------------+
 |//////////////////////////////////////////////////////////// | | max:84Mb
 +---------------------------------------------------------------------+
 |-----------------------------------------------------------| used:72,11Mb

[xinhua]

in Wiki i found followed statement:

DO NOT USE -d64 (64-bit) if you do not use ABOVE the maximum 32 bit heap space (2-4 GB of heap). Using 64 bit addressing requires MORE memory to do the same amount of work and provides no advantage for applications that do not need this much memory.

i used -d64 in jvm opt, is that reason?

[peterj]

Add up the heap usage data you provided. Compare that to the memory usage. The difference is being used by the JVM to do whatever it does. If that seems excessive (and I agree with you that it does seem to be excessive), then you will have to ask Sun because the Java app (JBossAS in this case) has no bearing on that usages (it was limited to the 1GB heap you set).

[xinhua]

hi, peter

thank you for your reply.

I think -d64 caused too much unnecessary memory, but I can not simplely get -d64 away, because i use Timesten and jboss need to load 64bit timesten lib by starting.

and also , i start to suspect that Solaris didnot take my -Xss opt at all, there are 157 threads runing in system and my System stack size is 8M , plus the stupid -d64 , is that 5.5 memory usage resonable?

it is realy wired....

[peterj]

If I understand you correctly, you are using the -d64 option, and the reason you are using this option is because of a library, named Timesten, that you are using in your application.

So let me ask you this. If you run without -d64 and without Timesten, does JBossAS show reasonable memory usage? (I realize that to do this you will not be able to deploy your application.) If the answer to that question is "yes," then why are you asking this question in the JBoss forums and not directing this question to the Timesten developers, or to Sun (it's their JVM that implements the -d64 option).

Also, the -Xss governs the stack size for each thread. Thus you are using 157*8K or memory just for the stacks.

Your heap setting is 1GB. The JVM will use at most that much memory for Java objects. The rest of the memory is used by the JVM. Wait a minute, that is not correct. The Timesten library is a shared object, correct? Well, it also uses memory. Try this: run JBossAS without your app (and thus without Timesten) but keep the -d64 option on the command line. Look at the memory usage. Compare that to the 5.5GB you mentioned. The difference is what is being used by Timesten. If that is excessive, talk to the developers of Timesten.

[xinhua]

Hi, peter, thank you for your advice.

yes you right, i should contact Sun or Oracle, why i did put the question here is because maybe someone here encoountered the same problem as mine and know how to solve it. And also, i am not sure it would be jvm problem so i wonder if i can tune JbossAS to reduce memory usage.

Sofar, my webservice and Oracle Timesten db which takes more then 10G memory runs on the same productive server(16G+Solaris64bit) and i can not deploy undeploy app / shutdown start jboss for test. (unfortunately i do not have a sever with same equipment for test.) if i run my webservice (remote connect Timesten with client driver) on a 32bitSolaris with 4G memory, i didnot see any unormal memory usage.

Today, i wonder if solaris really take my -Xss def and will install dtrace script on server to see how many memory each thread used. Yes, surely will ask Sun about this also .

[peterj]

xinhua wrote:
i can not deploy undeploy app / shutdown start jboss for test

You don't have to shutdown your existing JBossAS to do a test - you could run a second instance.

xinhua wrote:
i wonder if i can tune JBossAS to reduce memory usage

Memory usage of any Java app (which includes JBossAS) is dictated by the heap space allocated. Thus you could, for example, reduce the 1GB heap you are currently allocating to 800MB to save space. But that 200MB of savings would be a negligible part of the 5.5GB you reported. See my presentation on tuning the heap: http://www.cecmg.de/doc/tagung_2007/agenda07/24-mai/2b3-peter-johnson/index.html

You could also change the number of threads that handle HTTP request by editing the maxThreads attribute of the Connector tag in server.xml (in server/xxx/deploy/jboss-web.deployer). But reducing that can reduce the response time for your users (if they have to wait for an available thread), or even result in "server is busy" errors. Besides, if each thread uses only 8K, then 100 threads use less than 1MB, which is an insignificant part of the 5.5GB.

[xinhua]

hi,peter

thank you for advice :).

Creating a new jbossAs instance on our productive server for testing is not a good idea (i have only 16G phisical memory and timesten will take almost 12-14G) . If i do that my boss will yell at me :)

i have seen the report from tomcat in webconsole, we have only 108 threads on http. Default 150 should be ok for our case.

The problem is, no matter max heap size & max perm size i set, 5.5G mem usage is far beyond the limit. By default Solalis stack size is 8M , and prstat -L can not list memory usage for each lwp, so i suspect solaris server jvm didnot take -xss (maybe it only take -XX:ThreadStackSize)...


Gruss

[xinhua]

by the way, your presentation is very helpful, thanks a lot :)

[xinhua]

It is ISM of Solaris.

Timesten runs in ISM and my JVM didnot close ISM by default !!

What a lesson :)