-
1. Re: JBoss/JVM process RAM usage increasing steadily
RAM for a Java app is consumed by more than just heap (RAM usage for the heap will never exceed 2048MB in your case).
See this for a list of everything that can use up RAM:
http://www.coderanch.com/t/111262/Linux-UNIX/read-memory-usage-process-running#560894
How many threads are running? Have the number of threads increased? If so, that could account for the additonal RAM usage.
-
2. Re: JBoss/JVM process RAM usage increasing steadily
We have tried to replicate this issue on Test bed with Heap size set to 1GB.
On test bed as soon as JBoss is started we find that there are a lot of threads (JBoss System Thread(1) - around 300 threads) that are created.
I have no idea what these threads are for, any information on this will be helpful.
This is even before any request is made to the servers. But this is not observer on Production.
Thread count is tabulated below. Though number of threads has decreased the RSS value is increasing.
Date
Time
Total Thread
Daemon Thread
Non-Daemon Thread
JBoss System Thread(1)
http Thread
VSZ
RSS
SZ
2012-03-12
17:55:30
431
403
28
300
61
2343840
1028664
585960
mapped: 1835679K writeable/private: 1803527K shared: 0K
2012-03-12
18:22:42
428
399
28
300
61
2409304
1500656
602326
mapped: 1835679K writeable/private: 1803527K shared: 0K
2012-03-13
09:52:52
423
395
28
300
61
2409304
1501628
602326
mapped: 1835679K writeable/private: 1803527K shared: 0K
-
3. Re: JBoss/JVM process RAM usage increasing steadily
You have the heap set to 2GB. The JVM will reserve that memory immediately, but will not start using it until it needs it. That would expalin the RSS increasing - as the app runs and it uses the heap the JVM will start to make use of some of that memory that it has allocated and the OS will make it resident. Notice that after the app has been runnign for a while that the RSS rmains fairly constant.
Note that the VSZ remain fairly constant. The increase after the initial size can be attributed to the perm gen increasing as classes are loaded.
I don't see anything here that would cause me alarm.
As far as what each of the threads are doing, you could take a JVM thread dump and see.
-
4. Re: JBoss/JVM process RAM usage increasing steadily
Values mentioned in the above post were of server where Heap size was configured as 1GB.
Below are values from the server where Heap is configured to 2GB but RSS=5.5GB and this keeps increasing till it uses up the entire RAM available.
Also the thread count is almost constant for over 24hrs but RSS value is increasing.
Date
Time
Total
threads
Daemon
threads
Non daemon
threads
‘JBoss
System threads’
HTTP
threads
VSZ
RSS
SZ
mapped:
writeable/private:
shared:
2012/3/13
11:50:48
277
245
32
2
208
3644988
3026036
911247
mapped: 3653180K
writeable/private: 3403160K
shared: 3272K
2012/3/13
14:32:03
276
244
32
2
208
3817224
3186660
954306
mapped: 3825416K
writeable/private: 3545084K
shared: 3272K
2012/3/13
18:00:03
276
244
32
2
208
4175848
3614728
1043962
mapped: 4184040K
writeable/private: 3950492K
shared: 3272K
2012/3/14
00:00:03
276
244
32
2
208
4696012
4165484
1174003
mapped: 4704204K
writeable/private: 4480168K
shared: 3272K
2012/3/14
06:00:03
276
244
32
2
208
5168076
4573556
1292019
mapped: 5176268K
writeable/private: 4884224K
shared: 3272K
2012/3/14
12:00:04
276
244
32
2
208
5953416
5349432
1488354
mapped: 5961608K
writeable/private: 5643848K
shared: 3272K
2012/3/14
15:00:03
276
244
32
2
208
6349600
5778032
1587400
mapped: 6357792K
writeable/private: 6077580K
shared: 3272K
-
5. Re: JBoss/JVM process RAM usage increasing steadily
Which JVM are you using? OpenJDK? Or the one from Oracle? If from Oracle, revert to version 1.6.0_26 and see if that helps (we have noticed various issues with versions more recent than that). If that doesn't help, you'll have to contact Oracle to see if there is a memory leak issue with the JVM itself.
This almost sounds as if a system resource (such as a file handle or a socket connection) is not being released. But JBoss AS itself maintains pools of resources and reuses them, but who knows what the native JVM code is doing (only Oracle would know).