JBoss memory usage pattern

[lorenooliveira]

Taking a look at one of my applications running in a production server, I noticed that there is a "normal but strange" behavior of memory usage.

Let me explain: watching the execution of my JBoss 4.2.2 without any application deployed, I can see that is constantly grows and frees the used heap space, usually a few megabytes in a development server. When I deploy my application, the pattern is the same, but using more memory in average.

Well, in the production server I can see that my JBoss, even without any workload, has a minimum memory usage of 1.5GB. Still without any workload, the heap usage grows to 3.6GB, when a minor GC runs and the heap usage comes back to 1.5GB. Every 40s my JBoss grows its heap usage from 1,5GB to 3.6GB, and this pattern repeats indefinitely. When the workload grows, the difference is that the period for growing the memory usage falls to 8s.

So, my question is: is this normal?

[peterj]

What kind of monitoring are you doing? This sounds like what I have observed when I have VisualVM monitoring JBoss AS - memeory slowly fills up and eventually a GC happens. Without any load. If you have monitoring tools that are making continual calls (to MBeans, for example), then you will see something like this.

[lorenooliveira]

Yes, that's exactly how I'm doing. I'm watching the server behavior using visualvm. Is there a better approach to monitor an application but JMX? I mean, how could one get data about some application health with a minimum of overhead and a maximum of accuracy?

[wdfink]

A easy way to monitor the jvm without big influence is 'jstat -gc PID'.
The behaviour of growing utilisation is normal for the JVMs eden space where all new created objects are.

A good starting point will be the 'Java GC tuning site'

[lorenooliveira]

Hi Wolf-Dieter,

Actually, I started all this observation after reading about gc performance tunning. I'm trying to find evidences that lead me to make some sort of tunning. So far, I didn't find anything in this sense, but I found this behavior. The strange point here is the volume. I mean, the heap usage is growing almost 2GB in 40s with low (or none) workload, and in 8s with a higher workload.

Is this normal (in this volume)? In the point of view of the users, we are not getting malfunction reports.However, we should grow our workload in the next weeks...

[peterj]

The problem with using VisualVM in a production environment is that it is too heavy of a solution. VisualVM updates its status constanly (every second?). Most monitoring tools update status only every 5 to 15 minutes. Of course that is adjustable, but the more often you monitor the greater the impact on production.

If you are interested in gathering GC data during production, use -verbose:gc and redirect the data to a file (-Xloggc:<file>). That will give you sufficient data and you can turn -verbose:gc on/off from an MBean (a little java app run script a scrfipt could do this for you).

Here are some paper I wrote on GC analysis:

Java Garbage Collection Statistical Analysis 101

Java Garbage Collection Performance Analysis 201

at http://www.cmg.org/cgi-bin/search.cgi?q=java+peter+johnson&x=30&y=10

[lorenooliveira]

Peter, I understand the point that visualvm can be, itself, altering in some sense the environment that it monitors. However, I tried to understand how big is this influnce by means of a simple test (I just don't know whether it was too simple...).

Using visualvm with its default configuration (1s pooling), I noticed that the heap usage of my application oscilates in a window of aproximately 2gb. Then I changed the pooling to 10 and 30 seconds, in two following tests. With these new configurations, the heap usage is still in the same window.

Am I testing this the right way? Should this change in the pooling interval change the size of the windows noticed in the monitoring?

[peterj]

I don't know if the "polling" setting in VisualVM changes how often it polls the JVM, or how often it updates the display. I would just use -verbose:gc and -Xloggc and then monitor the loggc file. That would eliminate anything funny that VisualVM is doing.

If you say you are generating 2GB of garbage in about 40 seconds, then that is a lot of garbage being generated (50MB per second). I usually have to run a benchmark with a heavy load to get that kind of churn. What is your CPU utilization? Are you certain that, other than the VisualVM connection, that the app server is idle?

[lorenooliveira]

I tried with the JVM flag, and I'm getting the same measurements. The only difference is the time between collections. When I just watch the output of verbosegc, it takes longer to collect the garbage. When a both watch the output of verbosegc and also monitor through visualvm, the collections are much more frequent, but the usage is kept in the same window.

I can say that the CPU usage is low. This is a big machine, with a lot of processors and 6gb dedicated to this vm. CPU utilization is under 10%, with some short peaks of 30%.

When this snapshot was taken, I can say that 95% of the normal workload was not present, but there where at least one more monitoring tool pooling data from the application (zabbix).

As it is a production server, I'm afraid of trying a memory dump. But I took two histograms of the server, one with low heap usage, and the other close to the top of the heap usage. The difference between them was char arrays, Strings, int arrays, and byte arrays. Because they are histograms, I cannot track back who is instantiating these objects. However, doing this in the same application running in the development environment and with a quite lower workload, I can see that all this stuff is being generated by the container, and not by my application...

[peterj]

I'd be interested to see what it looks like if zabbix is not running.

One thing you could do is take thread dumps and examine them. Look for threads that are not just waiting on objects, those are the active threads. The stack trace for those threads should give you some idea on what they are processing. This might help you track down what is generating the garbage.