High CPU usage by Jboss process with default setting

[kyosuke]

Hi,

I am using Jboss 3.0.1, however CPU usage by its process become high( about 8% ) once every minute, although any Web access is not done.
Does anyone know about this spike?

I investigated what system call is made by jboss process at the time of the spike with "tusc" command.

The following is the result of tusc.

# tusc -s open -i PID
npen("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 85
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 70
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 70
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 70
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 70
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 70
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 70
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 70
open("jboss-3.0.1/server/A/conf/log4j.xml", O_RDONLY|O_LARGEFILE, 0666) = 70
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 85
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 85
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 85
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 85
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 85
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 70
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 70
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 70
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 70
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 70
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 70
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 70
open("jboss-3.0.1/server/A/conf/log4j.xml", O_RDONLY|O_LARGEFILE, 0666) = 70
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 85
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 85
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 85
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 85
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 85
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 70
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 70
open("jboss-3.0.1/server/A/deploy", O_RDONLY, 0130100) = 70

The output is done every 5 seconds, and in this case, only one line was output every output.
I confirmed the spike occurs whenever the return value of open() is changed 85 to 70.
So, the change of the return value is done once every minute.
However I couldn't figure out the reason of the spike.

Next, I tried to confirm how the latest version of Jboss behaves.
I installed jboss-4.0.2, started it up, and confirmed if spike happens.
(Any configuration to jboss was not done.)
As the result, I confirmed Jboss process's CPU usage become high up to about 16% once every minute.

Therefore I believe this spike is Jboss's default behavior.
Why does this happen and is there any workaround for this spike?

Any information is appreciated.

Thanks,
Kyosuke

[peterj]

This behaviour could be caused by the JVM doing its remote object cleanup (which is a full garbage collection). By default, this occurs every minute. You can change this behavior by setting the sun.rmi.dgc.client.gcInterval and sun.rmi.dgc.server.gcInterval properties on the command line.

See http://wiki.jboss.org/wiki/Wiki.jsp?page=SPECjAppServer2002TuningComponentsJVM for more details.

Hope this helps.

[kyosuke]

Hi PeterJ,

Thank you for your reply.
Your help is appreciated.

I changed the sun.rmi.dgc.client.gcInterval and sun.rmi.dgc.server.gcInterval properties, and confirmed we can change the cycle of the behavior with the properties.

In other words, I confirmed the garbage collection causes the spike.

In my opinion, it is problem to consume that high CPU rate even once every hour due to the action of the GC.
I guess this behavior by the GC would happen also on other J2EE AS, such as Oracle AS or IBM WebSphere, because those ASs also must utilize Sun's GC.

How does web AS administrator usually cope with this spike issue?
I think the cyclic high CPU usage is problem for Web server receiving constant much accesses.

only by change the value of sun.rmi.dgc.client.gcInterval and sun.rmi.dgc.server.gcInterval to big number, such as 3600000?

Thanks
Kyosuke

[peterj]

Actually, within the course of an hour you will typically get such spikes frequently because a full garbage collection will run quite often, unless you can configure the heap size and young generation size such that you get only young generation collections (which are faster and produce a much smaller spike). See http://wiki.jboss.org/wiki/Wiki.jsp?page=SPECjAppServer2002TuningComponentsJVM item #3 for details.

But achieving that goal (young generation collections only) depends very much on the nature of your application and its workload. In other words, it is fairly easy to do under a predefined load in a performance test scenario (I have often accompilshed it with a heap of 1.2GB and young gen of 400MB, which of course means I have at least 2GB of RAM), but often harder to accomplish in a production environment because those pesky human beings who use the system are so unpredictable. :-)

[kyosuke]

Hi Perter Johnson,

I appreciate you.
Your information is very helpful.

I understand this is a behavior that AS can't help at any rate, therefore we have to design and configure Web AS system with the behavior in mind.

Thanks a lot.
Kyosuke