High consume memory on RHQ 4.1.0

[marco.simoes]

Hi guys. I have in my enviroment the RHQ 4.1.0 server and postgre installed on the same machine. I monitor 29 machines with RHQ. I configured 2GB of memory ram for the RHQ machine. After few hours, the java process consume more then 2GB and the swap space was used.

I configured 6GB of memory ram and after 3 days,  swap space was used again.

Anyone had this problem ?

I run  RHQ on Centos 6.4 x86_64 with openJdk 1.7.0_45.

[tsegismont]

Did you mean RHQ 4.10 ?

[pilhuhn]

Hey Marco,

as Thomas already wrote - did you mean 4.10?  If that is not 4.10, but an earlier version, please try to upgrade and check the behavior.

If this is 4.10 I would be very much interested in getting a heap dump to investigate what is going on and to improve the usage.

Heiko

[marco.simoes]

Sorry for the mistake.

The version I'm using is 4.10.

I'll provide the heap dump and share with you.

Thanks for the quick response.

[john.sanda]

marco.simoes, is the RHQ Storage Node co-located on the same machine as your RHQ Server? If so you ought to check its memory usage.

[marco.simoes]

Yes John, the storage node is run on the same machine of rqh server, and the consumption memory is high.

I think the both process, rhq-server and storage node, run on the same machine can affect the performance, but i put 6GB of memory ram and the problems occurred again. 

[john.sanda]

By default the storage node is configured with a 512 MB heap. It can and will consume non-heap memory though as some things are stored off heap in memory mapped files. What kind of load are you putting on your storage node?

Do you know how how many metrics/minute you are collecting?

You can determine this by managing the RHQ Server. Commit it into inventory if it isn't already. Then in the resource tree drill down to the Measurement Subsystem. Go to the Monitoring tab, and then the Schedules subtab.Enable the Measurements Inserted and Measurements Inserted Per Minute schedules. Go ahead and set the collection interval to something low like 1 or 2 minutes.

Do you know how many numeric measurement schedules are enabled?

We can easily determine this with a JPQL query. Go to the Administration top-level navigation and then go to the System Settings. Enable debug mode. Then in your browser go to http://<HOST_NAME>:7080/coregui/#Test/ServerAccess/Hibernate. Execute the following query,

SELECT COUNT(schedule)

FROM MeasurementSchedule schedule

WHER schedule.definition.dataType = 0 AND schedule.enabled = true

In term of deployment, you can put storage nodes on their own dedicated machines. If you are experiencing swap issues, you may want to consider that.

[marco.simoes]

Thanks for the reply John. I'll get the information you gave me.

[pilhuhn]

Hey Marco,

do you have any news of us (or did I miss them)?

Thanks

Heiko