Agent memory monitoring
genman Jun 25, 2013 4:10 PMI've had a few agents run out of memory, mostly due to too many resources, but sometimes due to memory leaks in bad plugins.
I know the agent is supposed to restart when it is low on memory? How does the agent do this and why doesn't it always work? How can we check or alert for this?
I also did some memory profiling, mostly to check on leaking. These are instance counts, not memory usage, but one thing that struck me is each resource, be it a MySQL table or something quite small, uses quite a lot of memory.
One thing I noticed is each Resource.childResources object is a Set wrapping a ConcurrentHashMap, which is quite expensive in terms of memory use. This was probably the only thing that seemed a bit excessive. The JDom stuff is probably from parsing JBoss XML files, and I wonder if those references need to be kept or not.
Instance Counts for All Classes (including platform)
328758 instances of class java.lang.String
294973 instances of class [C
128861 instances of class java.util.HashMap$Entry
96602 instances of class [Ljava.util.HashMap$Entry;
92602 instances of class [Ljava.lang.Object;
90346 instances of class java.util.ArrayList
71976 instances of class java.util.concurrent.locks.ReentrantLock$NonfairSync
71856 instances of class java.util.concurrent.ConcurrentHashMap$Segment
71856 instances of class [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
67739 instances of class java.util.HashMap
64907 instances of class java.util.LinkedHashMap$Entry
60993 instances of class org.rhq.core.domain.measurement.MeasurementScheduleRequest
55696 instances of class java.util.HashMap$KeySet
52850 instances of class java.util.HashSet
41600 instances of class java.lang.Integer
29953 instances of class org.rhq.core.pc.measurement.ScheduledMeasurementInfo
28859 instances of class java.util.LinkedHashMap
26987 instances of class java.lang.Long
25175 instances of class java.util.TreeMap$Entry
19993 instances of class [I
18786 instances of class org.rhq.core.domain.configuration.PropertySimple
17563 instances of class org.rhq.core.pc.plugin.CanonicalResourceKey$KeyTypePlugin
17430 instances of class java.util.TreeMap
17298 instances of class java.util.Hashtable$Entry
13148 instances of class javax.management.modelmbean.DescriptorSupport$FieldName
13052 instances of class org.rhq.core.pc.inventory.ResourceContainer$ResourceComponentInvocationHandler
12755 instances of class java.util.LinkedHashSet
12488 instances of class [B
11886 instances of class java.util.LinkedList$Entry
11845 instances of class java.util.LinkedList
11821 instances of class org.mc4j.ems.store.CompleteValueHistory
11671 instances of class org.rhq.core.domain.configuration.Configuration
11664 instances of class org.mc4j.ems.impl.jmx.connection.bean.attribute.DAttribute
7775 instances of class [S
6956 instances of class java.util.HashMap$Values
6927 instances of class java.util.HashMap$EntrySet
6669 instances of class javax.xml.bind.JAXBElement
6552 instances of class org.jdom.Text
6378 instances of class org.rhq.core.domain.configuration.definition.PropertyDefinitionSimple
6170 instances of class java.lang.Class
6164 instances of class org.rhq.core.clientapi.descriptor.configuration.SimpleProperty
6156 instances of class org.jdom.Attribute
4933 instances of class [Ljava.util.Hashtable$Entry;
4915 instances of class java.util.concurrent.ConcurrentHashMap$HashEntry
4913 instances of class java.io.File
4844 instances of class java.util.Hashtable