RHQ 4.9 performance issue
genman Oct 2, 2013 7:29 PMHi, I'm seeing the RHQ server use a lot of CPU. I didn't really do a lot of deep analysis, but this is what I'm seeing a lot of.
23:12:50,760 INFO [org.rhq.enterprise.server.util.concurrent.AvailabilityReportSerializer] (http-/0.0.0.0:7080-33) tid=433; agent=x: releasing write lock after being locked for millis=6075 23:13:46,624 INFO [org.rhq.enterprise.server.util.concurrent.AvailabilityReportSerializer] (http-/0.0.0.0:7080-14) tid=193; agent=y: releasing write lock after being locked for millis=5629 23:14:34,826 INFO [org.rhq.enterprise.server.util.concurrent.AvailabilityReportSerializer] (http-/0.0.0.0:7080-3) tid=178; agent=z: releasing write lock after being locked for millis=6273
This is with about 200 agents.
It seems like it's doing a lot of this. Basically hitting JMX stuff for whatever reason which is fairly inefficient. Maybe this is better in JDK 1.7, but I am using 1.6.
"http-/0.0.0.0:7080-192" daemon prio=10 tid=0x00007f5118124000 nid=0x40d waiting for monitor entry [0x00007f505547f000] java.lang.Thread.State: BLOCKED (on object monitor) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1088) - waiting to lock <0x000000040017b1e0> (a com.sun.jmx.interceptor.DefaultMBeanServerInterceptor) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:833) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:288) at $Proxy140.execute(Unknown Source) at org.rhq.enterprise.communications.command.server.CommandProcessor.handleIncomingInvocationRequest(CommandProcessor.java:290) at org.rhq.enterprise.communications.command.server.CommandProcessor.invoke(CommandProcessor.java:184) at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:967) at org.jboss.remoting.transport.servlet.ServletServerInvoker.processRequest(ServletServerInvoker.java:416) at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:527) at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:263) at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:288) at $Proxy145.processRequest(Unknown Source) at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet$3.run(ServerInvokerServlet.java:413)
There seems to be a lot of security stuff going on:
"http-/0.0.0.0:7080-54" daemon prio=10 tid=0x00007fd7e4043000 nid=0xf80 waiting for monitor entry [0x00007fd8679f5000] java.lang.Thread.State: BLOCKED (on object monitor) at java.util.Collections$SynchronizedMap.get(Collections.java:1979) - waiting to lock <0x00000004006cc6e8> (a java.util.Collections$SynchronizedMap) at java.security.ProtectionDomain$2$1.get(ProtectionDomain.java:414) at sun.security.provider.PolicyFile.implies(PolicyFile.java:1108) at org.jboss.security.jacc.DelegatingPolicy.implies(DelegatingPolicy.java:160) at java.security.ProtectionDomain.implies(ProtectionDomain.java:224) at java.security.AccessControlContext.checkPermission(AccessControlContext.java:352) at java.security.AccessController.checkPermission(AccessController.java:549) at java.lang.SecurityManager.checkPermission(SecurityManager.java:532) at java.lang.reflect.AccessibleObject.setAccessible(AccessibleObject.java:107) at org.rhq.enterprise.server.safeinvoker.HibernateDetachUtility.nullOutFieldsByFieldAccess(HibernateDetachUtility.java:513) "http-/0.0.0.0:7080-57" daemon prio=10 tid=0x00007fd7e4046000 nid=0x1931 runnable [0x00007fd8673ef000] java.lang.Thread.State: RUNNABLE at java.security.AccessController.getStackAccessControlContext(Native Method) at java.security.AccessController.checkPermission(AccessController.java:523) at java.lang.SecurityManager.checkPermission(SecurityManager.java:532) at java.lang.reflect.AccessibleObject.setAccessible(AccessibleObject.java:107) at org.rhq.enterprise.server.util.CriteriaQueryGenerator.getFilterFields(CriteriaQueryGenerator.java:695) at org.rhq.enterprise.server.util.CriteriaQueryGenerator.setBindValues(CriteriaQueryGenerator.java:887) at org.rhq.enterprise.server.util.CriteriaQueryGenerator.getCountQuery(CriteriaQueryGenerator.java:882) at org.rhq.enterprise.server.util.CriteriaQueryRunner.execute(CriteriaQueryRunner.java:78) at org.rhq.enterprise.server.resource.ResourceManagerBean.findResourcesByCriteria(ResourceManagerBean.java:2614) at sun.reflect.GeneratedMethodAccessor134.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72)
Again, I haven't really profiled the server heavily, but it is a bit of a concern.