JMX threads blocked - agent creates too many threads
This is probably a Bugzilla item, but my RHQ agent (4.9 patched) keeps creating new threads (> 500 currently), which block. The blocked thread is first, the non blocked thread is after.
It seems the JMX remoting (RMI) should at least time out after some time, but I'm guessing this is not the case. Or possibly change JMXServerComponent.reconnectIfDisconnected to use read/write locks that can time out.
"ResourceContainer.invoker.daemon-12388" daemon prio=10 tid=0x00002aaad4548800 nid=0x31da waiting for monitor entry [0x00002aab13d7c000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.rhq.plugins.jmx.JMXServerComponent.reconnectIfDisconnected(JMXServerComponent.java:134)
- waiting to lock <0x00002aaab526dad0> (a org.rhq.plugins.hadoop.HadoopComponent)
at org.rhq.plugins.jmx.JMXServerComponent.getAvailability(JMXServerComponent.java:114)
at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocation.call(ResourceContainer.java:654)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
"ResourceContainer.invoker.daemon-10540" daemon prio=10 tid=0x00002aaae8010800 nid=0x2fbc runnable [0x00000000464e3000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
- locked <0x00002aaab59666f0> (a java.io.BufferedInputStream)
at java.io.DataInputStream.readByte(DataInputStream.java:248)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:228)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110)
at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl_Stub.getDefaultDomain(Unknown Source)
at javax.management.remote.rmi.RMIConnector$RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1444)
at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.queryNames(RMIConnector.java:824)
at sun.reflect.GeneratedMethodAccessor44429.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.mc4j.ems.impl.jmx.connection.support.providers.proxy.JMXRemotingMBeanServerProxy.invoke(JMXRemotingMBeanServerProxy.java:59)
at $Proxy47.queryNames(Unknown Source)
at org.mc4j.ems.impl.jmx.connection.support.providers.JMXRemotingConnectionProvider.isConnected(JMXRemotingConnectionProvider.java:171)
at org.rhq.plugins.jmx.JMXServerComponent.reconnectIfDisconnected(JMXServerComponent.java:134)
- locked <0x00002aaab515bc08> (a org.rhq.plugins.jmx.JMXServerComponent)
at org.rhq.plugins.jmx.JMXServerComponent.getAvailability(JMXServerComponent.java:114)
at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.rhq.core.pc.inventory.ResourceContainer$ComponentInvocation.call(ResourceContainer.java:654)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
