A possible leak in management and failover tests

[clebert.suconic]

I have found a few leaks already on the code. Nothing that stops production...but affects the testsuite because of stop/start the servers. (stress tests are affect by those)

I have added -XX:+PrintClassHistogram to the testsuite. If you call kill -3 you would see QueueImpl, MessageReference growing a lot, mainly on all the MultiThreadFailoverTests.

One of the leaks I saw was Remoting has a reference to Management that has references to Queues. The management reference is never cleared. (maybe?)

Another I saw was during a run on XALargeMessageMultiThreadFaviloverTest. Somewhere on management has a reference to the resourceManager. Operations are never released of that resourcemanager (maybe more than 1 resourceManager instance here). queues and messages are building up and the test could get OME depending on the memory settings.


I would need more time to investigate those.. but maybe Tim and Jeff would have a better clue since this is your guys areas.

[clebert.suconic]

another one is channels on RemotingConnection:

references on QueueImpl(name=7sub2)
!-- FieldReference private final org.jboss.messaging.core.server.Queue org.jboss.messaging.core.server.impl.ServerConsumerImpl.messageQueue=TOSTRING(org.jboss.messaging.core.server.impl.ServerConsumerImpl@326b9c84), class=org.jboss.messaging.core.server.impl.ServerConsumerImpl
!--!-- FieldReference volatile java.lang.Object java.util.concurrent.ConcurrentHashMap$HashEntry.value=TOSTRING(java.util.concurrent.ConcurrentHashMap$HashEntry@54e9a7c2), class=java.util.concurrent.ConcurrentHashMap$HashEntry
!--!--!-- arrayRef [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;[0] id=@939842843
!--!--!--!-- FieldReference transient volatile java.util.concurrent.ConcurrentHashMap$HashEntry[] java.util.concurrent.ConcurrentHashMap$Segment.table=TOSTRING(java.util.concurrent.ConcurrentHashMap$Segment@a25ef73[Unlocked]), class=java.util.concurrent.ConcurrentHashMap$Segment
!--!--!--!--!-- arrayRef [Ljava.util.concurrent.ConcurrentHashMap$Segment;[10] id=@1082879033
!--!--!--!--!--!-- FieldReference final java.util.concurrent.ConcurrentHashMap$Segment[] java.util.concurrent.ConcurrentHashMap.segments=TOSTRING({0=org.jboss.messaging.core.server.impl.ServerConsumerImpl@326b9c84}), class=java.util.concurrent.ConcurrentHashMap
!--!--!--!--!--!--!-- FieldReference private final java.util.Map org.jboss.messaging.core.server.impl.ServerSessionImpl.consumers=TOSTRING(org.jboss.messaging.core.server.impl.ServerSessionImpl@4cc5f20f), class=org.jboss.messaging.core.server.impl.ServerSessionImpl
!--!--!--!--!--!--!--!-- FieldReference private final org.jboss.messaging.core.server.ServerSession org.jboss.messaging.core.server.impl.ServerSessionPacketHandler.session=TOSTRING(org.jboss.messaging.core.server.impl.ServerSessionPacketHandler@6c229ed4), class=org.jboss.messaging.core.server.impl.ServerSessionPacketHandler
!--!--!--!--!--!--!--!--!-- FieldReference private org.jboss.messaging.core.remoting.ChannelHandler org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.handler=TOSTRING(org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl@5722cc7e), class=org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl
!--!--!--!--!--!--!--!--!--!-- FieldReference volatile java.lang.Object java.util.concurrent.ConcurrentHashMap$HashEntry.value=TOSTRING(java.util.concurrent.ConcurrentHashMap$HashEntry@4b7361e2), class=java.util.concurrent.ConcurrentHashMap$HashEntry
!--!--!--!--!--!--!--!--!--!--!-- arrayRef [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;[0] id=@69165857
!--!--!--!--!--!--!--!--!--!--!--!-- FieldReference transient volatile java.util.concurrent.ConcurrentHashMap$HashEntry[] java.util.concurrent.ConcurrentHashMap$Segment.table=TOSTRING(java.util.concurrent.ConcurrentHashMap$Segment@28a50da4[Unlocked]), class=java.util.concurrent.ConcurrentHashMap$Segment
!--!--!--!--!--!--!--!--!--!--!--!--!-- arrayRef [Ljava.util.concurrent.ConcurrentHashMap$Segment;[13] id=@1032432821
!--!--!--!--!--!--!--!--!--!--!--!--!--!-- FieldReference final java.util.concurrent.ConcurrentHashMap$Segment[] java.util.concurrent.ConcurrentHashMap.segments=OBJ(java.util.concurrent.ConcurrentHashMap@1532712380)
!--!--!--!--!--!--!--!--!--!--!--!--!--!--!-- FieldReference private final java.util.Map org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.channels=TOSTRING(org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl@4959d87f), class=org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl

[timfox]

This is all a bit vague - I'm not sure there's enough information here for us to understand/do anything about this right now.

[timfox]

Your dump just shows a connection holdings references to channels which is completely normal.

What we need to know is what has the reference to the remoting connection.

Also "somewhere on management has a reference to resource manager", well yes, the ManagementService has a reference to messaging server control which has a reference to resource manager, that's normal. But it doesn't help, what we need to know is what has the reference to the ManagementService and why it's not being cleared.

"Remoting has a reference to management". Really? I can't find any such reference in the code.

[clebert.suconic]

This *is* a bit vague, because it' s hard to find the main referencce that is holding both management and those connections.

Connections and management are still hanging around on that test, and it' s being hard to find why. I hoped you (Tim) or Jeff would have a clue.