Memory leak by closed connections
rschneider Jul 10, 2013 8:12 AMI found this bug (pretty sure about it being a bug, would love to directly file an issue in the issue tracker, but that doesn't seem to be possible without having some SalesForce "support case reference") when porting a real application which explicitly supports offline scenarios from JBoss 6 to 7.1.3:
When an EJB client gets disconnected from its JBoss 7 server, for example because that server has been shut down, but keeps running and eventually reconnects to the restarted server, a new org.jboss.remoting3.ConnectionImpl instance is created for that new connection. The old connection can not be garbage collected, however, as there are still two strong references lingering around to that connection in the jboss-ejb-client module. I refer to version 1.0.22.Final, which seems to be the most current final version (big difference from 1.0.11.Final, which is by default packaged in JBoss 7.1.3 - one of the code parts in which the problematic references reside has been refactored somewhere in between!)
The first reference can be found in org.jboss.ejb.client.remoting.RemotingCleanupHandler in the collection named "connections", which obviously only gets connections added to it, and NEVER removed. The second reference is similar in nature and can be found in org.jboss.ejb.client.remoting.RemotingConnectionManager in the collection "managedConnections". It is also evident from the code that connections can never be removed from that collection, not even after they were closed due to whatever circumstance.
As both of these collections seem to be used only for closing a bunch of connections at once, I devised a small fix for the problem which replaces the strong-referenced collections with WeakHashMaps. Closed connections automatically fall out of those, and connections still in use are always referenced by at least one other, strong reference. After applying that fix (and fixing another reference-to-closed-connection-bug in JBoss Remoting 3, as I described in https://community.jboss.org/message/827298#827298) dead connections finally were garbage collected. I attached a patch file to this posting. I'm not at all sure whether this fix is the most elegant one (okay, probably not...) but it successfully solved the problem for my application.
Steps to reproduce the issue:
- Create an EJB client and service, deploy service in JBoss7
- Have the EJB client call the service on the server frequently
- Shut down the server. The client must be built to continue calling the server, even though exceptions are now thrown.
- Restart the server. The client will reconnect and continue without errors.
- Fetch a heap dump from the client. There are now two connection objects in the heap, a closed one and an open one. Rinse and repeat for having 3,4,5,...
It is reproducible with the modules delivered within JBoss 7.1.3 as well as the newer versions of ejb-client 1.0.22.Final and remoting3 3.2.16.GA.