3 Replies Latest reply on Aug 20, 2007 12:03 PM by starksm64

JBAS-4593 -- Leaking classloader via JMX Notifications

brian.stansberry Aug 15, 2007 4:24 PM

Mike Millson asked me to have a look at http://jira.jboss.com/jira/browse/JBAS-4593; reporting here what I've found:

The heap info posted on the case shows a path leading from a UnifiedClassLoader3 through a JMX Notification and on to a heap root.
When an instance of UnifiedClassLoader3 is created, it causes the generation of a JMX Notification with the classloader as the Notification userData. The path to heap root shows one of those notification being held in a buffer in Sun's internal JMX implementation classes. User is using the platform mbeanserver as discussed on http://wiki.jboss.org/wiki/Wiki.jsp?page=JBossMBeansInJConsole.the ke

From poking around in the JDK code, I believe the buffer that's holding the Notification is doing so to deliver the notifications to a remote JMX client. Looks like the Sun JMX code creates a buffer the first time a remote JMX connection is established. Buffer holds a configurable number of Notifications, default is 1000. Size of buffer can be controlled by system property "jmx.remote.x.notification.buffer.size".

At this point I'm not sure what to do about the issue. Some options:

1) Get rid of the JMX notifications.
2) Wrap the classloader in a WeakReference before adding to the notification.

3) Workaround: help user figure out what's making the remote JMX connection and how to get it to drain the buffer.
4) Workaround: help user configure a smaller buffer size so the leak is cleared more quickly as the old notification is purged from the buffer.

1. Re: JBAS-4593 -- Leaking classloader via JMX Notifications

clebert.suconic Aug 17, 2007 10:58 AM (in response to brian.stansberry)

I could see is UnifiedLoaderRepository3 indeed sends a notificaton of the classLoader... and if you buffer such notification that would generate a leak.

http://anonsvn.jboss.org/repos/jbossas/branches/Branch_4_2/jmx/src/main/org/jboss/mx/loading/UnifiedLoaderRepository3.java

I don't know how to avoid the buffer on the sun's implementation.

One thing that I thought though, was to make the attribute source on javax.management.Notification a WeakReference, but sounds a weird solution. (Brain storm).

http://anonsvn.jboss.org/repos/jbossas/branches/Branch_4_2/jmx/src/main/javax/management/Notification.java

Another possibility would be to not send ClassLoading notifications when using Sun's JMX? (Another brain storm)

This is a job for JMX/ClassLoader experts.
Actions
2. Re: JBAS-4593 -- Leaking classloader via JMX Notifications

shasse Aug 20, 2007 10:54 AM (in response to brian.stansberry)

I can confirm that in our case at least the remote JMX connections are coming from a jconsole instance that is used to watch the perm gen size and initiate various MBean actions.

Scott
Actions
3. Re: JBAS-4593 -- Leaking classloader via JMX Notifications

starksm64 Aug 20, 2007 12:03 PM (in response to brian.stansberry)

There are no expected listeners, its just part of the jmx management
interface to emit notifications when class loaders are added/removed
from a repo. I do see that there is a
org.jboss.aop.deployment.LoaderRepositoryUrlUtil class referencing the
CLASSLOADER_ADDED/CLASSLOADER_REMOVED event types to track some internal
sequence number, but that is the only explicit usage in the 4.2 codebase
I can find. In the past there were thoughts about using these notifications to handle implicit deployment dependencies based on usage of a class from the repository, but this was never fleshed out.

I don't see a problem with wrapping the class loader in a WeakReference
to avoid the jmx layer being a strong referent. NotificationListeners
would have to be aware that the Notification user data has changed to a
WeakReference. This could be added as an optional behavior based on a
system property. Notification emission could also be disabled altogether as well based on this property.
Actions

Go to original post