Oracle TimesTen driver; cannot reload native library

[genman]

The plugin I wrote works, but it seems like the classloader for the plugin doesn't unload its libraries, so I must do a full restart of the agent. It seems that ClassLoader is GC'd but maybe not in all cases actually it will go away.

java.sql.SQLException: Problems with loading native library/missing methods: Native Library /u01/app/timesten/product/11.2.2.6.4/TimesTen/tt1122/lib/libttJdbcCS.so already loaded in another classloader
        at com.timesten.jdbc.JdbcOdbcConnection.connect(JdbcOdbcConnection.java:1809)
        at com.timesten.jdbc.TimesTenDriver.connect(TimesTenDriver.java:305)
        at com.timesten.jdbc.TimesTenDriver.connect(TimesTenDriver.java:161)
        at org.rhq.plugins.database.DriverDataSource.getConnection(DriverDataSource.java:96)
        at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:341)
        at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:416)
        at org.rhq.plugins.database.BasePooledConnectionProvider.<init>(BasePooledConnectionProvider.java:140)

I'm wondering if there is a trick to getting this to work? Does somebody have a good trick to diagnose the classloader leak?

[lkrejci]

Use Eclipse Mat, generate a bunch of heapdumps of you agent process, load some of them in mat, search for RootPluginClassLoader class (which is a parent classloader of all plugin classloaders) and have mat show you the GC roots of instances of that class. That will show you what, if anything, is holding on to any of the plugin classloaders when it shouldn't. Usually if you see that there are as many RootPluginClassLoader instances as there were restarts of the plugin container, you know there's something leaking (and yes, I just discovered a leak in postgres JDBC driver using that method   - Avoid a classloader leak through a dangling statement cancel timer thread. by metlos · Pull Request #188 · pgjdbc/pgjdbc…).

[genman]

Well, I found a leak in my own plugin, but not anything obviously in the driver. I have the DriverManager unload in place, so that's not it.

I do see this strangeness:

Class Name                                                            | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap
----------------------------------------------------------------------------------------------------------------------------------------
class com.timesten.jdbc.OdbcLobId @ 0xe04cc3b8 Native Stack           |            2 |            8 |              192 |            8
'- <classloader> org.rhq.core.pc.plugin.PluginClassLoader @ 0xe04833c8|            2 |          96 |              192 |        31,256 
  '- parent org.rhq.core.pc.plugin.PluginClassLoader @ 0xe0483428     |            1 |          96 |                96 |        25,464
----------------------------------------------------------------------------------------------------------------------------------------

I don't have the source to OdbcLobId, but in its disassembly I don't see anything native related. I'm wondering if somehow the native library is loading classes the wrong way? See "Native Stack" above.

If you'd like to check the heap, I can mail it to you zipped.