4 Replies Latest reply on Aug 29, 2016 2:36 PM by wdarby

Memory & thread leak when application undeployed?

wdarby Aug 24, 2016 3:37 PM

In migrating from JBoss AS 7.1 to Wildfly 10.0 I discovered that my application classes are not being unloaded when the application is unloaded, and therefore any objects referenced by static members are not garbage collected. Using an hprof analysis I determined this is because the threads named "default-threads - #" do not release their contextClassLoader, and thus the classes are not garbage collected.

In addition, every time I deploy the application 4 new "default-threads - #" threads are started. So it appears both memory and threads are being leaked when applications are unloaded.

Is there any way to tune the "default-threads- #" pool (I can't tell what subsystem starts them).

Does this appear to be a bug that should be reported?

Thanks

1. Re: Memory & thread leak when application undeployed?

jaikiran Aug 25, 2016 1:58 AM (in response to wdarby)

WildFly 10.1.0.Final has been recently released WildFly 10.1 is now available! · WildFly, can you try this against that version and see if you can reproduce it? Can you also attach the complete thread dump here so that one of us can take a look? What kind of components does your application have? JSF, Weld or any such?
Actions

2. Re: Memory & thread leak when application undeployed?

wdarby Aug 25, 2016 8:53 PM (in response to wdarby)

Thanks for the response. I upgraded to 10.1 and see the same behavior. I'm attaching a link the jstack log file. It is for a Wildfly process that has had the application deployed and undeployed 3 times. Note there are 12 "default-threads #", 4 for each deployment (the number increases with each deployment).

Also below is the hprof dump for trace from the threads to the class and static data

Let me know what you think

Thanks

Class Name	Shallow Heap	Retained Heap
org.hibernate.internal.SessionFactoryImpl @ 0xe3c28020	120	9,018,008
sessionFactory com.leeriver.trading.pjm.data.PjmMarketDaoImpl @ 0xe3c27ff8	24	24
target org.springframework.aop.target.SingletonTargetSource @ 0xe3c27fe8	16	40
targetSource org.springframework.aop.framework.ProxyFactory @ 0xe3c27fa8	64	2,560
advised org.springframework.aop.framework.JdkDynamicAopProxy @ 0xe3c27f90	24	2,584
h com.sun.proxy.$Proxy165 @ 0xe3c27f80	16	2,600
dao com.leeriver.trading.pjm.analysis.data.constraint.access.RealTimeConstraintDataType @ 0xe3c27a38	24	1,352
rtConstraintDataType class com.leeriver.trading.pjm.analysis.data.constraint.access.RealTimeConstraintDataSource @ 0xe3c279c0	16	1,368
[515] java.lang.Object[2560] @ 0xe469f848	10,256	76,152
elementData java.util.Vector @ 0xe38f70a8	32	76,184
classes org.jboss.modules.ModuleClassLoader @ 0xe38f6f48	88	170,712
<classloader> class com.leeriver.trading.pjm.eDataFeed.EDataConstraintChecker @ 0xe3961ad0	8	8
<super> class com.leeriver.trading.pjm.eDataFeed.EDataConstraintChecker$$EnhancerBySpringCGLIB$$cd18c63a @ 0xe430b710	112	2,608
[63] java.lang.Object[160] @ 0xe4531540	656	7,328
elementData java.util.Vector @ 0xe38f56e0	32	7,360
classes org.jboss.modules.ModuleClassLoader @ 0xe38f5580	88	16,672
contextClassLoader org.jboss.threads.JBossThread @ 0xe3e2c2d8 default-threads - 7 Thread	136	600
contextClassLoader org.jboss.threads.JBossThread @ 0xe3e44228 default-threads - 8 Thread	136	600
contextClassLoader org.jboss.threads.JBossThread @ 0xe3e445a0 default-threads - 5 Thread	136	464
contextClassLoader org.jboss.threads.JBossThread @ 0xe3e44890 default-threads - 6 Thread	136	600
Total: 4 entries
referent java.lang.ref.Finalizer @ 0xe3936a88 »	40	154,944
Total: 2 entries

3. Re: Memory & thread leak when application undeployed?

wdarby Aug 25, 2016 10:17 PM (in response to wdarby)

Forgot to mention the configured subsystems:
<extensions>
        <extension module="org.jboss.as.clustering.infinispan"/>
        <extension module="org.jboss.as.connector"/>
        <extension module="org.jboss.as.deployment-scanner"/>
        <extension module="org.jboss.as.ee"/>
        <extension module="org.jboss.as.ejb3"/>
        <extension module="org.jboss.as.jdr"/>
        <extension module="org.jboss.as.jmx"/>
        <extension module="org.jboss.as.logging"/>
        <extension module="org.jboss.as.mail"/>
        <extension module="org.jboss.as.naming"/>
        <extension module="org.jboss.as.pojo"/>
        <extension module="org.jboss.as.remoting"/>
        <extension module="org.jboss.as.sar"/>
        <extension module="org.jboss.as.security"/>
        <extension module="org.jboss.as.transactions"/>
        <extension module="org.jboss.as.weld"/>
        <extension module="org.wildfly.extension.batch.jberet"/>
        <extension module="org.wildfly.extension.bean-validation"/>
        <extension module="org.wildfly.extension.io"/>
        <extension module="org.wildfly.extension.messaging-activemq"/>
        <extension module="org.wildfly.extension.request-controller"/>
        <extension module="org.wildfly.extension.security.manager"/>
        <extension module="org.wildfly.extension.undertow"/>
    </extensions>
Actions
4. Re: Memory & thread leak when application undeployed?

wdarby Aug 29, 2016 2:36 PM (in response to wdarby)

After exploring, I learned the "default-threads - #" are part of the JCA subsystem, which is initially configured with max-threads and core-threads of 50. These are used by MDBs if the <resource-adapter-ref> is set to activemq-ra (a pooled-connection factory in messaging-activemq). Since my application starts 4 MDBs, there are 4 instances of default-threads created per deployment.

I modified the configuration for 8 core-threads and 8 max-threads in the JCA <defaul-workmanager>, and for the <pooled-connection-factory> set min-pool-size to 4 and max-pool-size to 8 (it's a relatively small application). Now the threads are reused every other deployment, which releases the classes and corresponding static memory. Not a bug after all, just a configuration issue.
Actions

Go to original post