stopping HornetQ threads on undeploy in Tomcat

[jhrobbin]

Hi folks,

This is my first HornetQ post.  Thanks in advance for any guidance.

I'm troubleshooting an app that runs on Tomcat.  This app is failing with Permgen errors after several hot deploys.

I am browsing the logs (catalina.out), the thread dump and heap dump (using VisualVM).

Tomcat is complaining about about HornetQ threads not stopping:

Aug 14, 2012 5:10:38 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/api] appears to have started a thread named [Thread-1 (HornetQ-client-factory-threads-1021336883-271086502)] but has failed to stop it. This is very likely to create a memory leak.

Aug 14, 2012 5:10:38 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/api] appears to have started a thread named [Thread-0 (HornetQ-client-factory-pinger-threads-1021336883-1338023827)] but has failed to stop it. This is very likely to create a memory leak.

Aug 14, 2012 5:10:38 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

The thread dump shows HornetQ threads:

"Thread-3 (HornetQ-client-factory-pinger-threads-1021336883-1338023827)" daemon prio=6 tid=0x0000000007092000 nid=0x12b4 waiting on condition [0x0000000014b0f000]

   java.lang.Thread.State: WAITING (parking)

    at sun.misc.Unsafe.park(Native Method)

    - parking to wait for  <0x00000000f3db2968> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)

    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)

    at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)

    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)

    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)

    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)

    at java.lang.Thread.run(Thread.java:662)

I found this forum post "HornetQ fails to stop threads in Tomcat" which recommends setting the Global Pool property to false.  I have added this property to the connection factory as described in the "Client-Side Thread Management" section of documentation.  I think HornetQ is recognizing this setting because I see the name of the threads changes from "HornetQ-client-global-scheduled-threads" to "HornetQ-client-factory-pinger-threads" after setting the Global Pool property.

I am using version 2.2.10.Final of HornetQ.  We are also using Spring (3.1.1.RELEASE), Tomcat 7.x..  Spring is using the SpringJmsBootstrap class.

The connection-factory config in hornetq-jms.xml looks like this:

<connection-factory name="ConnectionFactory">
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="ConnectionFactory"/>
</entries>
<use-global-pools>false</use-global-pools>
<!-- <scheduled-thread-pool-max-size>10</scheduled-thread-pool-max-size>   -->
<!-- <thread-pool-max-size>-1</thread-pool-max-size> -->
</connection-factory>

Lastly, I debugged the shutdown in Eclipse and confirmed the ServerLocatorImpl.doClose is getting called.

Please let me know if there is any other info I can provide to help troubleshoot.  Many thanks!

[jhrobbin]

Another thing I noticed in the Tomcat log is confirming HornetQ has indeed stopped.  The following is written to the log BEFORE Tomcat complains about HornetQ related threads that failed to stop:

INFO: HornetQ Server version 2.2.10.Final (HQ_2_2_10_FINAL_AS7, 122) [3140af11-da7f-11e1-95df-c417fe569f49] stopped

If HornetQ Server is stopped then how is it possible there are threads running at that poing?

Any advice is greatly appreciated!

[leosbitto]

I have solved similar issue in the past by calling setUseGlobalPools(false) on org.hornetq.jms.client.HornetQConnectionFactory, found by studying the source code. Nowdays this is even documented: https://access.redhat.com/knowledge/docs/en-US/JBoss_Enterprise_Application_Platform/5/html/HornetQ_User_Guide/thread-pooling.client.side.html

[leosbitto]

Oh, I see that you actually use <use-global-pools>false</use-global-pools> in some configuration file. I have not used this kind of configuration, I have instantiated HornetQConnectionFactory as a Spring bean and here comes the trick: I had attribute destroy-method="close" there. So when destroying the Spring ApplicationContext, method close() gets called at HornetQConnectionFactory, which (in combination with <property name="useGlobalPools" value="false" />) leads to proper shutdown of HornetQ client threads. Yes, those are HornetQ client threads - that explains why HornetQ server shutdown has nothing to do with them, even if the server is embedded in the same JVM.

[jhrobbin]

I think you are right, the HornetQ server is shutting down but the HornetQ client is leaving running "HornetQ-client-factory-pinger-threads" in Tomcat.

I am using JNDI to instantiate the HornetQConnectionFactory as describe here and as demonstrated in the "spring-integration" example that comes with the HornetQ download.

I can confirm the constructor of HornetQConnectionFactory is called during server startup but HornetQConnectionFactory.close() is not called when the application is stopped in Tomcat.

Is calling HornetQConnectionFactory.close() the key to stopping the HornetQ client factory threads when stopping the application?

If so, how can I make sure HornetQConnectionFactory.close() is called when configuring HornetQ with Spring using JNDI?

Many Thanks!

[leosbitto]

Justin Robbins wrote:

Is calling HornetQConnectionFactory.close() the key to stopping the HornetQ client factory threads when stopping the application?

I think so, check the source code yourself to be sure. The method close() is delegated to org.hornetq.core.client.impl.ServerLocatorImpl, which stops the threads if useGlobalPools is set to false.

If so, how can I make sure HornetQConnectionFactory.close() is called when configuring HornetQ with Spring using JNDI?

What about adding the attribute destroy-method="close" to the definition of the Spring bean which represents HornetQConnectionFactory (loaded from JNDI in your case, but you could construct it directly as well)? I hope that your deployment of the Spring ApplicationContext gets its method close() called when undeploying your application (that's the only correct way when using Spring), which should lead to calling all destroy-methods of the beans which have them configured.

[jhrobbin]

What about adding the attribute destroy-method="close" to the definition of the Spring bean which represents HornetQConnectionFactory (loaded from JNDI in your case, but you could construct it directly as well)?

You are right, I can't add destroy-method="close" to the definition because the HornetQConnectionFactory is provided from JNDI and not a Spring bean.  I tried defining the ConnectionFactory as a Spring bean but HornetQ was still using the instance from JNDI.

However, I am able to inject a reference to JNDI created ConnectionFactory into another Spring bean.  In that bean's destroy I'm now calling close() on HornetQConnectionFactory.  Perhaps there is a better way...  I need to do some more testing but I think this might suffice.