stopping HornetQ threads on undeploy in Tomcat
jhrobbin Aug 14, 2012 5:36 PMHi folks,
This is my first HornetQ post. Thanks in advance for any guidance.
I'm troubleshooting an app that runs on Tomcat. This app is failing with Permgen errors after several hot deploys.
I am browsing the logs (catalina.out), the thread dump and heap dump (using VisualVM).
Tomcat is complaining about about HornetQ threads not stopping:
Aug 14, 2012 5:10:38 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/api] appears to have started a thread named [Thread-1 (HornetQ-client-factory-threads-1021336883-271086502)] but has failed to stop it. This is very likely to create a memory leak.
Aug 14, 2012 5:10:38 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/api] appears to have started a thread named [Thread-0 (HornetQ-client-factory-pinger-threads-1021336883-1338023827)] but has failed to stop it. This is very likely to create a memory leak.
Aug 14, 2012 5:10:38 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
The thread dump shows HornetQ threads:
"Thread-3 (HornetQ-client-factory-pinger-threads-1021336883-1338023827)" daemon prio=6 tid=0x0000000007092000 nid=0x12b4 waiting on condition [0x0000000014b0f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000f3db2968> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
I found this forum post "HornetQ fails to stop threads in Tomcat" which recommends setting the Global Pool property to false. I have added this property to the connection factory as described in the "Client-Side Thread Management" section of documentation. I think HornetQ is recognizing this setting because I see the name of the threads changes from "HornetQ-client-global-scheduled-threads" to "HornetQ-client-factory-pinger-threads" after setting the Global Pool property.
I am using version 2.2.10.Final of HornetQ. We are also using Spring (3.1.1.RELEASE), Tomcat 7.x.. Spring is using the SpringJmsBootstrap class.
The connection-factory config in hornetq-jms.xml looks like this:
<connection-factory name="ConnectionFactory"> | |
<connectors> | |
<connector-ref connector-name="in-vm"/> | |
</connectors> | |
<entries> | |
<entry name="ConnectionFactory"/> | |
</entries> | |
<use-global-pools>false</use-global-pools> | |
<!-- <scheduled-thread-pool-max-size>10</scheduled-thread-pool-max-size> --> | |
<!-- <thread-pool-max-size>-1</thread-pool-max-size> --> | |
</connection-factory> |
Lastly, I debugged the shutdown in Eclipse and confirmed the ServerLocatorImpl.doClose is getting called.
Please let me know if there is any other info I can provide to help troubleshoot. Many thanks!