horentq-failure-check-thread stopps removing death client connections
michael10 Mar 28, 2013 10:10 AMWe are using hornetq 2.2.14 and OIO
It seems that the hornetq-failure-check-thread is blocking a very long time (hours) on getting snchronized "lock" Object. There is no death-lock detected
Thread[hornetq-failure-check-thread,5,jboss]
org.hornetq.core.server.impl.ServerConsumerImpl.setStarted(ServerConsumerImpl.java:499)
org.hornetq.core.server.impl.ServerSessionImpl.doRollback(ServerSessionImpl.java:1427)
org.hornetq.core.server.impl.ServerSessionImpl.rollback(ServerSessionImpl.java:692)
org.hornetq.core.server.impl.ServerSessionImpl.doClose(ServerSessionImpl.java:300)
org.hornetq.core.server.impl.ServerSessionImpl.access$100(ServerSessionImpl.java:87)
org.hornetq.core.server.impl.ServerSessionImpl$1.done(ServerSessionImpl.java:1089)
org.hornetq.core.persistence.impl.nullpm.NullStorageManager.afterCompleteOperations(NullStorageManager.java:400)
org.hornetq.core.server.impl.ServerSessionImpl.close(ServerSessionImpl.java:1079)
org.hornetq.core.server.impl.ServerSessionImpl.connectionFailed(ServerSessionImpl.java:1352)
org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:579)
org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:336)
org.hornetq.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread.run(RemotingServiceImpl.java:631)
And a lot of:
Thread[Thread-69 (HornetQ-server-HornetQServerImpl::serverUUID=f98644db-6fa1-11e2-aa99-efbb161a08cc-3765233),5,HornetQ-server-HornetQServerImpl::serverUUID=f98644db-6fa1-11e2-aa99-efbb161a08cc-3765233]
org.hornetq.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:174)
org.hornetq.core.protocol.core.impl.ChannelImpl.sendBatched(ChannelImpl.java:162)
org.hornetq.core.protocol.core.impl.CoreSessionCallback.sendMessage(CoreSessionCallback.java:76)
org.hornetq.core.server.impl.ServerConsumerImpl.deliverStandardMessage(ServerConsumerImpl.java:798)
org.hornetq.core.server.impl.ServerConsumerImpl.handle(ServerConsumerImpl.java:313)
org.hornetq.core.server.impl.QueueImpl.handle(QueueImpl.java:2200)
org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1751)
org.hornetq.core.server.impl.QueueImpl.doPoll(QueueImpl.java:1630)
org.hornetq.core.server.impl.QueueImpl.access$1300(QueueImpl.java:77)
org.hornetq.core.server.impl.QueueImpl$ConcurrentPoller.run(QueueImpl.java:2487)
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:619)
Thread[Thread-227 (HornetQ-server-HornetQServerImpl::serverUUID=f98644db-6fa1-11e2-aa99-efbb161a08cc-3765233),5,HornetQ-server-HornetQServerImpl::serverUUID=f98644db-6fa1-11e2-aa99-efbb161a08cc-3765233]
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)
java.util.concurrent.Semaphore.acquire(Semaphore.java:286)
org.hornetq.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:182)
org.hornetq.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:225)
org.hornetq.core.protocol.core.impl.ChannelImpl.sendBatched(ChannelImpl.java:162)
org.hornetq.core.protocol.core.impl.CoreSessionCallback.sendMessage(CoreSessionCallback.java:76)
org.hornetq.core.server.impl.ServerConsumerImpl.deliverStandardMessage(ServerConsumerImpl.java:798)
org.hornetq.core.server.impl.ServerConsumerImpl.handle(ServerConsumerImpl.java:313)
org.hornetq.core.server.impl.QueueImpl.handle(QueueImpl.java:2200)
org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1751)
org.hornetq.core.server.impl.QueueImpl.doPoll(QueueImpl.java:1630)
org.hornetq.core.server.impl.QueueImpl.access$1300(QueueImpl.java:77)
org.hornetq.core.server.impl.QueueImpl$ConcurrentPoller.run(QueueImpl.java:2487)
org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:619)
...
We run into this problem when the terminal server, logoff client sessions after inactivity and hornetq must clean up death connections. This could happen very often.
If the cleanup thread did not work a long time, we sometimes run in a out of memory.
I think there is a relation to discussion: https://community.jboss.org/thread/203648.
I hope there is another opportunity, so we do not have to change from OIO to NIO.
I attache a thread dump.
If needed, I will send you a heap-dump.
Greetings
-
ThreadDump.txt.zip 13.5 KB