12 Replies Latest reply on Apr 3, 2013 1:24 PM by Clebert Suconic

    horentq-failure-check-thread stopps removing death client connections

    Michael Woldrich Newbie

      We are using hornetq 2.2.14 and OIO

      It seems that the hornetq-failure-check-thread is blocking a very long time (hours) on getting snchronized "lock" Object. There is no death-lock detected

       

      Thread[hornetq-failure-check-thread,5,jboss]

          org.hornetq.core.server.impl.ServerConsumerImpl.setStarted(ServerConsumerImpl.java:499)

          org.hornetq.core.server.impl.ServerSessionImpl.doRollback(ServerSessionImpl.java:1427)

          org.hornetq.core.server.impl.ServerSessionImpl.rollback(ServerSessionImpl.java:692)

          org.hornetq.core.server.impl.ServerSessionImpl.doClose(ServerSessionImpl.java:300)

          org.hornetq.core.server.impl.ServerSessionImpl.access$100(ServerSessionImpl.java:87)

          org.hornetq.core.server.impl.ServerSessionImpl$1.done(ServerSessionImpl.java:1089)

          org.hornetq.core.persistence.impl.nullpm.NullStorageManager.afterCompleteOperations(NullStorageManager.java:400)

          org.hornetq.core.server.impl.ServerSessionImpl.close(ServerSessionImpl.java:1079)

          org.hornetq.core.server.impl.ServerSessionImpl.connectionFailed(ServerSessionImpl.java:1352)

          org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:579)

          org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:336)

          org.hornetq.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread.run(RemotingServiceImpl.java:631)

       

       

      And a lot of:

       

      Thread[Thread-69 (HornetQ-server-HornetQServerImpl::serverUUID=f98644db-6fa1-11e2-aa99-efbb161a08cc-3765233),5,HornetQ-server-HornetQServerImpl::serverUUID=f98644db-6fa1-11e2-aa99-efbb161a08cc-3765233]

          org.hornetq.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:174)

          org.hornetq.core.protocol.core.impl.ChannelImpl.sendBatched(ChannelImpl.java:162)

          org.hornetq.core.protocol.core.impl.CoreSessionCallback.sendMessage(CoreSessionCallback.java:76)

          org.hornetq.core.server.impl.ServerConsumerImpl.deliverStandardMessage(ServerConsumerImpl.java:798)

          org.hornetq.core.server.impl.ServerConsumerImpl.handle(ServerConsumerImpl.java:313)

          org.hornetq.core.server.impl.QueueImpl.handle(QueueImpl.java:2200)

          org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1751)

          org.hornetq.core.server.impl.QueueImpl.doPoll(QueueImpl.java:1630)

          org.hornetq.core.server.impl.QueueImpl.access$1300(QueueImpl.java:77)

          org.hornetq.core.server.impl.QueueImpl$ConcurrentPoller.run(QueueImpl.java:2487)

          org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)

          java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

          java.lang.Thread.run(Thread.java:619)

       

      Thread[Thread-227 (HornetQ-server-HornetQServerImpl::serverUUID=f98644db-6fa1-11e2-aa99-efbb161a08cc-3765233),5,HornetQ-server-HornetQServerImpl::serverUUID=f98644db-6fa1-11e2-aa99-efbb161a08cc-3765233]

          sun.misc.Unsafe.park(Native Method)

          java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

          java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)

          java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)

          java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)

          java.util.concurrent.Semaphore.acquire(Semaphore.java:286)

          org.hornetq.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:182)

          org.hornetq.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:225)

          org.hornetq.core.protocol.core.impl.ChannelImpl.sendBatched(ChannelImpl.java:162)

          org.hornetq.core.protocol.core.impl.CoreSessionCallback.sendMessage(CoreSessionCallback.java:76)

          org.hornetq.core.server.impl.ServerConsumerImpl.deliverStandardMessage(ServerConsumerImpl.java:798)

          org.hornetq.core.server.impl.ServerConsumerImpl.handle(ServerConsumerImpl.java:313)

          org.hornetq.core.server.impl.QueueImpl.handle(QueueImpl.java:2200)

          org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1751)

          org.hornetq.core.server.impl.QueueImpl.doPoll(QueueImpl.java:1630)

          org.hornetq.core.server.impl.QueueImpl.access$1300(QueueImpl.java:77)

          org.hornetq.core.server.impl.QueueImpl$ConcurrentPoller.run(QueueImpl.java:2487)

          org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)

          java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

          java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

          java.lang.Thread.run(Thread.java:619)

       

      ...

       

      We run into this problem when the terminal server, logoff client sessions after inactivity and hornetq must clean up death connections. This could happen very often.

      If the cleanup thread did not work a long time, we sometimes run in a out of memory.

       

      I think there is a relation to discussion: https://community.jboss.org/thread/203648.

      I hope there is another opportunity, so we do not have to change from OIO to NIO.

       

      I attache a thread dump.

      If needed, I will send you a heap-dump.

       

      Greetings