0 Replies Latest reply on Mar 6, 2014 4:57 AM by Koen Janssens

    hornetq paging blocks producers ?

    Koen Janssens Newbie

      Hi,

       

      I have noticed that hornetq blocks message producers while paging. This has a significant negative impact on the performance and i wonder whether this is really necessary.

      AFAIK, journaling is there to make sure no messages are lost and i can understand that a message must be written to the journal before it can be be considered 'sent'.

      However, paging looks like an internal memory management trick to me and should not block producers. From a 10000 feet perspective, it looks to me that paging could be done in an asynchrous way.

       

      To make matters worse, we are using XA transactions, meaning that jboss keeps a hold of a DB connection until hornetq paging ends

       

      Is there a way to prevent producers from waiting on the paging ??

       

      I'm using hornetq 2.3.5 (inside JBoss EAP 6.1.1.GA (AS 7.2.1.Final-redhat-10))

       

      Example thread that 'blocks' until hornetq has paged (there are a few hundred threads in this state):

       

      "http-/172.31.110.115:8280-1" daemon prio=10 tid=0x000000001aec9800 nid=0x478a waiting on condition [0x000000005818e000]

         java.lang.Thread.State: TIMED_WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x000000073816f838> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2176)

        at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:353)

        - locked <0x000000073816f878> (a java.lang.Object)

        at org.hornetq.core.client.impl.ClientSessionImpl.prepare(ClientSessionImpl.java:1563)

        at org.hornetq.ra.HornetQRAXAResource.prepare(HornetQRAXAResource.java:130)

        at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.prepare(XAResourceWrapperImpl.java:169)

        at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelPrepare(XAResourceRecord.java:213)

        at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2515)

        at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2482)

        at com.arjuna.ats.arjuna.coordinator.BasicAction.prepare(BasicAction.java:2059)

        - locked <0x00000007f48daee8> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)

        at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1481)

        - locked <0x00000007f48daee8> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)

        at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:98)

        at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)

        at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)

        at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)

        at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)

        at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:91)

        at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:252)

        at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:315)

        at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:214)

       

       

      Internal hornetq server threads are waiting for the page:

       

      "Thread-805 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=d42c81f5-a467-11e3-b049-d95b9e3ddf91-1424188578-484700984)" prio=10 tid=0x00002aaacfbf8800 nid=0x6ded waiting on condition [0x00002aab41492000]

         java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x0000000700ba7c80> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)

        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)

        at org.hornetq.core.paging.impl.PagingStoreImpl.page(PagingStoreImpl.java:846)

        at org.hornetq.core.persistence.impl.journal.JournalStorageManager.addToPage(JournalStorageManager.java:3783)

        at org.hornetq.core.postoffice.impl.PostOfficeImpl.processRoute(PostOfficeImpl.java:986)

        at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:691)

        at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:593)

        at org.hornetq.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:1640)

        at org.hornetq.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1323)

        at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:455)

        at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:631)

        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547)

        - locked <0x0000000738a99b30> (a java.lang.Object)

        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523)

        at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:564)

        at org.hornetq.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:160)

        at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:724)