hornetq paging blocks producers ?
Hi,
I have noticed that hornetq blocks message producers while paging. This has a significant negative impact on the performance and i wonder whether this is really necessary.
AFAIK, journaling is there to make sure no messages are lost and i can understand that a message must be written to the journal before it can be be considered 'sent'.
However, paging looks like an internal memory management trick to me and should not block producers. From a 10000 feet perspective, it looks to me that paging could be done in an asynchrous way.
To make matters worse, we are using XA transactions, meaning that jboss keeps a hold of a DB connection until hornetq paging ends
Is there a way to prevent producers from waiting on the paging ??
I'm using hornetq 2.3.5 (inside JBoss EAP 6.1.1.GA (AS 7.2.1.Final-redhat-10))
Example thread that 'blocks' until hornetq has paged (there are a few hundred threads in this state):
"http-/172.31.110.115:8280-1" daemon prio=10 tid=0x000000001aec9800 nid=0x478a waiting on condition [0x000000005818e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000073816f838> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2176)
at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:353)
- locked <0x000000073816f878> (a java.lang.Object)
at org.hornetq.core.client.impl.ClientSessionImpl.prepare(ClientSessionImpl.java:1563)
at org.hornetq.ra.HornetQRAXAResource.prepare(HornetQRAXAResource.java:130)
at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.prepare(XAResourceWrapperImpl.java:169)
at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelPrepare(XAResourceRecord.java:213)
at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2515)
at com.arjuna.ats.arjuna.coordinator.BasicAction.doPrepare(BasicAction.java:2482)
at com.arjuna.ats.arjuna.coordinator.BasicAction.prepare(BasicAction.java:2059)
- locked <0x00000007f48daee8> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)
at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1481)
- locked <0x00000007f48daee8> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)
at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:98)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:91)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:252)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.required(CMTTxInterceptor.java:315)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:214)
Internal hornetq server threads are waiting for the page:
"Thread-805 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=d42c81f5-a467-11e3-b049-d95b9e3ddf91-1424188578-484700984)" prio=10 tid=0x00002aaacfbf8800 nid=0x6ded waiting on condition [0x00002aab41492000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000700ba7c80> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
at org.hornetq.core.paging.impl.PagingStoreImpl.page(PagingStoreImpl.java:846)
at org.hornetq.core.persistence.impl.journal.JournalStorageManager.addToPage(JournalStorageManager.java:3783)
at org.hornetq.core.postoffice.impl.PostOfficeImpl.processRoute(PostOfficeImpl.java:986)
at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:691)
at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:593)
at org.hornetq.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:1640)
at org.hornetq.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1323)
at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:455)
at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:631)
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:547)
- locked <0x0000000738a99b30> (a java.lang.Object)
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:523)
at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:564)
at org.hornetq.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:160)
at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)