0 Replies Latest reply on May 16, 2016 4:50 AM by venom_27

    Application loads CPU IO on ChannelImpl.sendBlocking

    venom_27
    venom_27 May 16, 2016 4:50 AM

      We've noticed CPU IO utilization grows suddenly on several nodes in cluster (~20 nodes, Wildfly 8.2.0, Amazon, Ubuntu 14). It can happen at any node and any time. IO growth from 0 to 10-20%.

      Dumps show that thread with max IO has next stacktrace:

      Thread 2741: (state = BLOCKED)

      - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)

      - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame)

      - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(long, java.util.concurrent.TimeUnit) @bci=106, line=2176 (Compiled fram

      e)

      - org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(org.hornetq.core.protocol.core.Packet, byte) @bci=320, line=364 (Compiled frame)

      - org.hornetq.core.client.impl.ClientSessionImpl.commit(javax.transaction.xa.Xid, boolean) @bci=83, line=1462 (Compiled frame)

      - org.hornetq.ra.HornetQRAXAResource.commit(javax.transaction.xa.Xid, boolean) @bci=53, line=145 (Compiled frame)

      - org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.commit(javax.transaction.xa.Xid, boolean) @bci=19, line=107 (Compiled frame)

      - com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelCommit() @bci=121, line=461 (Compiled frame)

      - com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(boolean, com.arjuna.ats.arjuna.coordinator.AbstractRecord) @bci=66, line=2810 (Compiled frame)

      - com.arjuna.ats.arjuna.coordinator.BasicAction.doCommit(com.arjuna.ats.arjuna.coordinator.RecordList, boolean) @bci=23, line=2726 (Compiled frame)

      - com.arjuna.ats.arjuna.coordinator.BasicAction.phase2Commit(boolean) @bci=120, line=1820 (Compiled frame)

      - com.arjuna.ats.arjuna.coordinator.BasicAction.End(boolean) @bci=316, line=1504 (Compiled frame)

      - com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(boolean) @bci=43, line=96 (Compiled frame)

      - com.arjuna.ats.arjuna.AtomicAction.commit(boolean) @bci=2, line=162 (Compiled frame)

      - com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate() @bci=123, line=1166 (Compiled frame)

      - com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit() @bci=60, line=126 (Compiled frame)

      - com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit() @bci=4, line=75 (Compiled frame)

      - org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.afterDelivery() @bci=35, line=71 (Compiled frame)

      - sun.reflect.GeneratedMethodAccessor186.invoke(java.lang.Object, java.lang.Object[]) @bci=36 (Compiled frame)

      - sun.reflect.DelegatingMethodAccessorImpl.invoke(java.lang.Object, java.lang.Object[]) @bci=6, line=43 (Compiled frame)

      - java.lang.reflect.Method.invoke(java.lang.Object, java.lang.Object[]) @bci=57, line=606 (Compiled frame)

      - org.jboss.as.ejb3.inflow.AbstractInvocationHandler.handle(java.lang.reflect.Method, java.lang.Object[]) @bci=3, line=60 (Compiled frame)

      - org.jboss.as.ejb3.inflow.MessageEndpointInvocationHandler.doInvoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=36, line=135 (Compiled frame)

      - org.jboss.as.ejb3.inflow.AbstractInvocationHandler.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=24, line=73 (Compiled frame)

      - org.app.someapp.communication.receiver.IncomingQueueMDB$$$endpoint2.afterDelivery() @bci=24 (Compiled frame)

      - org.hornetq.ra.inflow.HornetQMessageHandler.onMessage(org.hornetq.api.core.client.ClientMessage) @bci=155, line=330 (Compiled frame)

      - org.hornetq.core.client.impl.ClientConsumerImpl.callOnMessage() @bci=157, line=1116 (Compiled frame)

       

       

      - org.hornetq.core.client.impl.ClientConsumerImpl.access$500(org.hornetq.core.client.impl.ClientConsumerImpl) @bci=1, line=56 (Compiled frame)

      - org.hornetq.core.client.impl.ClientConsumerImpl$Runner.run() @bci=4, line=1251 (Compiled frame)

      - org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run() @bci=69, line=104 (Compiled frame)

      - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1145 (Compiled frame)

      - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)

      - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame)

       

      It happen on ClientSessionImpl.commit or ClientSessionImpl.prepare. BTW, hortnetQ has default config (standalone-full-ha.xml)

      What would it be? Any ideas? Let me know if any additional info need.