'hornetq-buffer-timeout' thread: huge CPU time. Is it normal ?

[icemaker]

Hello,

We are using HornetQ (HornetQ Server version 2.2.13.Final) with Jboss AS 7.1.1.Final.

After about 4 hours our server runs extremely slow.

When dumping all threads infos we noticed that each time we have the problem the thread named 'hornetq-buffer-timeout' has a huge CPU time. In this example 190 seconds :

09:58:09,793 INFO  [com.real.aof.web.controller.MonitoringController] (http--0_0_0_0_0_0_0_0-8443-56) CPU Time of 152 : 190882ms (User time: 121462ms / System time: 69420ms)
09:58:09,793 INFO  [com.real.aof.web.controller.MonitoringController] (http--0_0_0_0_0_0_0_0-8443-56) "hornetq-buffer-timeout" Id=152 WAITING on java.util.concurrent.Semaphore$NonfairSync@666fd386
  at sun.misc.Unsafe.park(Native Method)
  -  waiting on java.util.concurrent.Semaphore$NonfairSync@666fd386
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
  at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
  at org.hornetq.core.journal.impl.TimedBuffer$CheckTimer.run(TimedBuffer.java:438)
  at java.lang.Thread.run(Thread.java:744)

What is exactly this thread doing ?

Could it be normal to have such a big CPU time for this thread ?

[jbertram]

IO for the HornetQ journal is done through this thread so it should be used often, but I can't say with certainty what the ratio of CPU time vs. server run time should be.

In any case, the version of HornetQ you're using is fairly old so I would recommend upgrading whenever possible.  org.hornetq.core.journal.impl.TimedBuffer in particular received an optimization since 2.2.13.Final (see 2f949ed940d226a6afe031686253023dccbc4155 on master).

Can you describe exactly what you mean by "After about 4 hours our server runs extremely slow"?

[icemaker]

Thanks for your answer

Well all processes are suddenly running extremely slow and we don't really know why. CPU usage seems ok, Memory usage is also ok.

So we suspected a blocking thread somewhere but could not find one.

I'll try to update HornetQ.

Do I just need to update HornetQ jars in Jboss AS 7.1.1 modules or is there something more to do ?

[jbertram]

You can try updating the HornetQ jars in JBoss AS 7.1.1, but I'm not 100% sure that will work.  You may hit an incompatibility somewhere.  You might look at WildFly 8.0.0.Final or 8.1.0.CR1 or perhaps JBoss EAP 6.x which is built using later versions of AS 7.1.x.