4 Replies Latest reply on Nov 26, 2014 8:57 AM by yairogen

HornetQ client producer threads hang in ClientProducerCreditsImpl.acquireCredits

manu_1185 Jan 5, 2013 9:57 AM

Hi,

We are using hornetq version 2.2.14 Final in our production environment. In the last few days, our user count has increased quite a bit and every day, our server hangs because all the producer threads are blocked. Given below is the stack trace:

"pool-5-thread-100" prio=10 tid=0x00007fc694112000 nid=0x7599 waiting on condition [0x00007fc62cb4a000]

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for <0x000000050098fb68> (a java.util.concurrent.Semaphore$NonfairSync)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:838)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)

at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)

at java.util.concurrent.Semaphore.acquire(Semaphore.java:468)

at org.hornetq.core.client.impl.ClientProducerCreditsImpl.acquireCredits(ClientProducerCreditsImpl.java:74)

at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:305)

at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:142)

at org.hornetq.jms.client.HornetQMessageProducer.doSend(HornetQMessageProducer.java:451)

at org.hornetq.jms.client.HornetQMessageProducer.send(HornetQMessageProducer.java:246)

at com.bsb.hike.pubsub.jms.JMSProducer.send(JMSProducer.java:129)

at com.bsb.hike.pubsub.jms.JMSProducer.send(JMSProducer.java:117)

at com.bsb.hike.pubsub.ProducerPool$MessageSendTask.run(ProducerPool.java:150)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

at java.lang.Thread.run(Thread.java:679)

It starts working after a restart but happens again after sometime. We are using BLOCK in address-full-policy, and max-size-bytes is 1GB for all addresses. Initially we thought that we might be sending messages at a higher rate than consuming of messages....so after sometime the address might be getting blocked. So we wrote a program to use the management api to get messageCount from queues. In out application we always have a consumer consuming, so the output of the above mentioned program showed that queues have very less (<100) or no messages at all times (even when producer threads are blocked).

We are stuck here and are looking for some suggestion. Any help will be appreciated.

1. Re: HornetQ client producer threads hang in ClientProducerCreditsImpl.acquireCredits

clebert.suconic Jan 5, 2013 7:14 PM (in response to manu_1185)

Look at previous messages.. or the manual on the paging chapter. The default configuration is block, you could either increase max-size, set to page or make sure you consume all messages.
Actions
2. Re: HornetQ client producer threads hang in ClientProducerCreditsImpl.acquireCredits

manu_1185 Jan 6, 2013 1:06 AM (in response to clebert.suconic)

In our application, we are always consuming messages....messageCount of queues retrieved from management API suggests the same (since messageCount is very less or 0 all the time). We are using AUTO_ACKNOWLEDGE for acking, so there is no chance that messages are getting piled up in queues because we are not acking them. Also max-size-bytes is very high (1GB).

I want to know if anything else can cause this issue (any other scenario or any bug).

Else our last option will be to analyze this by increasing max-size-bytes and/or switch to PAGE option.
Actions
3. Re: HornetQ client producer threads hang in ClientProducerCreditsImpl.acquireCredits

manu_1185 Jan 11, 2013 6:24 AM (in response to manu_1185)

I was able to reproduce this with somewhat larger messages. In my test, max-size-bytes was configured to be 100MB (for all addresses....though I was using only 1 queue in persistent mode with persistent messages). 10 clients were sending messages (at an interval of 5 secs between messages). Each client send a total of 100 messages (so the total number of messsages was 1000). I tried different message sizes. It ran fine for message size of 10 KB and 15 KB. But when I increased the message size to 20 KB, out of 1000 messages, only a few were received (ranging from 15-150 in different tests) . All the producer threads were blocked in acquireCredits.

Can we fix this by changing some configuration?
Actions
4. Re: HornetQ client producer threads hang in ClientProducerCreditsImpl.acquireCredits

yairogen Nov 26, 2014 8:57 AM (in response to manu_1185)

Did you ever find the solution for this? I am getting stuck on the same thing.
Actions

Go to original post