HornetQ 2.2.5.final :ThreadPool issue on load

[kapilnikhra]

I have the following environment

HornetQ->2.2.5.Final

java 1.6

java application running in jboss 6.0

Windows 7

We are  doing some performance and load testing and it seems when we increase the load at around 100 concurrent messages we start seeing slow performance on the queue write operation ( there are max. 10 clients reading the message from the queue in parallel).

We also see a whole bunch of following warning in the hornetq  logs when we see slow performance.

"WARNING [org.hornetq.core.server.impl.QueueImpl]  Couldn't finish waiting executors. Try increasing the thread pool size"

originated from

org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)

and

org.hornetq.core.persistence.impl.journal.OperationContextImpl$1.run(OperationContextImpl.java:239)

I am attaching the hornetQ logs file.

We Increased the thread pool in the hornetq-configuration.xml to the following value.

   <thread-pool-max-size>1000</thread-pool-max-size>             ( even tried with -1 )

   <scheduled-thread-pool-max-size>200</scheduled-thread-pool-max-size>

But the issue was not resolved.

Please let us know, if there is another configuration for the thread pool.

[clebert.suconic]

Are you sure you have successfully upgrade HornetQ to 2.2.5? I only saw that on paging and 2.2.2.

Can you attach a thread dump at the time this is happening?

[kapilnikhra]

Thanks Clebert for looking into this.

I am attaching the thread dump from the hornetQ process ( this is not a complete dump, as this is the limit of what i could copyfrom the windows console).

I am also attaching the hornetQ logs, so that you can verify the version.

Please let me know, if you need additional information.

[clebert.suconic]

The dump is free. I don't see any activity.. just empty threads..

Are you sure the issue was happening at the time you extracted the dump?

Also, redirect it to a console, and you may get the whole thing

[kapilnikhra]

I am not sure what you are looking for in the threads, but i took one more dump ( this time the thread-pool-max-size and scheduled-thread-pool-max-size was left as default). Since this is happening  randamloy, i tried to take at the time i saw the warning in the logs ( you can see the warnings in this log and than the dump)

[kapilnikhra]

I took one more dump just before i saw the warning. See if this help, it shows some blocked threads.

[clebert.suconic]

What's your use case?

It seems you are over-using LastValueQueues. Maybe we could improve / optimize and remove a synchronized LastValueQueue.addTail

[clebert.suconic]

I suspect you configured it as LVQ by mistake, did you?

[clebert.suconic]

https://issues.jboss.org/browse/HORNETQ-739

I'm waiting some description for your use case.

[kapilnikhra]

Yes, the queue was setup as LVQ and after changing that i do not see the warning messages. Will do some further testing to confirm this.

But looks good for now.

[clebert.suconic]

Ok, the use case for LVQ is like receiving a stock ticket where the last value is replaced.

I will keep the JIRA open as an improvement, but I believe you were using it wrong.

[kapilnikhra]

Thanks Clebert.

ya we did not intent to use the LVQ setting and were unsing it unknowlighly. We have tested after changing the settign and have not seen the Warning message since than.

[acarbs12]

So how did you change this I believe that we may in advertantly have this set up as well! We are getting this Exception.

[clebert.suconic]

@acarbs12: Kapil had a wrong setting on the queue... setting last-value-queue without setting the last-value-attribute. That was the issue.

[puneet.bansal]

Hi Clebert,

Our server was running fine for last 3 months and suddenly we have also started seeing this same issue. We dont have any LVQ (verfied that there is no last-value-queue).

Stack trace:

[Old I/O server worker (parentId: 292826622, [id: 0x11742dfe, jms3.abcd.com/10.28.7.20:5445])] 08:16:14,851 WARNING [org.hornetq.core.server.impl.QueueImpl]  Couldn't finish waiting executors. Try increasing the thread pool size

java.lang.Exception: trace

          at org.hornetq.core.server.impl.QueueImpl.blockOnExecutorFuture(QueueImpl.java:471)

          at org.hornetq.core.server.impl.QueueImpl.addTail(QueueImpl.java:383)

          at org.hornetq.core.postoffice.impl.PostOfficeImpl.addReferences(PostOfficeImpl.java:1140)

          at org.hornetq.core.postoffice.impl.PostOfficeImpl.access$200(PostOfficeImpl.java:77)

          at org.hornetq.core.postoffice.impl.PostOfficeImpl$1.done(PostOfficeImpl.java:990)

          at org.hornetq.core.persistence.impl.nullpm.NullStorageManager.afterCompleteOperations(NullStorageManager.java:385)

          at org.hornetq.core.postoffice.impl.PostOfficeImpl.processRoute(PostOfficeImpl.java:979)

          at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:634)

          at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:561)

          at org.hornetq.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:1365)

          at org.hornetq.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1107)

          at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:440)

          at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:474)

          at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:496)

          at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:457)

          at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:459)

          at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:73)

          at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)

          at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:362)

          at org.jboss.netty.channel.StaticChannelPipeline$StaticChannelHandlerContext.sendUpstream(StaticChannelPipeline.java:514)

          at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:287)

          at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169)

          at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134)

          at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)

          at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:362)

          at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:357)

          at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)

          at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)

          at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:90)

          at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

          at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)

          at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:181)

          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

          at java.lang.Thread.run(Thread.java:619)