-
1. Re: HornetQ 2.2.5.final :ThreadPool issue on load
clebert.suconic Jul 13, 2011 2:32 AM (in response to kapilnikhra)Are you sure you have successfully upgrade HornetQ to 2.2.5? I only saw that on paging and 2.2.2.
Can you attach a thread dump at the time this is happening?
-
2. Re: HornetQ 2.2.5.final :ThreadPool issue on load
kapilnikhra Jul 13, 2011 4:00 PM (in response to clebert.suconic)Thanks Clebert for looking into this.
I am attaching the thread dump from the hornetQ process ( this is not a complete dump, as this is the limit of what i could copyfrom the windows console).
I am also attaching the hornetQ logs, so that you can verify the version.
Please let me know, if you need additional information.
-
HornetQ_Dump.txt.zip 19.1 KB
-
hornetq.log.zip 1.4 KB
-
-
3. Re: HornetQ 2.2.5.final :ThreadPool issue on load
clebert.suconic Jul 13, 2011 8:12 PM (in response to kapilnikhra)The dump is free. I don't see any activity.. just empty threads..
Are you sure the issue was happening at the time you extracted the dump?
Also, redirect it to a console, and you may get the whole thing
-
4. Re: HornetQ 2.2.5.final :ThreadPool issue on load
kapilnikhra Jul 13, 2011 10:00 PM (in response to clebert.suconic)I am not sure what you are looking for in the threads, but i took one more dump ( this time the thread-pool-max-size and scheduled-thread-pool-max-size was left as default). Since this is happening randamloy, i tried to take at the time i saw the warning in the logs ( you can see the warnings in this log and than the dump)
-
HornetQ_Dump_2.txt.zip 5.3 KB
-
-
5. Re: HornetQ 2.2.5.final :ThreadPool issue on load
kapilnikhra Jul 13, 2011 10:16 PM (in response to kapilnikhra)I took one more dump just before i saw the warning. See if this help, it shows some blocked threads.
-
HornetQ_Dump_3.txt.zip 6.3 KB
-
-
6. Re: HornetQ 2.2.5.final :ThreadPool issue on load
clebert.suconic Jul 14, 2011 12:20 AM (in response to kapilnikhra)What's your use case?
It seems you are over-using LastValueQueues. Maybe we could improve / optimize and remove a synchronized LastValueQueue.addTail
-
7. Re: HornetQ 2.2.5.final :ThreadPool issue on load
clebert.suconic Jul 14, 2011 12:49 AM (in response to clebert.suconic)I suspect you configured it as LVQ by mistake, did you?
-
8. Re: HornetQ 2.2.5.final :ThreadPool issue on load
clebert.suconic Jul 14, 2011 11:43 AM (in response to clebert.suconic) -
9. Re: HornetQ 2.2.5.final :ThreadPool issue on load
kapilnikhra Jul 14, 2011 12:58 PM (in response to clebert.suconic)Yes, the queue was setup as LVQ and after changing that i do not see the warning messages. Will do some further testing to confirm this.
But looks good for now.
-
10. Re: HornetQ 2.2.5.final :ThreadPool issue on load
clebert.suconic Jul 15, 2011 12:21 PM (in response to kapilnikhra)Ok, the use case for LVQ is like receiving a stock ticket where the last value is replaced.
I will keep the JIRA open as an improvement, but I believe you were using it wrong.
-
11. Re: HornetQ 2.2.5.final :ThreadPool issue on load
kapilnikhra Jul 15, 2011 1:04 PM (in response to clebert.suconic)Thanks Clebert.
ya we did not intent to use the LVQ setting and were unsing it unknowlighly. We have tested after changing the settign and have not seen the Warning message since than.
-
12. Re: HornetQ 2.2.5.final :ThreadPool issue on load
acarbs12 Dec 9, 2011 2:55 PM (in response to kapilnikhra)So how did you change this I believe that we may in advertantly have this set up as well! We are getting this Exception.
-
13. Re: HornetQ 2.2.5.final :ThreadPool issue on load
clebert.suconic Dec 13, 2011 8:26 AM (in response to acarbs12)@acarbs12: Kapil had a wrong setting on the queue... setting last-value-queue without setting the last-value-attribute. That was the issue.
-
14. Re: HornetQ 2.2.5.final :ThreadPool issue on load
puneet.bansal Mar 1, 2012 12:53 PM (in response to clebert.suconic)Hi Clebert,
Our server was running fine for last 3 months and suddenly we have also started seeing this same issue. We dont have any LVQ (verfied that there is no last-value-queue).
Stack trace:
[Old I/O server worker (parentId: 292826622, [id: 0x11742dfe, jms3.abcd.com/10.28.7.20:5445])] 08:16:14,851 WARNING [org.hornetq.core.server.impl.QueueImpl] Couldn't finish waiting executors. Try increasing the thread pool size
java.lang.Exception: trace
at org.hornetq.core.server.impl.QueueImpl.blockOnExecutorFuture(QueueImpl.java:471)
at org.hornetq.core.server.impl.QueueImpl.addTail(QueueImpl.java:383)
at org.hornetq.core.postoffice.impl.PostOfficeImpl.addReferences(PostOfficeImpl.java:1140)
at org.hornetq.core.postoffice.impl.PostOfficeImpl.access$200(PostOfficeImpl.java:77)
at org.hornetq.core.postoffice.impl.PostOfficeImpl$1.done(PostOfficeImpl.java:990)
at org.hornetq.core.persistence.impl.nullpm.NullStorageManager.afterCompleteOperations(NullStorageManager.java:385)
at org.hornetq.core.postoffice.impl.PostOfficeImpl.processRoute(PostOfficeImpl.java:979)
at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:634)
at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:561)
at org.hornetq.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:1365)
at org.hornetq.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1107)
at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:440)
at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:474)
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:496)
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:457)
at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:459)
at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:73)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:362)
at org.jboss.netty.channel.StaticChannelPipeline$StaticChannelHandlerContext.sendUpstream(StaticChannelPipeline.java:514)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:287)
at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169)
at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:362)
at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:357)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:90)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:181)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)