1 2 Previous Next 15 Replies Latest reply on Mar 4, 2012 3:27 AM by Puneet Bansal

    HornetQ 2.2.5.final :ThreadPool issue on load

    kapil nikhra Newbie

      I have the following environment

       

      HornetQ->2.2.5.Final

      java 1.6

      java application running in jboss 6.0

      Windows 7

       

      We are  doing some performance and load testing and it seems when we increase the load at around 100 concurrent messages we start seeing slow performance on the queue write operation ( there are max. 10 clients reading the message from the queue in parallel).

       

      We also see a whole bunch of following warning in the hornetq  logs when we see slow performance.

       

      "WARNING [org.hornetq.core.server.impl.QueueImpl]  Couldn't finish waiting executors. Try increasing the thread pool size"

       

      originated from

      org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)

      and

      org.hornetq.core.persistence.impl.journal.OperationContextImpl$1.run(OperationContextImpl.java:239)

       

       

      I am attaching the hornetQ logs file.

       

      We Increased the thread pool in the hornetq-configuration.xml to the following value.

       

         <thread-pool-max-size>1000</thread-pool-max-size>             ( even tried with -1 )

         <scheduled-thread-pool-max-size>200</scheduled-thread-pool-max-size>

       

      But the issue was not resolved.

       

      Please let us know, if there is another configuration for the thread pool.

        • 1. Re: HornetQ 2.2.5.final :ThreadPool issue on load
          Clebert Suconic Master

          Are you sure you have successfully upgrade HornetQ to 2.2.5? I only saw that on paging and 2.2.2.

           

           

          Can you attach a thread dump at the time this is happening?

          • 2. Re: HornetQ 2.2.5.final :ThreadPool issue on load
            kapil nikhra Newbie

            Thanks Clebert for looking into this.

             

            I am attaching the thread dump from the hornetQ process ( this is not a complete dump, as this is the limit of what i could copyfrom the windows console).

             

            I am also attaching the hornetQ logs, so that you can verify the version.

             

            Please let me know, if you need additional information.

            • 3. Re: HornetQ 2.2.5.final :ThreadPool issue on load
              Clebert Suconic Master

              The dump is free. I don't see any activity.. just empty threads..

               

               

              Are you sure the issue was happening at the time you extracted the dump?

               

               

              Also, redirect it to a console, and you may get the whole thing

              • 4. Re: HornetQ 2.2.5.final :ThreadPool issue on load
                kapil nikhra Newbie

                I am not sure what you are looking for in the threads, but i took one more dump ( this time the thread-pool-max-size and scheduled-thread-pool-max-size was left as default). Since this is happening  randamloy, i tried to take at the time i saw the warning in the logs ( you can see the warnings in this log and than the dump)

                • 5. Re: HornetQ 2.2.5.final :ThreadPool issue on load
                  kapil nikhra Newbie

                  I took one more dump just before i saw the warning. See if this help, it shows some blocked threads.

                  • 6. Re: HornetQ 2.2.5.final :ThreadPool issue on load
                    Clebert Suconic Master

                    What's your use case?

                     

                    It seems you are over-using LastValueQueues. Maybe we could improve / optimize and remove a synchronized LastValueQueue.addTail

                    • 7. Re: HornetQ 2.2.5.final :ThreadPool issue on load
                      Clebert Suconic Master

                      I suspect you configured it as LVQ by mistake, did you?

                      • 9. Re: HornetQ 2.2.5.final :ThreadPool issue on load
                        kapil nikhra Newbie

                        Yes, the queue was setup as LVQ and after changing that i do not see the warning messages. Will do some further testing to confirm this.

                         

                        But looks good for now.

                        • 10. Re: HornetQ 2.2.5.final :ThreadPool issue on load
                          Clebert Suconic Master

                          Ok, the use case for LVQ is like receiving a stock ticket where the last value is replaced.

                           

                           

                          I will keep the JIRA open as an improvement, but I believe you were using it wrong.

                          • 11. Re: HornetQ 2.2.5.final :ThreadPool issue on load
                            kapil nikhra Newbie

                            Thanks Clebert.

                             

                            ya we did not intent to use the LVQ setting and were unsing it unknowlighly. We have tested after changing the settign and have not seen the Warning message since than.

                            • 12. Re: HornetQ 2.2.5.final :ThreadPool issue on load
                              acarbs12 Newbie

                              So how did you change this I believe that we may in advertantly have this set up as well! We are getting this Exception.

                              • 13. Re: HornetQ 2.2.5.final :ThreadPool issue on load
                                Clebert Suconic Master

                                @acarbs12: Kapil had a wrong setting on the queue... setting last-value-queue without setting the last-value-attribute. That was the issue.

                                • 14. Re: HornetQ 2.2.5.final :ThreadPool issue on load
                                  Puneet Bansal Newbie

                                  Hi Clebert,

                                   

                                  Our server was running fine for last 3 months and suddenly we have also started seeing this same issue. We dont have any LVQ (verfied that there is no last-value-queue).

                                  Stack trace:

                                  [Old I/O server worker (parentId: 292826622, [id: 0x11742dfe, jms3.abcd.com/10.28.7.20:5445])] 08:16:14,851 WARNING [org.hornetq.core.server.impl.QueueImpl]  Couldn't finish waiting executors. Try increasing the thread pool size

                                  java.lang.Exception: trace

                                            at org.hornetq.core.server.impl.QueueImpl.blockOnExecutorFuture(QueueImpl.java:471)

                                            at org.hornetq.core.server.impl.QueueImpl.addTail(QueueImpl.java:383)

                                            at org.hornetq.core.postoffice.impl.PostOfficeImpl.addReferences(PostOfficeImpl.java:1140)

                                            at org.hornetq.core.postoffice.impl.PostOfficeImpl.access$200(PostOfficeImpl.java:77)

                                            at org.hornetq.core.postoffice.impl.PostOfficeImpl$1.done(PostOfficeImpl.java:990)

                                            at org.hornetq.core.persistence.impl.nullpm.NullStorageManager.afterCompleteOperations(NullStorageManager.java:385)

                                            at org.hornetq.core.postoffice.impl.PostOfficeImpl.processRoute(PostOfficeImpl.java:979)

                                            at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:634)

                                            at org.hornetq.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:561)

                                            at org.hornetq.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:1365)

                                            at org.hornetq.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1107)

                                            at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:440)

                                            at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:474)

                                            at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:496)

                                            at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:457)

                                            at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:459)

                                            at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:73)

                                            at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)

                                            at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:362)

                                            at org.jboss.netty.channel.StaticChannelPipeline$StaticChannelHandlerContext.sendUpstream(StaticChannelPipeline.java:514)

                                            at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:287)

                                            at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169)

                                            at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134)

                                            at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)

                                            at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:362)

                                            at org.jboss.netty.channel.StaticChannelPipeline.sendUpstream(StaticChannelPipeline.java:357)

                                            at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)

                                            at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)

                                            at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:90)

                                            at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)

                                            at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)

                                            at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:181)

                                            at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                                            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                                            at java.lang.Thread.run(Thread.java:619)

                                  1 2 Previous Next