10 Replies Latest reply on Apr 17, 2010 9:44 AM by Tommy Mei

    producers blocked

    Tommy Mei Newbie
      Hi Tim, we are using hornetq and ran into this problem. Any idea why the producers are blocked?  You mentioned something about producer flow control? Thanks!

       

       

      The thread dumps shows that the producers are blocked. See lots of these in the thread dump.
      "http-8080-Processor53199" daemon prio=10 tid=0x00000000504b7000 nid=0x72f7 waiting on condition [0x000000005d8f9000..0x000000005d8fad10]
         java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00002aab5e345830> (a java.util.concurrent.Semaphore$NonfairSync)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)
              at java.util.concurrent.Semaphore.acquire(Semaphore.java:441)
              at org.hornetq.core.client.impl.ClientProducerCreditsImpl.acquireCredits(ClientProducerCreditsImpl.java:69)
              at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:287)
              at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:139)
              at org.hornetq.jms.client.HornetQMessageProducer.doSend(HornetQMessageProducer.java:455)
              at org.hornetq.jms.client.HornetQMessageProducer.send(HornetQMessageProducer.java:219)
              at org.springframework.jms.connection.CachedMessageProducer.send(CachedMessageProducer.java:121)
        • 1. Re: producers blocked
          Tim Fox Master

          HornetQ, like most messaging systems, has a concept of producer flow control.

           

          The idea behind producer flow control is it slows down or stops producing sending more messages when memory resource limits on the server have been exceeded, to avoid OOMing the server.

           

          There is a chapter on this in the user manual:

           

          http://hornetq.sourceforge.net/docs/hornetq-2.0.0.GA/user-manual/en/html/flow-control.html#d0e3632

           

          In 19.2.1.3 there is a typo, it should read:

           

          <address-settings>
                            <address-setting match="jms.queue.exampleQueue">           
                               <max-size-bytes>100000</max-size-bytes>
                               <address-full-policy>BLOCK</address-full-policy>  
                            </address-setting>
                         </address-settings>
          • 2. Re: producers blocked
            Tommy Mei Newbie

            This is our setting in hornetq-configuration.xml

             

            <address-settings>

                    <!--default for catch all-->

                            <address-setting match="#">

                                    <redelivery-delay>0</redelivery-delay>

                                    <max-size-bytes>104857600</max-size-bytes>

                                    <page-size-bytes>10485760</page-size-bytes>

                                    <address-full-policy>BLOCK</address-full-policy>

                            </address-setting>

                    </address-settings>

            This is our setting in hornetq-jms.xml

             

            <connection-factory name="ConnectionFactory">
                            <connectors>
                                    <connector-ref connector-name="netty-connector" />
                            </connectors>
                            <entries>
                                    <entry name="ConnectionFactory" />
                            </entries>
                            <!-- 10mb -->
                            <consumer-window-size>10000000</consumer-window-size>
                            <producer-window-size>10000000</producer-window-size>
                            <block-on-durable-send>false</block-on-durable-send>
                    </connection-factory>
            which limit do you think we are hitting? The "max-size-bytes" or the "producer-window-size"? We are monitoring the messages in the queue and it doesn't show that the messages are queueing up.
            • 3. Re: producers blocked
              Tommy Mei Newbie

              Also we are using Spring's CacheConnectionFactory and therefore caching the Producers, would this cause a problem?

              • 4. Re: producers blocked
                Tim Fox Master

                The producer-window-size is not a limit. It's the amount of credits each producer allocates in a single go.

                 

                The limit is max-size-bytes, which is what you're hitting.

                 

                In 2.0.GA each producer will allocate producer-window-size at startup.

                 

                You've probably created a lot of producers, most of which are sitting around doing nothing.

                 

                BTW, this behaviour has changed in TRUNK

                • 5. Re: producers blocked
                  Tim Fox Master

                  I notice that you have max-size-bytes set to 100 MiB and have increased producer window size to about 10MiB

                   

                  This means that after creating 10 producers, further ones won't be able to send messages.

                  • 6. Re: producers blocked
                    Tim Fox Master

                    (Another reason I hate this whole Spring inspired caching approach!)

                    • 7. Re: producers blocked
                      Tim Fox Master

                      Like I say, this won't occur in TRUNK, since we have a more permissive producer flow control policy now.

                      • 8. Re: producers blocked
                        Tommy Mei Newbie

                        Hi Tim,

                         

                        We bumped max-size-bytes to 200mb and lowered producer-window-size and consumer-window-size to 1mb. We were able to run for 10 hours or so with this setting, putting 100 messages a second in the queue, the messages being 20-50 bytes each. But we eventually hit a state where the producers are blocked again. Any suggestions on how to diagnose this? Do our settings look right for the amount of traffic we are doing?

                         

                        I appreciate the response.

                        • 9. Re: producers blocked
                          Tim Fox Master

                          Like I said in my previous comments - this looks like expected behaviour to me. Your hitting the limit you configured and you're caching producers.

                           

                          This behaviour isn't in TRUNK anyway - have you tried that?

                          • 10. Re: producers blocked
                            Tommy Mei Newbie

                            Thanks Tim. We found out that there were too many producers being started. So far it's working nicely. We'll also check out the trunk.