producers blocked

[nycmatrix78]

Hi Tim, we are using hornetq and ran into this problem. Any idea why the producers are blocked?  You mentioned something about producer flow control? Thanks!

The thread dumps shows that the producers are blocked. See lots of these in the thread dump.

"http-8080-Processor53199" daemon prio=10 tid=0x00000000504b7000 nid=0x72f7 waiting on condition [0x000000005d8f9000..0x000000005d8fad10]

   java.lang.Thread.State: WAITING (parking)

        at sun.misc.Unsafe.park(Native Method)

        - parking to wait for  <0x00002aab5e345830> (a java.util.concurrent.Semaphore$NonfairSync)

        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)

        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)

        at java.util.concurrent.Semaphore.acquire(Semaphore.java:441)

        at org.hornetq.core.client.impl.ClientProducerCreditsImpl.acquireCredits(ClientProducerCreditsImpl.java:69)

        at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:287)

        at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:139)

        at org.hornetq.jms.client.HornetQMessageProducer.doSend(HornetQMessageProducer.java:455)

        at org.hornetq.jms.client.HornetQMessageProducer.send(HornetQMessageProducer.java:219)

        at org.springframework.jms.connection.CachedMessageProducer.send(CachedMessageProducer.java:121)

[timfox]

HornetQ, like most messaging systems, has a concept of producer flow control.

The idea behind producer flow control is it slows down or stops producing sending more messages when memory resource limits on the server have been exceeded, to avoid OOMing the server.

There is a chapter on this in the user manual:

http://hornetq.sourceforge.net/docs/hornetq-2.0.0.GA/user-manual/en/html/flow-control.html#d0e3632

In 19.2.1.3 there is a typo, it should read:

<address-settings>
                  <address-setting match="jms.queue.exampleQueue">            
                     <max-size-bytes>100000</max-size-bytes>
                     <address-full-policy>BLOCK</address-full-policy>   
                  </address-setting>
               </address-settings>

[nycmatrix78]

This is our setting in hornetq-configuration.xml

<address-settings>

        <!--default for catch all-->

                <address-setting match="#">

                        <redelivery-delay>0</redelivery-delay>

                        <max-size-bytes>104857600</max-size-bytes>

                        <page-size-bytes>10485760</page-size-bytes>

                        <address-full-policy>BLOCK</address-full-policy>

                </address-setting>

        </address-settings>

This is our setting in hornetq-jms.xml

<connection-factory name="ConnectionFactory">

                <connectors>

                        <connector-ref connector-name="netty-connector" />

                </connectors>

                <entries>

                        <entry name="ConnectionFactory" />

                </entries>

                <!-- 10mb -->

                <consumer-window-size>10000000</consumer-window-size>

                <producer-window-size>10000000</producer-window-size>

                <block-on-durable-send>false</block-on-durable-send>

        </connection-factory>

which limit do you think we are hitting? The "max-size-bytes" or the "producer-window-size"? We are monitoring the messages in the queue and it doesn't show that the messages are queueing up.

[nycmatrix78]

Also we are using Spring's CacheConnectionFactory and therefore caching the Producers, would this cause a problem?

[timfox]

The producer-window-size is not a limit. It's the amount of credits each producer allocates in a single go.

The limit is max-size-bytes, which is what you're hitting.

In 2.0.GA each producer will allocate producer-window-size at startup.

You've probably created a lot of producers, most of which are sitting around doing nothing.

BTW, this behaviour has changed in TRUNK

[timfox]

I notice that you have max-size-bytes set to 100 MiB and have increased producer window size to about 10MiB

This means that after creating 10 producers, further ones won't be able to send messages.

[timfox]

(Another reason I hate this whole Spring inspired caching approach!)

[timfox]

Like I say, this won't occur in TRUNK, since we have a more permissive producer flow control policy now.

[nycmatrix78]

Hi Tim,

We bumped max-size-bytes to 200mb and lowered producer-window-size and consumer-window-size to 1mb. We were able to run for 10 hours or so with this setting, putting 100 messages a second in the queue, the messages being 20-50 bytes each. But we eventually hit a state where the producers are blocked again. Any suggestions on how to diagnose this? Do our settings look right for the amount of traffic we are doing?

I appreciate the response.

[timfox]

Like I said in my previous comments - this looks like expected behaviour to me. Your hitting the limit you configured and you're caching producers.

This behaviour isn't in TRUNK anyway - have you tried that?

[nycmatrix78]

Thanks Tim. We found out that there were too many producers being started. So far it's working nicely. We'll also check out the trunk.