3 Replies Latest reply on May 6, 2011 11:36 PM by clebert.suconic

    Why is there no option to set the consumer window size to a # of messages?

    parmstrong

      If you are wanting to control the consumers on a queue wouldn't you want the option to specify a number of messages to pre-fetch and not just a size in bytes?  Am I understanding this wrong?  What if your messages are quite different in size but they take a similar amount of process it seems like you would want to prefetch a few messages to keep your network round trips lower and also help compensate for a slow consumer.  How do you translate that into bytes when your messages are different size?  It would be nice to have the option to specify a # of messages also and not just # of bytes.  Not to mention it is some what difficult to get an accurate size calculation for a message. 

        • 1. Why is there no option to set the consumer window size to a # of messages?
          clebert.suconic

          Consumer Window Size is used to full control / throttle messages towards the consumer. And we need the size of the message throttle messages on that sense.

           

          If you have the possibility of slow consume your messages and want to have other consumers to be able to receive the message right away, you should just set the consumerWindowSize=0. (What would be the equivalent if you had a slowConsumer=true option).

           

          >>What if your messages are quite different in size but they take a similar amount of process it seems like you would want to prefetch a few messages to keep your network round trips lower and also help compensate for a slow consumer.

           

           

          # of messages wouldn't help you either. Things would start to get really complicated on that sense. You should probably just set as slow consumer, or have a queue just for messages that will take a long time to process and set slow consumer while reading from that queue.

          • 2. Re: Why is there no option to set the consumer window size to a # of messages?
            parmstrong

                 I don't know how you figure.  Lets say I just want my consumer to prefetch 10 messages at a time.  That way a slow consumer will only ever be taking a long time on 10 messages and the fast consumer wouldn't be doing a network trip every time only ever 10th time. 

                 The reason this makes since for me is I am doing a last value queue but the messages that are prefetched are not considered when determining duplicate messages so I would like to keep most messages in the queue but also I would like to allow my consumers to take a few messages at a time.  So I would like to set the prefetch size to maybe 10 messages or so but one message can vary in size from maybe 500 bytes up to 1M it is hard to know what to set the size to in bytes.  So the number of messages prefetched would vary quite a bit depending on the size of the messages I could set the prefetch size to 1M and be prefetching 1 or several hundred.  Does that make sense or am I misunderstanding?  Why not have a setting that says set the prefetch size to 'X' messages or a max of  'Y' bytes?

            • 3. Re: Why is there no option to set the consumer window size to a # of messages?
              clebert.suconic

              For large messages like this, there's no point on caching them at the client...

               

              If you are sending large messages, I would recommend you to have your consumerWindowSize to something equivalent to at least one message.

               

              We don't have consumerFlowControl based on # messages. Say, you set it to 100 messages, and then you send 100 message to 2M. you blow out the memory on the client.

               

               

              Flow control is to prevent the client to blow up memory and not eat up your resources between client and server.

               

              You should either have slow consumer or not. The performance will be really good any way.