12 Replies Latest reply on Jun 30, 2010 6:11 AM by timfox

    Using Selectors for JMS Consumers - Selector seems sticky

    aengineer

      Can someone verify if my conclusion that selectors are not working as expected is correct. From what I am observing, once a selector is used to consume messages from a JMS destination, any subsequent consumer will also use the same selection criteria.

       

      That is if I were to do the following:
      - create a consumer with some selector,
      - consume some messages that satisfy the selector
      - close the consumer
      - open a new consumer with no selector
      - attempt to consume a message with this new consumer. .... In this case the first message that I receive is not the first one in the queue, but its the first one that still satisfies my selector. If there are no messages that satisfy the selector, then the first message from the Q is returned.

       

      To recreate, you can use the attached java client which will publish 6 messages to /queue/ExampleQueue and will create 2 consumers one after another. The first consumer has a selector, the second consumer does not.

       

      The 6 published messages have the following message body, and property values

       

      Msg#Body TextJMS Properties
      11{PROP1=VALUE1}
      22{PROP1=VALUE1}
      33{PROP2=VALUE2}
      44{PROP2=VALUE2}
      55{PROP1=VALUE1}
      66{PROP1=VALUE1,PROP2=VALUE2}

       

      The first consumer has a selector (PROP2 = 'VALUE2') and is expected to return msg#3 with body text "3". The second consumer has no selector and is expected to return msg#1 with body text "1". Instead it returns msg#4.

       

      Thanks
      Aspi Engineer
      Putnam Investments

        • 1. Re: Using Selectors for JMS Consumers - Selector seems sticky
          clebert.suconic

          I have imported your test to our testsuite here:

           

          http://anonsvn.jboss.org/repos/hornetq/trunk/tests/jms-tests/src/org/hornetq/jms/tests/selector/SelectorTest.java

           

           

          And this is what I found:

           

           

          When you created the consumer, the consumer is doing a read-ahead for performance reasons. When you closed the consumer without consuming a few messages, those message were placed back to the top of the queue, as it usually happens with Consumers.

           

           

          If you disable read-ahead (by setting consumerWindowSize to 0 on the ConnectionFactory), it will behave as you expect.

           

           

          I've actually placed lots of assertions on the test, and it is receiving messages as expected.

          • 2. Re: Using Selectors for JMS Consumers - Selector seems sticky
            aengineer

            I would like to disagree.

             

            I do not want to set consumer-window-size to 0 since this will give me poor performance.

             

            I would think that absolutely nothing should ever reorder the messages on the Q. If I open a consumer with a selector, consume just 1 message and then close the consumer, followed by opening a second consumer with NO selector, one would expect to consume the remaining messages in published order. From what you are telling me, consumer#1 retrieved back N messages from the server - all of which that satisfied the selector - and then when the consumer was closed all these messages went to the front of the Q. This is not how one would expect a FIFO queue.

             

            Thanks

            Aspi Engineer

            Putnam Investments

            • 3. Re: Using Selectors for JMS Consumers - Selector seems sticky
              clebert.suconic

              This is the same with any other consumer.

               

              The message goes to the client buffer with the delivering state.  Pending messages will come back to the top of the queue when you closed the consumer.

               

              Say, if you do this:

               

               

              consumer = ... createConsumer(queue1);

              consumer2 = ... createConsumer(queue1);

               

               

              and then you did consumer2.close(); and consumer.close();

               

              The messages that were at the latest consumer will be placed back to the queue.

               

               

              That's the pure principle of read-ahead. if you don't want this behaviour, I suspect you need to consumerWindowSize=0.

               

              I will ask Tim Fox to comment on this.

              • 4. Re: Using Selectors for JMS Consumers - Selector seems sticky
                aengineer

                In your post, you have referenced queue1 and queue2. I only have 1 queue that I am working with.

                 

                The behavior that I see is that using a selector on a queue essentially (and within limits) reorders the messages on the queue. This is not expected behavior. A messaging system must guarantee message delivery in published order. You can consume selected messages by using selectors, but the use of a selector should not permanently reorder unconsumed messages.

                 

                I am sure the JMS spec does not address this detail at all.

                 

                Thanks

                Aspi

                • 5. Re: Using Selectors for JMS Consumers - Selector seems sticky
                  clebert.suconic

                  I have post-edited the post. I was referring to the same queue.

                   

                  The message leaves the queue originally at the right order.

                   

                  JMS Spec doesn't cover anything about ordering after rollback, redeliveries or on this case read-ahead.

                   

                  Even if you set consumerWindowSize=0, you would still have really good numbers on the performance. Maybe the performance of an individual consumer would drop a little, but the whole system would still scale pretty well.

                   

                  There's no way to guarantee ordering during redelivery if you closed a consumer with pending messages.

                   

                  If you don't have control of your client code, I suggest you work with windowSize = 0.

                   

                   

                  I have asked Tim Fox (the project lead) to comment on this. He's in Europe so I'm not sure if he will be able to answer it today (Tuesday).

                  • 6. Re: Using Selectors for JMS Consumers - Selector seems sticky
                    clebert.suconic

                    I just want to comment something on your statement:

                     

                    "..but the use of a selector should not permanently reorder unconsumed messages..."

                     

                     

                    it's not the use of the selector that is changing the order. Is the use of prefetching and not consuming all the existent messages on the client buffer. The messages are always delivered to the client at the correct order.

                     

                    The consumer may be closed and the messages will be returned top the queue.

                     

                     

                    As I said I expect Tim to comment on this issue.

                    • 7. Re: Using Selectors for JMS Consumers - Selector seems sticky
                      timfox

                      This is a trade-off between performance and strict correctness.

                       

                      To get very strict ordering behaviour in the presence of cancellation consumer-window-size can get be set to zero, but of course this would affect performance.

                       

                      It would be very difficult to provide this very strict level of ordering with buffering without seriously affecting performance.

                      • 8. Re: Using Selectors for JMS Consumers - Selector seems sticky
                        aengineer

                        I think that I am going to disagree with the notion that if you want read-ahead or consumer side pre-fetch then this is the behavior to be expected.

                         

                        This is not how other messaging systems behave. I have tested the same client with Sun Message Queue (4.4U1 with the consumer flow limit set to 25 messages) and with Web Logic (10.3.3.0 with the 'Prefetch Mode for Synchronous Consumer' set to 'Enabled' and 'Maximum Messages per Session' set to 10) and for both messaging platforms, the test case works as I would have expected.

                         

                        To some extent this could be a non-issue if performance with 'consumer-window-size=0' is almost the same as performance with a reasonable non-zero value. However if performance with a 0 window size is relatively poorer, then this issue is significant.

                         

                        I will test performance and update this ticket based on my findings. But it just seems wrong for any messaging system to reorder messages.

                         

                        Thanks

                        Aspi Engineer

                        • 9. Re: Using Selectors for JMS Consumers - Selector seems sticky
                          clebert.suconic

                          I don't mean to really put emphasis on how other systems are implemented.. it's their choice of design how to do it.. and each system may have different decisions and still comply with JMS specifications (since JMS doesn't specify how cancellations are done).

                           

                          I just tried with a competitor system here (that I'm not sure I should disclose which one I tried), and it behaved exactly the same as HornetQ in regard to read-ahead and cancellations.

                           


                          All I can say is: Using multiple consumers in the presence of read-ahead would be a anti-pattern with HornetQ. The solution here would be one of those:

                           

                          - Don't use multiple consumers on a single queue. Use a topic with durable subscription and Selectors maybe. (Which seems actually a better design decision as I could see this far).

                          - Set consumerWindowSize = 0.  (I still prefer the former decision point though).

                           

                          I guarantee you that HornetQ will perform way faster than WebLogic even with consumerWindowSize = 0.

                          • 10. Re: Using Selectors for JMS Consumers - Selector seems sticky
                            clebert.suconic

                            "All I can say is: Using multiple consumers in the presence of read-ahead would be a anti-pattern with HornetQ."

                             

                            Please.. don't use this phrase outside of the context of our conversation here. (saying that to any other user who looks at this thread later)

                            This is only the case if you require strict ordering.

                            • 11. Re: Using Selectors for JMS Consumers - Selector seems sticky
                              timfox

                              Clebert Suconic wrote:


                              All I can say is: Using multiple consumers in the presence of read-ahead would be a anti-pattern with HornetQ. The solution here would be one of those:

                               


                              This is not true, and you are confusing the issue.,

                               

                              The issue here is not related to multiple consumers on queues. If you have multiple consumers on the same queue, then *JMS ordering guarantees do not apply*. The JMS spec is quite explicit about this:

                               

                              Section 4.4.9:

                               

                              "For PTP, JMS does not specify the semantics of concurrent QueueReceivers for
                              the same Queue; however, JMS does not prohibit a provider from supporting
                              this. Therefore, message delivery to multiple QueueReceivers will depend on the
                              JMS provider’s implementation. Applications that depend on delivery to
                              multiple QueueReceivers are not portable"

                               

                              The real issue here is not about multiple consumers on a queue, it is about single consumer *with a filter* on a queue closing with buffered messages.

                               

                              If the filter does not select all messages and closes, then any buffered messages will be put back on the head of the queue, changing the order.

                               

                              This is indeed contrary to the JMS spec, however, this is an edge case affecting a minority of users. The spec compliant behaviour can be obtained by setting consumerWindowSize to zero.

                               

                              As I mentioned before this is trade-off between performance and compliance. Most users will take the performance, the few that want the compliance will take the performance hit.

                               

                              Please also note that both Weblogic and OpenMQ are some of the slowest messaging systems out there.

                              • 12. Re: Using Selectors for JMS Consumers - Selector seems sticky
                                timfox

                                Clebert Suconic wrote:


                                - Don't use multiple consumers on a single queue. Use a topic with durable subscription and Selectors maybe. (Which seems actually a better design decision as I could see this far).


                                +1 on this. Selectors on durable subscriptions will avoid this issue and probably give you better performance too.