1 2 3 Previous Next 43 Replies Latest reply on Aug 19, 2010 11:13 AM by timfox

    Filter problem

    maximilien

      Hi,

       

      We are using queues with filters in our app and have a huge problem with it.

       

      If there is a message in the queue that doesn't match any consumer filter, hornetq server try to deliver the message indefinitelly consuming 100% cpu of a core.

       

      The loop is done in org.hornetq.core.server.impl.QueueImpl.deliver()  that call executor.execute(deliverRunner); because there is no match.

       

      I have attach a simple example that just send a message in a queue and add a consumer with a non matching filter.

       

      Thanks,

      Maximilien

        • 1. Re: Filter problem
          ataylor

          Before I spend any time looking at this could you:

           

          1. specify what versions you are using.

           

          and

           

          b. Make sure you have tested for th eproblem on trunk.

           

          cheers

          • 2. Re: Filter problem
            maximilien

            I just see there is a JIRA about this problem https://jira.jboss.org/browse/HORNETQ-322 that is mark as resolved since 2.1.0.beta1 and i still have the bug using 2.1.0.final, i will try on trunk

            • 3. Re: Filter problem
              maximilien

              Reading more precisely the JIRA it's not exactly the same problem.

              In it there is a consumer that match but doesn't have the chance to consume the message because the non matching one was in a loop.

              The fix just permits to the non matching consumer to leave the deliver function and then let the matching one to consume the message.

               

              In my case there is no matching consumer and the the server try indefinitely to deliver the message to non matching consumers and eat 100% cpu.

               

              Maximilien

              • 4. Re: Filter problem
                maximilien

                still no luck using trunk

                • 5. Re: Filter problem
                  timfox

                  Please post your example using a more well known file format, .zip, .tar.gz, .bz etc

                   

                  Rar is some windows thing and I don't have the ability to unpack it.

                  • 6. Re: Filter problem
                    maximilien

                    oops sorry, a zip version is there now ;-)

                    • 7. Re: Filter problem
                      timfox

                      Thanks, it's a very simple fix. Don't have time to do it right now, but will get it down in the next few days.

                      • 8. Re: Filter problem
                        maximilien

                        ok thanks, let me know when it is done

                        • 9. Re: Filter problem
                          timfox
                          • 10. Re: Filter problem
                            maximilien

                            ok thanks , i've just downloaded it from trunk, apply changes on our 2.1.1 version and it seems to solve the problem.

                            • 11. Re: Filter problem
                              maximilien

                              Hello,

                               

                              There is still some possible loop if there is a mix of non matching and busy consumers.

                               

                              Maybe the condition in the end of QueueImpl.deliver should be

                               

                               

                              if (messageReferences.size() > 0 && (busyCount + noMatchCount) != size)
                                    {
                                       // More messages to deliver so need to prompt another runner - note we don't
                                       // prompt another one if all consumers are busy or non matching
                              
                                       executor.execute(deliverRunner);
                                    }
                              

                               

                              Maximilien

                              • 12. Re: Filter problem
                                timfox

                                Do you have a test case?

                                • 13. Re: Filter problem
                                  maximilien

                                  The initial problem is solve but we have encountered another problem in the same function.

                                   

                                  QueueImpl.deliver() was in a loop, but this time beacause there was a message that had to be consumed by a started consumer.

                                  I don't know why hornetq server didn't success to deliver it and loop indefinitly trying to deliver it.

                                  Because it was in our production server i didn't had time to analyse the cause and just moved it to another queue then re-move it to the original queue and it solved the problem (100% CPU usage of a core).

                                   

                                  What is strange is that when the server was in a loop there was other similar messages that were successfully consumed by the consumer.

                                  I know that there isn't enought elements to analyse the problem now, but it's just to inform that there is probably a problem in this function.

                                  If the problem re-appear i will try to collect more informations.

                                   

                                  Maximilien

                                  • 14. Re: Filter problem
                                    maximilien

                                    It's not a case we encounter but here is a small example of mix consumers :

                                     

                                    one is no match

                                    one is matching but not started so busy

                                     

                                    then it loops

                                    1 2 3 Previous Next