8 Replies Latest reply on Jun 6, 2013 11:37 AM by veitg

    JBoss 7.1.3 final and MDB processing after restart

    veitg

      Hi.

       

      I'm using a build of the JBoss 7.1.3 final tag and encountered a strange behavior concerning MDB message processing. I have queues and MDBs defined within an EAR and using the default settings in standard-full.xml regarding mdb pool size of 20.

      I put 1000 messages from an external client via EJB facade to the message queue and I can see message processing with 15 threads in parallel (due to the default maxSession I guess). But here comes the strange part. When I shutdown the

      server in the middle of the message processing, and restart it, it starts processing the remaining messages in the queue only with 1 thread and 1 message at a time - not with 15 as before.

      Does anybody have a clue why is that?

       

      Thanks.

        • 1. Re: JBoss 7.1.3 final and MDB processing after restart
          nickarls

          With no inside info and without looking at the code but perhaps it's be design (fresh incoming tasks are processed faster while a background thread works on old jobs)?

          • 2. Re: JBoss 7.1.3 final and MDB processing after restart
            veitg

            Doesn't sound like a queue then .

             

            Implementation is quite simple: MDB with MessageListener interface onMessage() implementation. Simply printing out that it is working, sleeping 10 seconds and returning successfully. With existing messages in the queue, after restart only one-by-one is processed - without putting additional messages into the queue.

             

            {code}

            @MessageDriven(activationConfig = { @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),

                    @ActivationConfigProperty(propertyName = "destination", propertyValue = "queue/request") })

            public class EventProcessor implements MessageListener {

            ...

            {code}

             

             

            Just tried when an external clients puts additional messages into the queue, while the MDB is processing one-by-one: it starts processing multiple messages again! But indeed, it looks like only as long as these messages are processed. After that, it falls back to one-by-one. I hope it is not by design . I'll take a look if these new messages are really processed first or if the new messages simply increased the thread pool size for a while...

             

            UPDATE:

             

            Ok, it seems like you described. New messages are processed first, multi-threaded. Old messages only with one thread. Looks like a stack to me (LIFO). If it is really by-design, is there any chance of changing this behavior to FIFO?

            • 3. Re: JBoss 7.1.3 final and MDB processing after restart
              jbertram

              My guess is that it is due to buffering by the MDBs.  Try setting this:

               

                @ActivationConfigProperty(propertyName = "consumerWindowSize", propertyValue = "0")

               

              And see if that makes any difference.

              • 4. Re: JBoss 7.1.3 final and MDB processing after restart
                veitg

                Thanks! That indeed did the trick !

                 

                But now I'm wondering, what this setting has to do with multi-threaded processing. I couldn't really found much regarding consumerWindowSize in the hornetq docs:

                 

                 

                {quote}

                the window size (in bytes) for consumer flow control

                {quote}

                 

                AFAIK this setting has to do with slow consumers as described here:

                 

                 

                {quote}

                Slow consumers takes significant time to process each message and it is desirable to prevent buffering messages on the client side so that they can be delivered to another consumer instead.

                Consider a situation where a queue has 2 consumers; 1 of which is very slow. Messages are delivered in a round robin fashion to both consumers, the fast consumer processes all of its messages very quickly until its buffer is empty. At this point there are still messages awaiting to be processed in the buffer of the slow consumer thus preventing them being processed by the fast consumer. The fast consumer is therefore sitting idle when it could be processing the other messages.

                To allow slow consumers, set the consumer-window-size to 0 (for no buffer at all). This will prevent the slow consumer from buffering any messages on the client side. Messages will remain on the server side ready to be consumed by other consumers.

                Setting this to 0 can give deterministic distribution between multiple consumers on a queue.

                {quote}

                 

                In the example above, why can't a new message simply be given to consumer B, which is idle and skipping busy A? Is round-robin that strict?

                I could imagine putting new messages to the consumer buffers as long as they aren't filled. If filled, skip to the next consumer. If all consumer buffers are filled, block until a consumer buffer gets free and give him the next message.

                 

                I can't see the "advantage" of waiting for A until it has finished processing - since all consumers consume "the same" messages on the same queue and processing them in the same way. Well, at least in my case (MDB, no selectors) .

                 

                BTW: if using the property above, also ordering is kept. Brand new messages are processed last as expected (FIFO).

                 

                Maybe you could shed some light on this too, please .

                 

                Thanks!

                • 5. Re: JBoss 7.1.3 final and MDB processing after restart
                  jbertram

                  I couldn't really found much regarding consumerWindowSize in the hornetq docs...

                  See http://docs.jboss.org/hornetq/2.3.0.Final/docs/user-manual/html/flow-control.html#flow-control.consumer.window.  There's a whole section which disucsses flow control (including consumer-window-size).

                   

                   

                  In the example above, why can't a new message simply be given to consumer B, which is idle and skipping busy A? Is round-robin that strict?

                  The server has no idea if the consumer is busy or not.  Furthermore, the consumer is the one requesting the messages.  As I understand it, the server simply doesn't have the information or the power to do what you're asking.

                   

                   

                  Maybe you could shed some light on this too, please .

                  What specifically?

                  • 6. Re: JBoss 7.1.3 final and MDB processing after restart
                    veitg

                    The server has no idea if the consumer is busy or not.  Furthermore, the consumer is the one requesting the messages.  As I understand it, the server simply doesn't have the information or the power to do what you're asking.

                    So, if the server doesn't know if consumer A is busy, why doesn't get consumer B (which is idle) the next message? In terms of JBoss and EJBs: why does a slow MDB block another MDB from processing new messages?

                    It seems that I'm missing some insight here .

                     

                    Let's imagine MDB A has a consumerWindowSize of 1024. With a fictitious message size of 512 this MDB (more specific: the hornetq "client") will get/fetch 2 messages from the queue (which has a hundred more messages) and stores them in the local buffer.

                    The MDB A gets the first message from the buffer via onMessage() and takes 10 seconds to process the message. That means only 1 remaining message in the local buffer of MDB A. Meanwhile MDB B stays idle and doesn't get ANY message at all (why?).

                    What happens next? Does the hornetq "client" for MDB A fetches the next message from the queue to fill the local buffer again? Or does MDB A simply gets the remaining message from its local buffer for processing and the hornetq "client" fetches 2 new messages to fill

                    the local buffer for MDB A again? If it is round-robin, when does MDB B gets its message? If it is not the server, who decides whether/when MDB B gets a message from the queue?

                     

                    I have to point out, that I'm simply using MDBs via MessageListener and don't get in contact with hornetq specific classes/consumers.

                     

                    Sorry for my possible misunderstanding - I just want to know why things are, the way they are .

                    • 7. Re: JBoss 7.1.3 final and MDB processing after restart
                      jbertram

                      So, if the server doesn't know if consumer A is busy, why doesn't get consumer B (which is idle) the next message?

                      Presumably because consumer A has pulled all of the messages in the queue into its client-side buffer which prevents any other consumer from receiving them.  My previous point was simply to show that the server has no intelligence to distribute messages between clients based on how fast or slow those clients consume messages.  This is something you have to tune for yourself - slow consumers should buffer fewer messages than fast consumers (or none at all).

                       

                       

                      Let's imagine MDB A has a consumerWindowSize of 1024. With a fictitious message size of 512 this MDB (more specific: the hornetq "client") will get/fetch 2 messages from the queue (which has a hundred more messages) and stores them in the local buffer.

                      The MDB A gets the first message from the buffer via onMessage() and takes 10 seconds to process the message. That means only 1 remaining message in the local buffer of MDB A. Meanwhile MDB B stays idle and doesn't get ANY message at all (why?).

                      Actually my understanding is that MDB B should be able to get other messages from the queue while MDB A processes it's messages.  However, MDB B should not receive either of the messages which MDB A has buffered.  If you're seeing behavior different from that you should work up a test-case without MDBs and open a thread on the HornetQ User Forum.

                      • 8. Re: JBoss 7.1.3 final and MDB processing after restart
                        veitg

                        Presumably because consumer A has pulled all of the messages in the queue into its client-side buffer which prevents any other consumer from receiving them.  My previous point was simply to show that the server has no intelligence to distribute messages

                        I re-tried the test-case with your tips in mind, and indeed that was the problem. As the default consumerWindowsSize is 1 MB and our messages seem smaller than I expected, MDB A had all (1000) messages in the local buffer so the 2nd one didn't get a single message at all . Then, when I put additional messages to the queue, they were directly processed by MDB B since the queue was "empty" from the server's point of view and MDB A was still busy with the old messages.

                         

                        So, everything works as expected. Of course .

                         

                        Thanks for your support!