8 Replies Latest reply on May 17, 2013 11:26 AM by clebert.suconic

    Odd behavior with JMS QueueBrowser in 7.1.1.Final

    ftg314159

      I'm seeing some very odd and unpredictable behavior using an application client with JBoss JMS.  Searches have found similar reports by others both here and elsewhere, but either the circumstances don't match, they involve coding error I'm not making, or they're unanswered.

       

      Environment:

       

      JBoss is running unchanged as shipped and installed, with the exception of having added a user with a guest role to allow my application to access the predefined testQueue.  There is no application code running in JBoss; the application is an external POJO.

       

      Synopsis:

       

      After populating the queue with a set of messages, creating a QueueBrowser, and using getEnumeration() to access the messages works perfectly.  But if, while the QueueBrowser is open, I access the first message in the enumeration via nextElement() and then do a receive() to receive that message, the receive() works but the enumeration suddenly has no more elements.  If I do another getEnumeration(), I get an empty enumeration.  If I restart the application, I get the next message in the queue, and the same behavior (current enumeration and any new enumeration is now empty).  Moreover, if I run the application several times in quick succession, the initial executions may get no messages at all, and then suddenly messages are seen.

       

      Background (not critical, can be skipped):

       

      I'm converting an application to JMS from another in-house messaging system that, in JMS terms, allows selective acknowledgment of messages, i. e. if you receive a bunch of messages without acknowledging them, you can acknowledge older ones without implicitly acknowledging all later ones, as happens with JMS.

       

      To replicate this behavior, I decided to "deliver" messages to the application by using a QueueBrowser and getEnumeration().  As I retrieve messages from the enumeration, I maintain a LinkedHashMap which keys the message ID to a Boolean indicating whether the application has "released" the message, and I deliver the message payload to the application.  When the application "releases" a message, I mark the associated entry in the linkedHashMap as true, and then iterate through the map, doing a receive() with AUTO_ACKNOWLEDGE for each released message, removing the entry from the map,  until I hit one which has not yet been released.  The intent is to acknowledge and delete messages which the application has fully consumed, while leaving any later messages intact.  If the application releases messages out of order, the later released messages will not be received from the queue until such time as the application has released all older messages.

       

      Testing Results:

       

      In addition to the processing described above, I've tried several variations.

       

      I've tried getting a new enumeration every time I do a series of receive()s - no effect.

       

      I've tried closing and recreating the QueueBrowser after every series of receive()s - no effect.

       

      However, I have found a workaround.  If I open *two* separate connections, each with its own session, and use one for the QueueBrowser and the other to do the receive()s, it works perfectly.

       

      If I change the workaround to use two separate sessions on the *same* connection, the first nextElement() returns the first message, but the receive() for that message never returns, i. e. JMS believes the queue is empty.

       

      I'm ready to enter a JIRA with a reproducible case, but I wanted to check here first to be sure that what I'm doing is legal and expected to work in JMS.  I wish I could debug this here, but unfortunately 7.1.1.Final ships with HornetQ 2.2.13, and the source for this is no longer available on the HornetQ download site, just 2.2.14 and much older ones.  The HornetQQueueBrowser source code seems to have changed enough that the debugger claims that most places I want to stop are not breakpointable lines.

        • 1. Re: Odd behavior with JMS QueueBrowser in 7.1.1.Final
          clebert.suconic

          The issue here was because of your stop on the connection, and that was also stopping the consumer on the browser. Why are you doing that? It seems you would need a consumerWindowSize(0) on the connection otherwise you will have in deliveries messages on your consumer that won't be available for the browser.

          • 2. Re: Odd behavior with JMS QueueBrowser in 7.1.1.Final
            clebert.suconic

            I have fixed this here: HORNETQ-1195

            • 3. Re: Odd behavior with JMS QueueBrowser in 7.1.1.Final
              ftg314159

              Thanks, I've posted additional comments on the new bug report link you gave.

              • 4. Re: Odd behavior with JMS QueueBrowser in 7.1.1.Final
                clebert.suconic

                The discussion you added diverges from the bug.. I will copy your comments here and continue the disccusion here...

                 

                 

                you said:

                So you're saying that there is some sort of readahead here that is taking messages out of the view of a browser even if they haven't yet been received ?

                I didn't see anything in the JMS Javadoc about something like that.

                I have to keep this code compatible with any JMS server that follows the spec. Could you please elaborate on whether this is some aspect of JMS I missed, or whether it is an optimization issue specific to HornetQ ? Is there a way in the JMS API to do what you've suggested ?

                Also, I noticed that you tagged this for some action in a future release. Does this imply that there is something in HornetQ that needs to/will change, and that your suggestion is a workaround for now ?

                Thanks for your attention and input here.

                 

                 

                and then...

                Oh, to answer your previous question, I didn't have the start/stop in there initially. I added it because I thought that having the Connection in the started state may have been causing the problem. Originally, I only issued the start once, and never issued a stop.

                • 5. Re: Odd behavior with JMS QueueBrowser in 7.1.1.Final
                  clebert.suconic

                  There was another issue with receiveNoWait.. maybe that was another issue... if you could try the latest 2.3.x branch or master and see how it behaves? I have replicated your testcase and it fixed.

                   

                   

                  And to answer your question... this has nothing to do with JMS.. it's just common practice on message systems.

                   

                   

                  Browsers will only deliver you what's available.

                   

                  - All the message systems I know (I know quite a lot of them... about 10) will implement read ahead on the queue, meaning will flush lots of messages to the client to save on network latency.

                   

                  - So, messages will be taken out of the queue when you deliver ahead. When you called stop on the connection that cleared the buffer and sent messages back. The read ahead is expected behaviour and having the browser not delivering messages that are not available is a common and expected behaviour, allowed by the JMS spec.

                   

                   

                  So, you should really set Consumer Window Size to 0 if you need to use a browser with a Queue simultaneously. However if you need high performance delivery you should probably use a durable subscription for each consumer.

                  • 6. Re: Odd behavior with JMS QueueBrowser in 7.1.1.Final
                    clebert.suconic

                    let me complete, reiterate one thing... your code will work with the new version... .. however think about this:

                     

                    you will receive a few messages from server to client.. will receive one, then you will stop it...that will make the consumer to return the buffer to the server... so you will use a lot more resources than you would need.

                    • 7. Re: Odd behavior with JMS QueueBrowser in 7.1.1.Final
                      ftg314159

                      Thanks for your replies.

                       

                      I understand the model you've described, although I can't see the point of excluding unreceived messages from Browser processing simply because they've been moved closer to a presumed receiver.  I also don't see why the use of two Connections should provide a different experience than a single Connection.

                       

                      Just to be clear, you're saying that with message systems in general, unless I stop() the connection before refreshing the browser enumeration, I will miss messages in the Browser, but that this will cause extra overhead.  Unfortunately, I need the part of the JMS contract that requires messages to be retained until acknowledged, as I really don't want to have to harden them on the receiver's side as well, which I would have to do if I were actually receiving them.  Once I acknowledged anything, it would automatically acknowledge everything received whether the consumer was done with it or not.

                       

                      I'm guessing that the consumer window size thing is specific to HornetQ ?  I suppose I could use instanceof to check for HornetQQueuBrowser, but I'd prefer not to.

                       

                      What is your opinion of the resource consequence of using two Connections ?   My performance concerns are less with setup costs than with realtime latency in receiving/browsing messages.  And, given your knowledge of messaging systems, do you foresee any functional problem with the workaround of using two Connections considering what I'm trying to accomplish ?

                       

                      Lastly, could you elaborate on exactly what HornetQ behavior you changed and why ?  I'm still not clear, given the fact that you say it's acceptable for message systems under JMS to exclude unreceived messages from a browse, on what HornetQ behavior you thought was incorrect and changed.

                       

                      Thanks again for your timely attention to this.

                      • 8. Re: Odd behavior with JMS QueueBrowser in 7.1.1.Final
                        clebert.suconic

                        Let me put it in a different way:

                         

                        The browser will only deliver what hasn't been consumed yet. If you are fast consuming you won't ever have an issue as you would consume as fast as you can.

                         

                        Now, if you are slow consuming you will retain messages on the client buffer, as been configured on the connection factory.

                         

                        If you want to have this kind of scenario you really have a slow consumer scenario, and hence you have to configure it properly.

                         

                         

                        and You won't need two connections... the issue was already fixed.

                         

                         

                         

                         

                        I'm guessing that the consumer window size thing is specific to HornetQ ?  I suppose I could use instanceof to check for HornetQQueuBrowser, but I'd prefer not to.

                         

                         

                        Each system has its way to configure for slow consumers.

                         

                         

                        Don't quote me on that for documentation purposes.. just what I remember from memory:

                         

                        WebSphere MQ, you have I belive read ahead configuration

                        JBoss MQ, you set the connection-factory as slow consumer

                        Sun MQ, you set the queue as read ahead.

                        ActiveMQ.. I can't remember where.. but there's a setting for it as well... http://activemq.apache.org/slow-consumer-handling.html

                         

                         

                        In all these scenarios the messages wouldn't be available to a browser while on the consumer's buffer.

                         

                         

                        You can't just assume JMS as your way of learning messaging.. that's just a thin API in top of what a message system is.. There's always some configuration to be taken place at the broker of your choice.

                         

                        Deciding between fast reading or slow consumer will be an architectural decision you're making. You are currently forcing a cleanup on the buffer by setting connection.stop on your test, and that's not optimal. It will work but it will waste networking and CPU.