9 Replies Latest reply on Dec 4, 2007 5:06 PM by garneke

    different order between sending and receiving(FIFO)

    hasegawa.tomohiro

      I installed JBoss 4.0.3(and did not change configuration),
      and downloaded sample JMS program "jms_tutorial-1_3_1.zip"
      form http://java.sun.com/products/jms/tutorial/.

      I sent 3 messages by executing SimpleQueueSender
      and received them by executing SimpleQueueReceiver
      as following:

      1. execute 3 DOS window(Command prompt)
      2. run JBoss on DOS window 1
      3. execute "java SimpleQueueReceiver queue/testQueue"
      4. execute "java SimpleQueueSender queue/testQueue 3"

      The result is below:
      $ java SimpleQueueSender queue/testQueue 3
      Queue name is queue/testQueue
      Sending message: This is message 1
      Sending message: This is message 2
      Sending message: This is message 3

      $ java SimpleQueueReceiver queue/testQueue
      Queue name is queue/testQueue
      Reading message: This is message 3
      Reading message: This is message 1
      Reading message: This is message 2

      Why the order of receiving message is different from
      the order of sending one?

      When I executed SimpleQueueSender first
      and executed SimpleQueueReceiver second,
      the result is same order between sending and receiving.

      Is it Bug or JBoss specification?
      Please tell me the cause and the way of solving.

      BTW, this sample program work well on
      Sun Java System Application Server 2005Q2.

      My environment:
      OS: Windows XP SP2(RHEL WS 3.0, too)
      Java: J2SE 1.5.0_05
      JBoss: 4.0.3

      jndi.properties:
      java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
      java.naming.provider.url=jnp://localhost:1099
      java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces

        • 1. Re: different order between sending and receiving(FIFO)

          Nope, just a *stupid* receiver class.

          Because it only waits for 1 milli-second, you are see nacks back to the server
          because you do a receive(1) request then stop waiting.
          Which JBoss correct reports later as messages (1 and 2) being redelivered

          message.getJMSRedelivered()
          


          Once redelivery is involved, ordering is out of the window.

          Change the test to wait for a reasonable amount of time
          receive(100ms) // 0.1 seconds
          and then you won't have this receive/stop waiting/nack race condition.

          Queue name is queue/testQueue
          Reading message: SpyTextMessage {
          Header {
           jmsDestination : QUEUE.testQueue
           jmsDeliveryMode : 2
           jmsExpiration : 0
           jmsPriority : 4
           jmsMessageID : ID:43-11320851124363
           jmsTimeStamp : 1132085112436
           jmsCorrelationID: null
           jmsReplyTo : null
           jmsType : null
           jmsRedelivered : false
           jmsProperties : {}
           jmsPropReadWrite: false
           msgReadOnly : true
           producerClientId: ID:43
          }
          Body {
           text :This is message 3
          }
          }
          Reading message: SpyTextMessage {
          Header {
           jmsDestination : QUEUE.testQueue
           jmsDeliveryMode : 2
           jmsExpiration : 0
           jmsPriority : 4
           jmsMessageID : ID:43-11320851124251
           jmsTimeStamp : 1132085112425
           jmsCorrelationID: null
           jmsReplyTo : null
           jmsType : null
           jmsRedelivered : true
           jmsProperties : {JMS_JBOSS_REDELIVERY_COUNT=1}
           jmsPropReadWrite: false
           msgReadOnly : true
           producerClientId: ID:43
          }
          Body {
           text :This is message 1
          }
          }
          Reading message: SpyTextMessage {
          Header {
           jmsDestination : QUEUE.testQueue
           jmsDeliveryMode : 2
           jmsExpiration : 0
           jmsPriority : 4
           jmsMessageID : ID:43-11320851124332
           jmsTimeStamp : 1132085112433
           jmsCorrelationID: null
           jmsReplyTo : null
           jmsType : null
           jmsRedelivered : true
           jmsProperties : {JMS_JBOSS_REDELIVERY_COUNT=1}
           jmsPropReadWrite: false
           msgReadOnly : true
           producerClientId: ID:43
          }
          Body {
           text :This is message 2
          }
          }
          


          • 2. Re: different order between sending and receiving(FIFO)
            hasegawa.tomohiro

            Thank you so much for your quick reply.

            I confirmed that the value of the Property(JMSRedelivered) is set to TRUE, too.

            I sended messages again after changing timeout parameter of receive() method.
            I tried 20 times for each parameter.
            The result is below:

            receive() : the order was correct at 20/20 times.
            receive(1) : the order was wrong!(20/20 times)
            receive(100) : the order was wrong!(5/20 times)
            receive(500) : the order was wrong!(1/20 times)
            receive(600) : the order was correct at 20/20 times.
            receive(750) : the order was correct at 20/20 times.
            receive(1000) : the order was correct at 20/20 times.

            Now I have one more question.

            In terms of performance,
            Which way is the best to keep FIFO order on receiving JMS messages from Queue?
            (1)To use receive() method
            (2)To use receive(timeout) method, timeout is reasonable long time(ex. 1000)
            (3)To use the class implementing MessageListener interface

            • 3. Re: different order between sending and receiving(FIFO)

              This is a common FAQ.

              I've relented and added it as a feature request:
              http://jira.jboss.com/jira/browse/JBAS-2444

              But I'm not likely to implement it myself anytime soon.

              • 4. Re: different order between sending and receiving(FIFO)

                I certainly wouldn't recommend using AUTO_ACKNOWLEDGE if you want guarantees
                of delivery.

                In pseudo code:

                public Message MessageConsumer::receive(long timeout)
                {
                 Message message = internalReceive(timeout);
                
                 // Message is acknowledged before you even see the message
                 if (message != null && session.isAutoAcknowledge())
                 message.acknowledge();
                
                 return message;
                }
                


                • 5. Re: different order between sending and receiving(FIFO)
                  hasegawa.tomohiro

                  Thanks a lot for your reply.
                  Maybe, I will use (1) or (3) to receive messages.

                  BTW, perhaps this is FAQ, but I would like to confirm the following things:

                  "I understand that I should not use AUTO_ACKNOWLEDGE to guarantee delivery of messages in nontransacted sessions.
                  But, I can use AUTO_ACKNOWLEDGE in transacted sessions.
                  Because acknowledgement happens automatically in transacted sessions(using JMS API Local Transaction) when a transaction is committed."

                  Is it right?

                  • 6. Re: different order between sending and receiving(FIFO)

                    Acknowledgement mode is ignored in transactions.

                    • 7. Re: different order between sending and receiving(FIFO)
                      garneke

                      I am seeing basically the same issue as the original poster - however, I can find no solution.

                      -I am using multiple Solaris 8 platforms with older hardware.
                      -I have experienced the problem on both jboss v4.0.5GA and v4.2.2GA.
                      -I am using a default jboss configuration with the one exception that I have altered it to use an mysql backend.

                      In my situation I have multiple servers involved in the messages lifespan but I do not use any clustering and I have only one poster/consumer on any one Queue at any time.

                      Senerio:
                      1. Each message contains a unique (sequential) serial number in its body.
                      2. Messages are written to a JMS Queue "A" on server #1 at around 200 millisecond intervals.
                      3. Messages are read from the Queue "A" manipulated and sent thru other servers using ordinary TCP sockets. The destination server writes the resulting message back to a new JMS Queue "B" on server #5.
                      4. Final application is sitting in an infinite loop reading messages as fast as it can from Queue "B" with CLIENT_ACKNOWLEDGE set and using 'receive(2000)' - Note: 2 second timeout. Each message that is read is acknowledged then the unique serial number is parsed from the message contents and compared to previously received messages. I always receive about a half a dozen messages out of order in a 300-350 thousand message set. The unique serial number stays consistently in sync with the JMS Message ID that is stored in the message header. This implies that the messages are written to the JMS Queue "B" in the correct order. It is only when I read them back that they become out of order. Normally, an out of order message will only be off by one message. However, I have had examples where a message is actually retrieved 5 messages late ( roughly a second late ).

                      Between the unique serial number in the message body and the JMS Message ID and logging that I perform on the sending side I am confident that the messages are being written in order.

                      I have to be able to say(guarantee) that we can deliver message in a FIFO order.

                      I can not find any comments or postings suggestion that anyone has a fix for this.

                      I know current development is geared toward Jboss v5 and Jboss Messaging versus JbossMQ, but I need to find a fix for this!

                      What can I do? Do you have any suggestions?

                      • 8. Re: different order between sending and receiving(FIFO)
                        garneke

                        One last point. In addition to using a 2 second timeout on the receive call, I also pause for .25 seconds before calling receive again if I do timeout. I was hoping to avoid that "race condition" that I have read is possible ( or was possible ).

                        However, remember that messages are moving at around 200 ms intervals. I should never timeout. The only evidence that I have is that the logs on the sending side do not show any signs that writing to the Queue was delayed during any of the "out of order" periods. The logging on the receiving side is limited to just the bad messages.

                        Thanks In Advance, for your help!

                        • 9. Re: different order between sending and receiving(FIFO)
                          garneke

                          If anyone can help me I would appreciate it.

                          I have re-run my testing with some jboss server side logging enabled. I have also included some logging to let me know when the receiving Client experiences a timeout on the receive(2000) call. It times out more often that I would expect considering the fact that there is no lack of messages to read, but that could be associated with general house cleaning, etc.

                          The following is an snippet from my simple client log indicating that right after the client timed out on a receive(2000) call and subsequently paused for .25 of a second before calling receive again it read the wrong message. As you can see the JMS Msg ID: 1-1196727135200779567 was received about 6 messages late...

                          0:12:14 11/4/2007) Correct -- serial:[41553] MsgID:[1-1196727134895779565]
                          0:12:15 11/4/2007) Correct -- serial:[41554] MsgID:[1-1196727135077779566]
                          0:12:17 11/4/2007) (2) Sec TIMEOUT -- Delay .25 Secs, prev:[1-1196727135077779566]
                          0:12:20 11/4/2007) Wrong -- serial:[41556] MsgID:[1-1196727135337779568]
                          0:12:20 11/4/2007) Correct -- serial:[41557] MsgID:[1-1196727135484779569]
                          0:12:20 11/4/2007) Correct -- serial:[41558] MsgID:[1-1196727135686779570]
                          0:12:20 11/4/2007) Correct -- serial:[41559] MsgID:[1-1196727135792779571]
                          0:12:20 11/4/2007) Correct -- serial:[41560] MsgID:[1-1196727136006779572]
                          0:12:20 11/4/2007) Correct -- serial:[41561] MsgID:[1-1196727136131779573]
                          0:12:20 11/4/2007) Wrong -- serial:[41555] MsgID:[1-1196727135200779567]
                          0:12:20 11/4/2007) Wrong -- serial:[41562] MsgID:[1-1196727136262779574]
                          0:12:20 11/4/2007) Correct -- serial:[41563] MsgID:[1-1196727136444779575]
                          0:12:20 11/4/2007) Correct -- serial:[41564] MsgID:[1-1196727136657779576]
                          0:12:20 11/4/2007) Correct -- serial:[41565] MsgID:[1-1196727136778779577]
                          


                          It is important to Note: Although I had quite a few "Timeouts" I did not receive a message in the wrong order after every timeout. However, for each message that was out of order it was ALWAYS directly after a "Timeout" of the receive(2000). I have varied my subsequent delay after a receive() times out ranging from .125 - .5 seconds. Without fixing the problem! Do I need to pause in the order of multiple seconds?


                          The following is a snippet from the jboss server logs indicated the basic queue activity at the time of the error. It also shows that the message in question was delivered late. However, it shows an interesting "NACK" to its Acknowledgment. The simple client that I use is set to read with Client Acknowledgment and I only ACK successful receives. In other words, I do not have any NACK code on my side. Again, I am using a default installation of 4.2.2GA with the exception that my db is mysql. I have added this log entry as an image because it is so large...

                          [img]http://www.northstarssoccerclub.com/jboss_error.png[/img]

                          Thanks In Advance!