9 Replies Latest reply on Dec 18, 2004 8:09 PM by ovidiu.feodorov

    UIL2 message race condition before 3.2.6RC2

    thoennes

      Hi all,

      we are currently using JBoss 3.2.3 in production and have an issue where two messages arrive in a different order.

      The message are sent by a Singleton MDB in the server to a permanent queue. The client is a Java process having two JMS sessions to the server: one for receiving and one for sending messages.

      In very rare, nearly unreproducible case it happens, that two messages arrive in a different order. We just notice that both message arrive in the same millisecond, but could not reproduce this behaviour consistently.
      Therefore, we have no detailed TRACE log output for this issue.

      Looking at the JBoss site, I found two issues:

      http://sourceforge.net/tracker/?group_id=22866&atid=376685&func=detail&aid=885053
      ([ 885053 ] UIL2 thead leak during broken connection attempt)

      and a note in the document "Changes between 3.2.6RC2 and 3.2.6RC1"
      (http://sourceforge.net/docman/display_doc.php?docid=24862&group_id=22866)

      # Simulate the single thread per session to avoid concurrent delivery to a message listener, as required by the spec.

      * messaging/src/main/org/jboss/mq/SpyMessageConsumer.java
      * messaging/src/main/org/jboss/mq/SpySession.java


      Looking at the related code:

      http://cvs.sourceforge.net/viewcvs.py/jboss/jbossmq/src/main/org/jboss/mq/SpyMessageConsumer.java?r1=1.20.2.7&r2=1.20.2.8&only_with_tag=Branch_3_2

      I found a delivery lock around the onMessage() call.

      So my question is: Is it possible in UIL2 that several thread try to deliver a message via onMessage() at the same time causing our race condition?

      If this is true, this JBoss 3.2.6 change may fix our issue.

      Looking forward to your answer

      Cheers, Jörg


        • 1. Re: UIL2 message race condition before 3.2.6RC2

          Why do people insist on reporting bugs with old versions?
          The only answer you will get from me is "Suck it and see".

          I have no interest in debugging 3.2.3 if I already fixed it in 3.2.6.
          If you still have a problem with the latest production release (jboss-3.2.6),
          you wouldn't be wasting our time with already solved problems.

          It is usually trivial to reproduce race conditions by adding Thread.sleep()
          in a suitable location (like your message listener)
          or coordinating when certain blocks of code run on different threads
          using a semaphore or other locking mechanism.

          Of course, the latter will just cause a deadlock if there is no race condition :-)

          • 2. Re: UIL2 message race condition before 3.2.6RC2
            thoennes

            Dear Adrian,

            Adrian wrote:
            Why do people insist on reporting bugs with old versions?


            I did not report a bug, but just asked whether a JBoss patch contained in release 3.2.6 would fix our issue:

            Jörg wrote:
            Is it possible in UIL2 that several threads try to deliver a message via onMessage() at the same time causing our race condition?


            Since you are the JBoss architect, you probably know instantly whether the UIL2 uses one or several threads which forward messages to a message listeners onMessage() method.

            If the answer is "yes", this means for us that we have to upgrade to 3.2.6 to fix it.

            My knowledge of JBossMQ is still limited, even though we have purchased the documentation and attended the "Advanced Training" seminar. Unfortunately, JMS wasn't explained at the seminar. On the other hand, the wiki is now a fairly good source of information.

            Adrian wrote:
            The only answer you will get from me is "Suck it and see".


            This style of answering is not very polite. Actually, I have seen it quite a couple of times in this forum. I can understand that you feel annoyed by silly questions, but IMHO you should avoid expressing these feelings in this forum. We also have customers and a live system where we provide 1st and 2nd level support, but we always stay polite. In addition, we provide commercial support for the QuickFIX open source software, and I always answer questions in the mailing list and in the FIX forums and do not insult people even if the questions sound stupid or are FAQ.

            With all due respect, please remember the "Director of Support" in your signature. Do you really think it is a good advertisement for JBoss commercial support if the Director is as polite as you?

            Adrian wrote:
            I have no interest in debugging 3.2.3 if I already fixed it in 3.2.6.


            Understood. And if our problem is fixed in 3.2.6 this is great news.

            Adrian wrote:
            If you still have a problem with the latest production release (jboss-3.2.6),
            you wouldn't be wasting our time with already solved problems.


            As you probably know, JBoss is not the only piece of software in this world. There are production dependencies to be tested. So we cannot always upgrade if a new JBoss version is released, unless the complete system has been tested thoroughly.

            All we were looking for was a simple yes or no as to whether this was a bug with 3.2.3 that has now been fixed in 3.2.6 (i.e. giving us a reason to look at upgrading).

            Adrian wrote:
            It is usually trivial to reproduce race conditions by adding Thread.sleep() in a suitable location (like your message listener)


            Would you mind explaining this in more detail? We have only one message listener receiving messages from JBoss, so how could a Thread.sleep(...) help here?

            Actually, we tried to reproduce the issue by sending messages as fast as we could, but we did not succeed.

            So the question remains:

            How many UIL2 threads try to forward message to a onMessage method?


            Looking forward to a more qualified answer

            Jörg


            • 3. Re: UIL2 message race condition before 3.2.6RC2

              For the 10,000 time.

              I am Director of Support for Customers, i.e. those people who pay me to work on their
              problems. They have paid for me to spend time on their problem no matter how trivial
              or which version of JBoss.

              If you come along expecting me to do *your work* or ask about old versions that
              are of little interest to the open source project you will be lucky if you are just ignored
              rather than being the victim of a LART (looser attitude readjustment tool).

              I am actually very interested in whether the problem still exists in 3.2.6
              But I couldn't care less if it is already fixed.

              Now go away and see if 3.2.6 solves your problem.

              The answer to your question is that there are as many threads as there are
              MessageConsumers(QueueReceivers/TopicSubscribers) with a MessageListener
              (assuming you have invoked connection.start() on the relevent connections).

              • 4. Re: UIL2 message race condition before 3.2.6RC2
                thoennes

                Dear Adrian,

                Adrian wrote:

                I am Director of Support for Customers, i.e. those people who pay me to work on their problems. They have paid for me to spend time on their problem no matter how trivial or which version of JBoss.


                Agreed. I also get paid by customers to do their work. Its completely OK for you to say "Sorry, I have no time to look at this in more detail."

                BUT: Being paid by customer does not give you the right to be rude to all the others who do not pay you.

                Adrian wrote:

                If you come along expecting me to do *your work* or ask about old versions that are of little interest to the open source project you will be lucky if you are just ignored rather than being the victim of a LART (looser attitude readjustment tool).


                Again you missed the point. I was talking about politeness, not about doing someone elses work. I really start wondering how this fits with JBoss' "Remember we love you" slogan. At least this style is not very professional.

                Adrian wrote:

                The answer to your question is that there are as many threads as there are MessageConsumers(QueueReceivers/TopicSubscribers) with a MessageListener (assuming you have invoked connection.start() on the relevent connections).


                Thanks, I will give this some more thoughts and possibly come back to the forum if I found any issue.

                Jörg


                • 5. Re: UIL2 message race condition before 3.2.6RC2

                  It is you who miss the point.

                  Asking people to look at dead code (3.2.3 is over a year old)
                  is not only a waste of their time it has an opportunity cost
                  (something that is not fixed in the lastest version doesn't get fixed).

                  On the politeness issue we will have to agree to differ.

                  These are the open source forums, you get what you pay for.
                  Actually you get more than you pay for, since you pay nothing.
                  If you want a professional service, you can always buy a support contract.

                  If you come here with time wasting questions, you can expect to be
                  told where to go.

                  Like I've said many times before, I consider it infinitely more rude
                  when people don't make even the basic effort (like test the latest version)
                  before posting questions or claiming they've found a bug.
                  Instead they expect some benevolent open source "community" to do their
                  work for them and get a shock when it doesn't quite work out that way.

                  Regards,
                  Adrian

                  • 6. Re: UIL2 message race condition before 3.2.6RC2

                    If you are interested in helping the community there is an outstanding
                    task to improve this whole area.

                    See the following link and the related forum post:
                    http://jira.jboss.com/jira/browse/JBMQ-10

                    • 7. Re: UIL2 message race condition before 3.2.6RC2
                      thoennes

                      Dear Adrian,

                      one last post about this topic, you are not expected to answer this. Perhaps it is resolved easier if we meet at a JBoss event (training etc.) some day.

                      Adrian wrote:

                      Asking people to look at dead code (3.2.3 is over a year old)
                      is not only a waste of their time it has an opportunity cost


                      I did ask you about a special patch made in 3.2.6RC2:


                      Revision 1.20.2.8 - (view) (download) (annotate) - [select for diffs]
                      Sat Aug 28 11:42:08 2004 UTC (3 months, 3 weeks ago) by ejort
                      Branch: Branch_3_2
                      CVS Tags: JBoss_3_2_6_RC2, JBoss_3_2_7_RC1, JBoss_3_2_6
                      Changes since 1.20.2.7: +9 -3 lines
                      Diff to previous 1.20.2.7 , to branch point 1.20

                      Simulate the single thread per session to avoid concurrent delivery to a message listener, as required by the spec.


                      This is not one year old.

                      Adrian wrote:

                      These are the open source forums, you get what you pay for.
                      Actually you get more than you pay for, since you pay nothing.


                      Yes, we did not pay you to answer our questions in this forum. But at least we did pay JBoss to get training and documentation. According to Marc Fleurys "Professional Open Source" this is the way it works. And it works so well that now also the (HTML) docs are free.

                      Adrian wrote:

                      If you want a professional service, you can always buy a support contract.


                      We have been considering that for some time. But your personal style is indeed not a motivating factor here.

                      Adrian wrote:

                      Like I've said many times before, I consider it infinitely more rude
                      when people don't make even the basic effort (like test the latest version)
                      before posting questions or claiming they've found a bug.


                      We discovered this problem two weeks ago, but we were able to reproduce it only once since that time. If I were able to reproduce it consistently, i.e. if I run a test for say 2 hours, the problem appears at least once, then I could modify my setup (e.g. using JBoss 3.2.6) until the bug disappears. (Your statement about inserting Thread.sleep() made me curious whether there is some other way to trigger this bug. But I have no idea at the moment.)

                      As the empirical way did not show any success, I tried the analytical one looking at the Changelogs since 3.2.3 and at the source code.

                      The comment "Simulate the single thread per session to avoid concurrent delivery to a message listener, as required by the spec." did not sound like a bug fix in the first place, but most probably is one.

                      Therefore, I just asked for some background (motivation) for this very change in one source file.

                      If you answer would have been "cannot remember why I patched this," it would not have helped me, but it would have been OK.

                      So I continue to go through the source as I did in the forum thread

                      http://www.jboss.org/index.html?module=bb&op=viewtopic&t=45759

                      In this case I made some basic effort at least. BTW, do you use Doug Leas patched version now?

                      Perhaps we meet some day...

                      Regards, Jörg

                      • 8. Re: UIL2 message race condition before 3.2.6RC2
                        thoennes

                         

                        Adrian wrote:

                        If you are interested in helping the community there is an outstanding
                        task to improve this whole area.

                        See the following link and the related forum post:
                        http://jira.jboss.com/jira/browse/JBMQ-10


                        Nice point. But I would to have at least a basic introduction into the general IL interface. Neither the "Advanced Training" nor the current docs contain a detailed description. Perhaps I miss some wiki link?

                        Cheers, Jörg


                        • 9. Re: UIL2 message race condition before 3.2.6RC2
                          ovidiu.feodorov

                          The new advanced training does, to the extent that is feasable to present in a four days training that has to cover everything in JBoss.

                          http://www.jboss.com/services/training/courses#Advanced

                          However if you intend to extend functionality, looking at the source code and a debugger are usually much more helpful than any training.