9 Replies Latest reply on Jul 15, 2010 7:29 AM by adinn

    Why is one in 2000 transactions mysteriously timed out immediately?

    byron.hawkins

      Hi all,

       

      I have a remote client holding several instances of a stateful session bean, and it is repeatedly calling the same method on all the instances. The bean is hosted in JBoss 6M3, and I have also tried this scenario on JBoss 6M2 (using default transaction configuration in both versions). The method is managed with a CMT set to REQUIRED with timeout of 60 seconds, and it always executes in less than one second. It looks like this:

       

          @TransactionAttribute(value = TransactionAttributeType.REQUIRED)

          @TransactionTimeout(value = 60)

          public <T extends ResultType, E extends Exception> T executeCommand(Command<T, E> command)

              throws E

          {

              ...

          }

       

      Roughly one of every 2000 calls is terminated by a transaction timeout, which shows up like this in the JBoss server.log:

       

      2010-06-07 17:20:52,296 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (Thread-15) [com.arjuna.ats.arjuna.coordinator.TransactionReaper_18] - TransactionReaper::check timeout for TX 0:ffff0aa8c834:126a:4c0d8beb:583b in state  RUN

      2010-06-07 17:20:52,296 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (Thread-16) [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id 0:ffff0aa8c834:126a:4c0d8beb:583b invoked while multiple threads active within it.

      2010-06-07 17:20:52,296 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (Thread-16) [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action 0:ffff0aa8c834:126a:4c0d8beb:583b aborting with 1 threads active!

      2010-06-07 17:20:52,296 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (Thread-16) [com.arjuna.ats.arjuna.coordinator.TransactionReaper_7] - TransactionReaper::doCancellations worker Thread[Thread-16,5,jboss] successfully canceled TX 0:ffff0aa8c834:126a:4c0d8beb:583b

      2010-06-07 17:20:52,296 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] (WorkerThread#1[127.0.0.1:1947]) [com.arjuna.ats.arjuna.coordinator.BasicAction_40] - Abort called on already aborted atomic action 0:ffff0aa8c834:126a:4c0d8beb:583b


      On the client side, the caller receives the rollback exception less than 50ms after making the RMI call to the bean. Can anyone explain this?? If the bean method were throwing some unlogged exception, I would understand the rollback... but notice that the TransactionReaper is killing the transaction overtly. Something is wrong. Can anyone help me understand what's happening here??

       

      Thanks in advance for your help.

       

       

      Byron

        • 1. Re: Why is one in 2000 transactions mysteriously timed out immediately?
          adinn

          As you say, the transaction is being killed by the transaction reaper thread. Which version of JBoss are you using?

          • 2. Re: Why is one in 2000 transactions mysteriously timed out immediately?
            byron.hawkins

            The issue occurs on JBoss 6M2 and 6M3.

            • 3. Re: Why is one in 2000 transactions mysteriously timed out immediately?
              adinn

              Hmm, interesting. We have been tinkering with the Reaper in the 6Mx releases to make it run as fast as possible. Sounds like we might have made it faster than possible. We'll take a closer look.

              1 of 1 people found this helpful
              • 5. Re: Why is one in 2000 transactions mysteriously timed out immediately?
                byron.hawkins

                Awesome, that's just what I was looking for

                 

                We are actually getting ready to release our product in a few weeks, though. Would we be wise to deploy on JBoss5? Perhaps I misunderstood the website's description of the "milestone" concept... I was thinking that core components like transactions were not so much under development in these releases? Well, let me know if we should deploy on JBoss5 until 6 goes GA.

                 

                Thanks for the quick replies.

                • 6. Re: Why is one in 2000 transactions mysteriously timed out immediately?
                  adinn

                  Byron Hawkins wrote:

                   

                  Awesome, that's just what I was looking for

                   

                  We are actually getting ready to release our product in a few weeks, though. Would we be wise to deploy on JBoss5? Perhaps I misunderstood the website's description of the "milestone" concept... I was thinking that core components like transactions were not so much under development in these releases? Well, let me know if we should deploy on JBoss5 until 6 goes GA.

                   

                  Thanks for the quick replies.

                  Hmmm, product? JBoss 6 is most definitely community software at present. It is being used to try out and test new features and performance improvements and will not be converging towards a stable, supported version for a few months. So, although the components and the integrated AS are tested daily they also change daily. Features nay appear and disappear and configuration data may need to change format and content. Also, the QA process employed in order to arrive at these releases is not as thorough as for the product versions. So the current AS 6 is not something we would recommend for you to use as a product platform. Of course, your mileage may vary as may your confidence in your own ability to deal with any problems which arise or changes which occur.

                   

                  JBoss AS 5 is currently available in a supported product version which we definitely would recommend you use to underpin your product. It has been extensively QAed and will be bug-fixed and maintained in (as far as possible) a feature and API/config compatible version over a 5 year life-cycle. You can either buy support and get easy access to the EAP 5.0 release (and the EAP 5.1 release when it comes out) or you can pick and choose the correct code out of the JBoss App Server repo and build yourself a community version which corresponds to the released product. if you do the latter then you will have to diagnose problems and apply patches/upgrades yourself and generally keep your own eye on the ball.

                  1 of 1 people found this helpful
                  • 7. Re: Why is one in 2000 transactions mysteriously timed out immediately?
                    byron.hawkins

                    Thanks for the details, that helps me know what to do with our deployments. We actually have 2 products in development, one to be released early in 2011, and that one would benefit from new features of JBoss6 like the singleton EJB. Do you expect JBoss6 will GA sometime this year? If so, we will continue development of this product on the milestones and work around any issues that may appear.

                    • 8. Re: Why is one in 2000 transactions mysteriously timed out immediately?
                      hipjon

                      I'm having a similar problem with 6M3. Here's my original post:

                       

                      https://community.jboss.org/thread/154226

                       

                      I started noticing the issue since adding a call to another SLSB which uses TRANSACTION_NEW. Since my original post, I've done some more testing and have noticed some transactions fail even without making the SLSB call.

                       

                      I'll also note that I have encountered this issue, not only right after a TX is started, but sometimes many minutes into processing.

                       

                      Thanks,

                      Jon

                      • 9. Re: Why is one in 2000 transactions mysteriously timed out immediately?
                        adinn

                        Jon Kennedy wrote:

                         

                        I'm having a similar problem with 6M3. Here's my original post:

                         

                        https://community.jboss.org/thread/154226

                         

                        I started noticing the issue since adding a call to another SLSB which uses TRANSACTION_NEW. Since my original post, I've done some more testing and have noticed some transactions fail even without making the SLSB call.

                         

                        I'll also note that I have encountered this issue, not only right after a TX is started, but sometimes many minutes into processing.

                         

                        As Jonathan mentioned earlier in this thread this appears to be the problem identified in https://jira.jboss.org/browse/JBTM-686 which is fixed in JBossTS 4.10.0.

                         

                        If you look in the source tree for JBossAS 6.0.M3 the component versions for all the modules it consumes are in file component-matrix/pom.xml. You will find the following line identifying the TS version used:

                         

                        <version.jboss.jbossts>4.9.0.GA</version.jboss.jbossts>

                         

                        So, M3 still suffers from this bug. You will have to wait for 6.0.0.M4 (which will be released in a few days) in order to obrtain the fix for this bug.