11 Replies Latest reply on Jul 17, 2013 3:03 AM by ataylor

    HornetQ XA two phase commit performance is much worse than one phase commit

    dengyong

      Environment: HornetQ 2.2.5


      My scenario is: Message driven bean -> JMS JCA target connector (outbound)

      • The MDB poll events from JMS queue
      • The JMS target connector publishes events to JMS topic (no subscriber). In the case, the JMS connector use XA JMS session
      • The messages in JMS queue and JMS topic are none persistent

      Observation:

      1. If it only involves one JMS target connector. JBoss TM will use one phase commit protocol to commit the transaction. The XAResource.commit cost less than 1ms
      2. If it involves two JMS target connector. JBoss TM will use two phase commit to commit the transaction. The XAResource.prepare and commit is very expensive and will each cost more than 30ms

       

      Analysis:

      After investigate the codes of org.hornetq.core.transaction.impl.TransactionImpl, I found the difference is that in one phase commit case, there will be no disk IO operation.

       

      Question:

           Is this by design or can we improve it?

       

      Prepare API:

               beforePrepare();

       

               storageManager.prepare(id, xid);

       

               state = State.PREPARED;

      Commit API:

               beforeCommit();

       

               if (containsPersistent || xid != null && state == State.PREPARED)

               {

       

                  if (waitBeforeCommit)

                  {

                     // we will wait all the pending operations to finish before we can add this

      asyncAppendCommit();

                  }

                  else

                  {

      storageManager.commit(id);

                  }

       

                  state = State.COMMITTED;

               }

        • 1. Re: HornetQ XA two phase commit performance is much worse than one phase commit
          ataylor

          If there are no persistent messages and we are one phase then there is no need to persist anything, if there is a crash at this point its ok if non persistent messages are lost, if the TM calls commit again then we can just return no xid and the TM will be happy with this. With 2 phase however the transaction manager needs to know on failure the state of the transaction and so we must persist the transaction and its state after prepare and subsequently add the commit to the journal afterwards.

           

          If I were sending non persistant messages then I would probably wouldnt use XA anyway.

          1 of 1 people found this helpful
          • 2. Re: HornetQ XA two phase commit performance is much worse than one phase commit
            clebert.suconic

            We have to guarantee a sync before a prepare or commit. That will always depend on the disk capacity. So, it is per design as it requries a TX on disk.

             

            If we bypass this we will break TX.

             

            What you can do to improve perf on TX is process multiple messages. You would have to do something outside of MDBs though.

            • 3. Re: HornetQ XA two phase commit performance is much worse than one phase commit
              dengyong
              1. I increased the batch. In one phase case, the performance can easily reach 10K. But in two phase case, however I increase the batch size, the performance can only retain 1~2K. The performance is far from good enough.
              2. To improve performance, I suggest to bypass TX save on disk in prepare and commit API for none persistence message case. It can improve performance and won’t break anything. Because the messages are none persistent and we need not to guarantee its durability. Even we store the TX id on disk, during the transaction recovery, we have nothing to recover.

               

              Clebert Suconic wrote:

               

              We have to guarantee a sync before a prepare or commit. That will always depend on the disk capacity. So, it is per design as it requries a TX on disk.

               

              If we bypass this we will break TX.

               

              What you can do to improve perf on TX is process multiple messages. You would have to do something outside of MDBs though.

              • 4. Re: HornetQ XA two phase commit performance is much worse than one phase commit
                dengyong

                Andy:

                 

                We save the TX to disk in two phase case. This is only for transaction recovery case, right? Like HornetQ crashes after prepare phase, after HornetQ restarts, TM may want to recover the transaction. But for none persistent message case, the message is not durable and we have nothing to recover. Why don’t we bypass the TX save on disk in this case?

                 

                 

                Andy Taylor wrote:

                 

                If there are no persistent messages and we are one phase then there is no need to persist anything, if there is a crash at this point its ok if non persistent messages are lost, if the TM calls commit again then we can just return no xid and the TM will be happy with this. With 2 phase however the transaction manager needs to know on failure the state of the transaction and so we must persist the transaction and its state after prepare and subsequently add the commit to the journal afterwards.

                 

                If I were sending non persistant messages then I would probably wouldnt use XA anyway.

                • 5. Re: HornetQ XA two phase commit performance is much worse than one phase commit
                  ataylor

                  We save the TX to disk in two phase case. This is only for transaction recovery case, right? Like HornetQ crashes after prepare phase, after HornetQ restarts, TM may want to recover the transaction. But for none persistent message case, the message is not durable and we have nothing to recover. Why don’t we bypass the TX save on disk in this case?

                  because the transaction manager needs to know if the tx was prepared or not, there is nothing written to the journal until prepare has been called, so if we did what you say the tm would call commit after failover and get a tx not found which could cause a heuristic.

                  • 6. Re: HornetQ XA two phase commit performance is much worse than one phase commit
                    dengyong

                    Andy:

                     

                    Yeah, I got your point. But I don't agree that if we did not save TX on disk, it will cause heuristic result.


                    In failover (transaction recovery) case, TM will call RM XAResource.recover to query prepared or heuristically completed state transaction IDs first. If we did not save TX on disk, then the recover API won’t return the TX. Thus TM won’t commit this TX in this case and in hence won’t cause any heuristic result.

                     

                    I studied JBoss TM and this is its behavior.

                    Andy Taylor wrote:

                     

                     

                    We save the TX to disk in two phase case. This is only for transaction recovery case, right? Like HornetQ crashes after prepare phase, after HornetQ restarts, TM may want to recover the transaction. But for none persistent message case, the message is not durable and we have nothing to recover. Why don’t we bypass the TX save on disk in this case?

                     

                    because the transaction manager needs to know if the tx was prepared or not, there is nothing written to the journal until prepare has been called, so if we did what you say the tm would call commit after failover and get a tx not found which could cause a heuristic.

                    • 7. Re: HornetQ XA two phase commit performance is much worse than one phase commit
                      ataylor

                      Yeah, I got your point. But I still think we can optimize in this case. Below is my reason:

                      In failover (transaction recovery) case, TM will first call RM XAResource.recover to query prepared or heuristically completed state transaction IDs. If we did not save TX on disk, then the recover API won’t return the TX. Thus TM won’t commit this TX in this case and in hence won’t cause any heuristic result.

                      If the TM calls prepare and then a failover ocurs it will then try to recover, it will have persisted 2 branches for the transaction, one for HornetQ and one for lets say a database. It will then call recover on all available xa resources, it will commit(or rollback) the database branch first but then wont be able to find the other branch as the tx xid does not exist anywhere. From this point on the TM will log warnngs as it can't find the branch to commit(rollback), this means that both parties in the tx have had different outcomes, this is a Heuristic.

                       

                      I will say again, if you are using non persistent messages you are'nt bothered about missing messages so don't use XA, use a local tx.

                      • 8. Re: HornetQ XA two phase commit performance is much worse than one phase commit
                        dengyong

                        Andy:

                         

                        Yeah, I think of the local TX solutions. But there is a case that user may publish events to two or more topics. Then it will involve two HornetQ JMS sessions. If they are local, the transaction consistency can not be guarantee.

                        Andy Taylor wrote:

                         

                        I will say again, if you are using non persistent messages you are'nt bothered about missing messages so don't use XA, use a local tx.

                        • 9. Re: HornetQ XA two phase commit performance is much worse than one phase commit
                          jbertram

                          If you want an atomic commit between more than 1 resource manager then you'll have to use XA which will involve persisting information to disk.

                           

                          Furthermore, even if HornetQ didn't persist anything to disk the transaction manager itself persists data to disk on prepare and commit as well.  This is the price you must pay for XA semantics.

                          • 10. Re: HornetQ XA two phase commit performance is much worse than one phase commit
                            jbertram
                            1. To improve performance, I suggest to bypass TX save on disk in prepare and commit API for none persistence message case. It can improve performance and won’t break anything. Because the messages are none persistent and we need not to guarantee its durability. Even we store the TX id on disk, during the transaction recovery, we have nothing to recover.

                            I think you are neglecting to consider situations in which HornetQ is not the only resource manager involved in the XA transaction.  HornetQ cannot know what other resource managers (if any) are involved in the XA transaction.  To ensure proper XA semantics it must assume it isn't the only resource manager involved which means it must persist data about the transaction in every case otherwise a heuristic could result (as Andy already explained).

                            • 11. Re: HornetQ XA two phase commit performance is much worse than one phase commit
                              ataylor

                              Yeah, I think of the local TX solutions. But there is a case that user may publish events to two or more topics. Then it will involve two HornetQ JMS sessions. If they are local, the transaction consistency can not be guarantee.

                              well this is a moot point since the messages are non persistent anyway

                              1 of 1 people found this helpful