1 2 Previous Next 19 Replies Latest reply on Sep 9, 2008 1:45 PM by Clebert Suconic

    XA Recovery on Journal...

    Clebert Suconic Master

      I have done some small changed on XID on Journal.

      The Journal should be agnostic about XID IMO, The StorageManager should be the correct place to encode and decode that.

      So, I have changed it a little bit. journal.prepare is now receiving an EncodingSupport, and TransactionHolder is aways holding a byte[] representing the XID. JournalStorageManager later will translate that after load is done.


      I have also added a few more assertions around Journal tests to make sure the data is handled fine on prepareTransactions.

        • 1. Re: XA Recovery on Journal...
          Clebert Suconic Master

          I have a question related to PrepareTransaction, Recoveries and NonDurableMessages

          We only store the Xid on Journal, if there are PersistentMessages. (which is aways my expecation).

          That means, if we produce messages on JBM as part of a Distributed Transaction using only NonDurableMessages those XIDs won't be recovered in case of a server crash/restart.

          Would that cause any issues on the TransactionManager?

          • 2. Re: XA Recovery on Journal...
            Tim Fox Master

             

            "clebert.suconic@jboss.com" wrote:
            I have done some small changed on XID on Journal.

            The Journal should be agnostic about XID IMO, The StorageManager should be the correct place to encode and decode that.


            Yes, indeed. The journal is designed to be a generic journal which just deals with byte[]. The idea here is it could be used by other projects with journalling needs in the future. So, certainly it should not be polluted with domain specific abstractions.


            So, I have changed it a little bit. journal.prepare is now receiving an EncodingSupport, and TransactionHolder is aways holding a byte[] representing the XID. JournalStorageManager later will translate that after load is done.


            Sounds good.


            • 3. Re: XA Recovery on Journal...
              Tim Fox Master

               

              "clebert.suconic@jboss.com" wrote:
              I have a question related to PrepareTransaction, Recoveries and NonDurableMessages

              We only store the Xid on Journal, if there are PersistentMessages. (which is aways my expecation).

              That means, if we produce messages on JBM as part of a Distributed Transaction using only NonDurableMessages those XIDs won't be recovered in case of a server crash/restart.

              Would that cause any issues on the TransactionManager?


              This is related to https://jira.jboss.org/jira/browse/JBMESSAGING-1282

              • 4. Re: XA Recovery on Journal...
                Tim Fox Master

                Also, BasicXARecoveryTest is using netty and actual disk storage.

                All these kinds of tests should use InVM transport.

                We also need to test case where recovery occurs without the server being restarted.

                • 5. Re: XA Recovery on Journal...
                  Andy Taylor Master

                   

                  So, I have changed it a little bit. journal.prepare is now receiving an EncodingSupport, and TransactionHolder is aways holding a byte[] representing the XID. JournalStorageManager later will translate that after load is done.


                  cheers Clebert.

                  All these kinds of tests should use InVM transport.


                  Is InVM working? I kept getting a BufferOverflowException!

                  java.nio.BufferOverflowException
                   at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:165)
                   at org.jboss.messaging.core.remoting.impl.ByteBufferWrapper.putBytes(ByteBufferWrapper.java:322)
                   at org.jboss.messaging.core.message.impl.MessageImpl.encode(MessageImpl.java:137)
                   at org.jboss.messaging.core.remoting.impl.wireformat.SendMessage.encodeBody(SendMessage.java:101)
                   at org.jboss.messaging.core.remoting.impl.wireformat.PacketImpl.encode(PacketImpl.java:152)
                   at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.doWrite(RemotingConnectionImpl.java:435)
                   at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.access$900(RemotingConnectionImpl.java:161)
                   at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.send(RemotingConnectionImpl.java:825)
                   at org.jboss.messaging.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:261)
                   at org.jboss.messaging.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:129)
                   at org.jboss.messaging.tests.integration.xa.BasicXaRecoveryTest.testBasicSendWithCommit(BasicXaRecoveryTest.java:124)


                  • 6. Re: XA Recovery on Journal...
                    Andy Taylor Master

                     

                    We also need to test case where recovery occurs without the server being restarted.


                    I still have some more tests to write which will include this.

                    • 7. Re: XA Recovery on Journal...
                      Clebert Suconic Master

                       

                      "timfox" wrote:

                      This is related to https://jira.jboss.org/jira/browse/JBMESSAGING-1282


                      What I thought that should be happening, is TransactionImpl aways storing the prepare on disk. That way an empty transaction would be recovered instead of "killing" the whole branch on the TM.


                      public void prepare() throws Exception
                       {
                       if (state != State.ACTIVE)
                       {
                       throw new IllegalStateException("Transaction is in invalid state " + state);
                       }
                      
                       if (xid == null)
                       {
                       throw new IllegalStateException("Cannot prepare non XA transaction");
                       }
                      
                       pageMessages();
                      
                      // if (containsPersistent)
                      // {
                       storageManager.prepare(id, xid);
                      // Without the if (contains Persistent)
                      // }
                      
                       containsPersistent = true; // just to make sure commit will also use the storageManager.
                      
                       state = State.PREPARED;
                       }
                      



                      I'm not sure if the journal could work with empty transactions, but that should be easily adaptable.

                      • 8. Re: XA Recovery on Journal...
                        Tim Fox Master

                         

                        "clebert.suconic@jboss.com" wrote:
                        "timfox" wrote:

                        This is related to https://jira.jboss.org/jira/browse/JBMESSAGING-1282


                        What I thought that should be happening, is TransactionImpl aways storing the prepare on disk. That way an empty transaction would be recovered instead of "killing" the whole branch on the TM.



                        My understanding, is that if the transaction branch is empty, then prepare should return return XAResource.XA_RDONLY

                        (See ServerSessionImpl.XAPrepare)

                        And then the transaction manager knows not to ever call commit or rollback for that branch.

                        I however recommend clarify these finer points of XA with a member of the transactions team.

                        • 9. Re: XA Recovery on Journal...
                          Tim Fox Master

                          Ah no. We only return RD_ONLY if there the tx is completely empty, including non persistent operations, so yes I think we need to record a blank tx branch in the journal.

                          • 10. Re: XA Recovery on Journal...
                            Clebert Suconic Master

                             

                            "timfox" wrote:
                            Ah no. We only return RD_ONLY if there the tx is completely empty, including non persistent operations, so yes I think we need to record a blank tx branch in the journal.


                            Besides.. if we choose to store a Prepare Record for an Empty Transaction, we would need the commit as we need the CompletionRecord of the Prepare as we need that in order to reclaim files.

                            So, if you add a PrepareRecord, you need a CommitRecord or you would break reclaiming.

                            • 11. Re: XA Recovery on Journal...
                              Adrian Brock Master

                               

                              "timfox" wrote:
                              Ah no. We only return RD_ONLY if there the tx is completely empty, including non persistent operations, so yes I think we need to record a blank tx branch in the journal.


                              Returning "read only" would be the most optimal thing to do.
                              You then don't record the prepare in the journal.

                              However, once you do that the non-peristent message should be removed from memory
                              or anywhere else you reference it.

                              If you return "read only" you won't get the commit/rollback notification.

                              That obviously means non-persistent messages won't get redelivered
                              once they've tried to be prepared in a 2PC transaction, even if the eventual
                              outcome of the transaction is a rollback.

                              That is perfectly spec compliant though since there's no guarantee
                              the message would get delivered in the first place, let alone redelivered after a rollback. :-)

                              • 12. Re: XA Recovery on Journal...
                                Tim Fox Master

                                 

                                "adrian@jboss.org" wrote:

                                Returning "read only" would be the most optimal thing to do.
                                You then don't record the prepare in the journal.

                                However, once you do that the non-peristent message should be removed from memory
                                or anywhere else you reference it.


                                The problem with returning RD_ONLY as I see it, is that if I have a tx branch with a bunch of non persistent sends, and return RD_ONLY then commit will never get called, so those messages won't ever get routed in memory to its subscribers (since routing occurs on commit)

                                One other possibility is to return OK, but not actually persist the branch if it only contains non persistent operations, then if we subsequently get a commit or rollback for that branch on recovery, we can return OK to the commit/rollback even though we can't find the branch. The tx mgr then thinks the transaction was resolved ok, even though the non persistent messages were lost (which is fine anyway).



                                • 13. Re: XA Recovery on Journal...
                                  Clebert Suconic Master

                                   

                                  "Tim Fox" wrote:
                                  ..we can return OK to the commit/rollback even though we can't find the branch. The tx mgr then thinks the transaction was resolved ok, even though the non persistent messages were lost (which is fine anyway).



                                  If there is no other implications, that's the most optimal solution, as we won't need to record anything on the journal for NonPersistentMessages on that case.

                                  • 14. Re: XA Recovery on Journal...
                                    Adrian Brock Master

                                     

                                    "timfox" wrote:

                                    One other possibility is to return OK, but not actually persist the branch if it only contains non persistent operations, then if we subsequently get a commit or rollback for that branch on recovery, we can return OK to the commit/rollback even though we can't find the branch. The tx mgr then thinks the transaction was resolved ok, even though the non persistent messages were lost (which is fine anyway).


                                    The important part with returning OK is that you need to persist the XID
                                    so you can return it from XAResource.recover() - otherwise you might
                                    confuse the transaction manager.

                                    It doesn't matter whether there is actually anything behind the XID.

                                    http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4154677#4154677

                                    I guess the real problem occurs when you have a non-persistent send()
                                    in a failed transaction commit, i.e.. another branch failed to commit, not you.

                                    The TM then calls later recover() and you return nothing.
                                    The TM would be well within its rights to assume you already committed
                                    (although it could be confused that you lost memory of the XID you said was OK :-)

                                    1 2 Previous Next