1 2 Previous Next 21 Replies Latest reply on Sep 6, 2006 3:43 AM by mskonda

    XARecovery - MQ/Messaging merge

    mskonda Apprentice

      Hello Adrian,

      I am merging XA Recovery (see http://jira.jboss.com/jira/browse/JBAS-1341 and http://jira.jboss.com/jira/browse/JBMESSAGING-407) from MQ to Messaging. Could you let me know the branch/tag where the source code you have worked went for MQ XA stuff?

      I am wondering if you have any test cases to test the functionality.

      I'll appreciate your input

      Thanks
      Madhu

        • 1. Re: XARecovery - MQ/Messaging merge
          Adrian Brock Master

          The tests are in the testsuite
          testsuite/src/main/org/jboss/test/jbossmq/test/Recovery*

          • 2. Re: XARecovery - MQ/Messaging merge
            mskonda Apprentice

            The recovery (almost) is in place ? however, I have a test failing. I need your input in this case. the scenario is given below:

            I am using application sever as test base (test case works fine when I use statndalon recovery manager).

            - I have a durable subscriber on a destination.
            - I have prepared a transaction which I expect to be recovered when the server is back up.

            So, when the server is back, the recovery is fine ? that is, the transaction is committed (and the row is deleted) and the message reference state changed to ?C? from ?+?.

            However, the durable subscriber doesn?t receive this message for ever!

            The only(main) difference I see before killing and after restarting up the server is the ChannelSupport?s load method being called for resetting the channel state leaving the 'Loaded' column from ?Y? (before restart) to ?N? (after restart). Do you think this has something to do with the delivery to the client?

            Let me know if I am missing any obvious.

            Thanks
            Madhu

            • 3. Re: XARecovery - MQ/Messaging merge
              Tim Fox Master

              Let me just clarify what you're doing:

              1. You have a durable subscriber on a destination.

              2. You send a messsage to the sub in a global transaction

              3. The tx is committed.

              4. Prepare completes ok.

              5. You force a server "crash".

              6. On recovery of the server, the database looks ok (it is in state C).

              7. Try to receive() message using a consumer on the subscription and you cannot receive message.

              Have I got this right?

              [BTW recover() can be called on the XAResource *after* the channel has been loaded - so you can't rely on channel.load() loading all the recovered data.]

              How you verified that the recovered data is reloaded when the channel is loaded?

              • 4. Re: XARecovery - MQ/Messaging merge
                mskonda Apprentice

                 

                "timfox" wrote:
                Let me just clarify what you're doing:

                Have I got this right?

                Yes.

                [BTW recover() can be called on the XAResource *after* the channel has been loaded - so you can't rely on channel.load() loading all the recovered data.]


                Recovery takes place once every one has settled - the messaging server, application server etc. Infact, there's a time lag before any recovery kicks in.

                How you verified that the recovered data is reloaded when the channel is loaded?

                I have checked the database to see the rows in relevant tables.

                • 5. Re: XARecovery - MQ/Messaging merge
                  Tim Fox Master

                   

                  "mskonda" wrote:

                  Recovery takes place once every one has settled - the messaging server, application server etc. Infact, there's a time lag before any recovery kicks in.


                  I'm not sure what you mean by "settled"...

                  Can you explain in more detail when this occurs?


                  "mskonda" wrote:

                  I have checked the database to see the rows in relevant tables.


                  No, I mean have you checked that the query to load the data into the channel at startup contains your recovered data?

                  • 6. Re: XARecovery - MQ/Messaging merge
                    mskonda Apprentice

                    This is the correct order Tim

                    1. I have a Bean managed MDB listening to a destination

                    2. Once a message is received, it creates a transaction and enlists three XAResource (two of them are dummy while third one is MessagingXAResource)

                    3. It does the usual JMS Stuff and finally sends a message to the destination

                    4. The commit is called. I crash the server during this phase so I can I have a prepared transaction

                    5. The server is back, the recovery kicks in, the transction is recovered, the message is published (or strictly, the state is changed from + to C)

                    6. There's a durable subscriber on this destination (where MDB has published).

                    For some reason, i am not receving this published/committed message by the durbale subscriber.

                    • 7. Re: XARecovery - MQ/Messaging merge
                      Tim Fox Master

                      Hi Madhu-

                      A few comments:

                      1. We need to make sure that any tests for the recovery functionality live in the functional test suite and don't rely on ejbs, mdbs or any other non jms stuff.

                      The tests should manipulate the XAConnnection/XASession/XAResource, enlist them in global transactions and test them this way.

                      You may need to add something to simulate a server "crash".

                      2. It's my undertstanding that the transaction manager can call XAResource.recover() at any time, not just after server startup.

                      So when it calls recover() we need to do the following sequence of actions:

                      a) Look in the database for any transactions in the prepared state. If there are any, create transaction objects corresponding to these rows, and add them to the in memory transaction repository in the prepared state (The prepared transactions may already be in the in-memory transaction repository so be careful to avoid duplicates)

                      b) Return this list to the transaction manager

                      c) Transaction manager will then call commit on those transactions.

                      • 8. Re: XARecovery - MQ/Messaging merge
                        mskonda Apprentice

                         

                        "timfox" wrote:
                        Hi Madhu-

                        A few comments:

                        1. We need to make sure that any tests for the recovery functionality live in the functional test suite and don't rely on ejbs, mdbs or any other non jms stuff.

                        The tests should manipulate the XAConnnection/XASession/XAResource, enlist them in global transactions and test them this way.

                        You may need to add something to simulate a server "crash".


                        I take your point. I do have stanadaone tests but as you said, I'll make sure they don't depend upon non-jms stuff.

                        2. It's my undertstanding that the transaction manager can call XAResource.recover() at any time, not just after server startup.


                        The JBossTS RecoveryManager calls an implementation of XAResourceRecovery which looks for the appropriate xaresource to carry on the recovery. The recovery is not called until XAResource is found. look at the following code:
                        /**
                         * This method checks whether there's an xa resource available
                         *
                         * @return
                         */
                         public boolean hasMoreResources() {
                        
                         if (working)
                         return false;
                        
                         if (xaRes == null) {
                         xaRes = initXAResource();
                         }
                        
                         // test the resource
                         try {
                        
                         xaRes.getTransactionTimeout();
                        
                         working = true;
                        
                         } catch (Exception ignored) {
                        
                         // ignore this exception
                         log.warn("Exception:" + ignored.getMessage());
                         }
                        
                         return working;
                         }
                        

                        So when it calls recover() we need to do the following sequence of actions:


                        a) Look in the database for any transactions in the prepared state. If there are any, create transaction objects corresponding to these rows, and add them to the in memory transaction repository in the prepared state (The prepared transactions may already be in the in-memory transaction repository so be careful to avoid duplicates)

                        b) Return this list to the transaction manager

                        c) Transaction manager will then call commit on those transactions.


                        Yes, this is the process I've employed. When a transaction is committed, in this case, the message is recvoered and published for the first time. So the subscriber should be able to get it.

                        However, although, the tx is committed and the message is updated with relevant state, it comes to the same point as I mentioned, the message isn't received by the subscriber!


                        • 9. Re: XARecovery - MQ/Messaging merge
                          Tim Fox Master

                          It's hard to tell what you're doing without seeing your code.

                          Just to clarify:

                          The JBoss TS Recovery manager lives on the client side. How does it get hold of the list of XAResources to recover?

                          Does it use ServerConnectionEndpoint::getPreparedTransactions()?

                          When you are creating the Transaction instances from the database and adding to the transation repositiry are you sure you are populating their internal transaction callbacks?

                          • 10. Re: XARecovery - MQ/Messaging merge
                            mskonda Apprentice

                             

                            2006-09-05 15:55:47,999 TRACE [org.jboss.jms.tx.MessagingXAResource] MessagingXAResource[-2147483640] committing < 131075, 27, 25, 1-a6eddd7:c333:44fd8a7c:17aa6eddd7:c333:44fd8a7c:181 > (two phase)
                            2006-09-05 15:55:47,999 TRACE [org.jboss.jms.tx.ResourceManager] commiting xid < 131075, 27, 25, 1-a6eddd7:c333:44fd8a7c:17aa6eddd7:c333:44fd8a7c:181 >, onePhase=false
                            2006-09-05 15:55:48,001 TRACE [org.jboss.jms.client.container.ClientLogInterceptor] invoking ConnectionDelegate[-2147483641].sendTransaction(TransactionRequest[TWO_PHASE_COMMIT, < 131075, 27, 25, 1-a6eddd7:c333:44fd8a7c:17aa6eddd7:c333:44fd8a7c:181 >])
                            2006-09-05 15:55:48,002 TRACE [org.jboss.jms.client.delegate.DelegateSupport] invoking sendTransaction on server
                            2006-09-05 15:55:48,002 TRACE [org.jboss.jms.server.remoting.JMSServerInvocationHandler] invoking org.jboss.remoting.InvocationRequest@40c4d5
                            2006-09-05 15:55:48,002 TRACE [org.jboss.jms.server.container.ServerLogInterceptor] invoking ConnectionAdvised->ConnectionEndpoint[-2147483641].sendTransaction(TransactionRequest[TWO_PHASE_COMMIT, < 131075, 27, 25, 1-a6eddd7:c333:44fd8a7c:17aa6eddd7:c333:44fd8a7c:181 >])
                            2006-09-05 15:55:48,003 TRACE [org.jboss.jms.server.endpoint.ServerConnectionEndpoint] Two phase commit commit request received
                            2006-09-05 15:55:48,003 INFO [org.jboss.messaging.core.tx.TransactionRepository] ====== Xid(arjuna format) I: ======< 131075, 27, 25, 1-a6eddd7:c333:44fd8a7c:17aa6eddd7:c333:44fd8a7c:181 >
                            2006-09-05 15:55:48,003 INFO [org.jboss.messaging.core.tx.TransactionRepository] ====== Xid:(arjuna2jboss format) II ======org.jboss.messaging.core.tx.XidImpl@3312f2d0
                            2006-09-05 15:55:48,007 INFO [org.jboss.messaging.core.tx.TransactionRepository] ====== Xid already in gloablToLocalMap III ======org.jboss.messaging.core.tx.XidImpl@3312f2d0
                            2006-09-05 15:55:48,008 INFO [org.jboss.messaging.core.tx.TransactionRepository] ====== Tx I : ======: null
                            2006-09-05 15:55:48,008 INFO [org.jboss.messaging.core.tx.TransactionRepository] ====== Tx II: ======: TX(262145):PREPARED
                            2006-09-05 15:55:48,008 INFO [org.jboss.messaging.core.tx.TransactionRepository] ====== Tx using III: ======: TX(262145):PREPARED
                            2006-09-05 15:55:48,008 TRACE [org.jboss.jms.server.endpoint.ServerConnectionEndpoint] Committing TX(262145):PREPARED
                            2006-09-05 15:55:48,009 TRACE [org.jboss.messaging.core.tx.Transaction] executing before commit hooks TX(262145):PREPARED
                            2006-09-05 15:55:48,009 DEBUG [org.jboss.mx.loading.UnifiedClassLoader] New jmx UCL with url null
                            2006-09-05 15:55:48,010 DEBUG [org.jboss.mx.loading.RepositoryClassLoader] setRepository, repository=org.jboss.mx.loading.HeirarchicalLoaderRepository3@e7eec9, cl=org.jboss.mx.loading.UnifiedClassLoader3@168442e{ url=null ,addedOrder=0}
                            2006-09-05 15:55:48,019 TRACE [org.jboss.messaging.core.plugin.JDBCPersistenceManager] UPDATE JMSMessageReference SET State = 'C', TransactionId=NULL WHERE TransactionId = 262145 AND State = '+' removed 1 row(s)
                            2006-09-05 15:55:48,020 TRACE [org.jboss.messaging.core.plugin.JDBCPersistenceManager] DELETE FROM JMSMessageReference WHERE TransactionId = null AND State = '-' updated 0 row(s)
                            2006-09-05 15:55:48,022 TRACE [org.jboss.messaging.core.plugin.JDBCPersistenceManager] DELETE FROM JMSTransaction WHERE TransactionId = 262145 removed 1 row(s)
                            2006-09-05 15:55:48,024 TRACE [org.jboss.messaging.core.tx.Transaction] committed TX(262145):COMMITTED
                            2006-09-05 15:55:48,024 TRACE [org.jboss.messaging.core.tx.Transaction] executing after commit hooks TX(262145):COMMITTED
                            2006-09-05 15:55:48,024 TRACE [org.jboss.messaging.core.tx.Transaction] commit process complete TX(262145):COMMITTED
                            2006-09-05 15:55:48,024 TRACE [org.jboss.jms.server.endpoint.ServerConnectionEndpoint] ConnectionEndpoint[-2147483641] processed transaction successfully
                            2006-09-05 15:55:48,024 TRACE [org.jboss.jms.server.container.ServerLogInterceptor] ConnectionAdvised->ConnectionEndpoint[-2147483641].sendTransaction() OK
                            2006-09-05 15:55:48,024 TRACE [org.jboss.jms.client.delegate.DelegateSupport] got server response for sendTransaction
                            2006-09-05 15:55:48,024 TRACE [org.jboss.jms.client.container.ClientLogInterceptor] ConnectionDelegate[-2147483641].sendTransaction() OK


                            From the abvoe trace, you can see the tx has been committed.

                            • 11. Re: XARecovery - MQ/Messaging merge
                              Tim Fox Master

                              Madhu - can you answer these 3 questions:

                              The JBoss TS Recovery manager lives on the client side. How does it get hold of the list of XAResources to recover?

                              Does it use ServerConnectionEndpoint::getPreparedTransactions()?

                              When you are creating the Transaction instances from the database and adding to the transation repositiry are you sure you are populating their internal transaction callbacks?

                              • 12. Re: XARecovery - MQ/Messaging merge
                                Tim Fox Master

                                Ok, to cut a long story short - can you please post your code that creates the transaction objects from the database and adds them to the transaction repository, and I'll show you what's going wrong....

                                • 13. Re: XARecovery - MQ/Messaging merge
                                  mskonda Apprentice

                                   

                                  "timfox" wrote:

                                  The JBoss TS Recovery manager lives on the client side. How does it get hold of the list of XAResources to recover?

                                  You have to pass a parameter to let the JBossTS know who's the provider.

                                  Does it use ServerConnectionEndpoint::getPreparedTransactions()?

                                  yes, the code is modified to get the preparedTransactions (you can see from the trace I've posted)

                                  When you are creating the Transaction instances from the database and adding to the transation repositiry are you sure you are populating their internal transaction callbacks?

                                  This is something I asked team to look at my earlier JBMESSAGING-407 issue. I short, yes, when a transaction is created (resurrected), I associate with a TransactionCallback. See below:

                                  public void associateCallbackToPreparedTx(Transaction tx)
                                   {
                                   TransactionCallback callback = (TransactionCallback) tx.getKeyedCallback(this);
                                  
                                   if (callback == null)
                                   {
                                   callback = new TransactionCallback(tx);
                                  
                                   tx.addKeyedCallback(callback, this);
                                   }
                                   }
                                  


                                  *I am adding only a TransactionCallback! I didn't get through much about memory callbacks and others.

                                  • 14. Re: XARecovery - MQ/Messaging merge
                                    Tim Fox Master

                                    When you load the prepared transactions from the database, you need to "reconstitute" them in exactly the same state they were in after the prepare was originally executed.

                                    This means loading all the messages and acks into the InMemoryCallback. Otherwise when you call commit(), nothing will get delivered or acknowledged in memory since you haven't loaded the data.

                                    You also need to create a TxCallback (which I see you have already done) - the TXCallback doesn't need to have data populated in it, since it's data was already persisted when prepare was completed.

                                    1 2 Previous Next