XA Recovery on Journal...

[clebert.suconic]

I have done some small changed on XID on Journal.

The Journal should be agnostic about XID IMO, The StorageManager should be the correct place to encode and decode that.

So, I have changed it a little bit. journal.prepare is now receiving an EncodingSupport, and TransactionHolder is aways holding a byte[] representing the XID. JournalStorageManager later will translate that after load is done.


I have also added a few more assertions around Journal tests to make sure the data is handled fine on prepareTransactions.

[clebert.suconic]

I have a question related to PrepareTransaction, Recoveries and NonDurableMessages

We only store the Xid on Journal, if there are PersistentMessages. (which is aways my expecation).

That means, if we produce messages on JBM as part of a Distributed Transaction using only NonDurableMessages those XIDs won't be recovered in case of a server crash/restart.

Would that cause any issues on the TransactionManager?

[timfox]

I have done some small changed on XID on Journal.

The Journal should be agnostic about XID IMO, The StorageManager should be the correct place to encode and decode that.

Yes, indeed. The journal is designed to be a generic journal which just deals with byte[]. The idea here is it could be used by other projects with journalling needs in the future. So, certainly it should not be polluted with domain specific abstractions.

So, I have changed it a little bit. journal.prepare is now receiving an EncodingSupport, and TransactionHolder is aways holding a byte[] representing the XID. JournalStorageManager later will translate that after load is done.

Sounds good.

[timfox]

I have a question related to PrepareTransaction, Recoveries and NonDurableMessages

We only store the Xid on Journal, if there are PersistentMessages. (which is aways my expecation).

That means, if we produce messages on JBM as part of a Distributed Transaction using only NonDurableMessages those XIDs won't be recovered in case of a server crash/restart.

Would that cause any issues on the TransactionManager?

This is related to https://jira.jboss.org/jira/browse/JBMESSAGING-1282

[timfox]

Also, BasicXARecoveryTest is using netty and actual disk storage.

All these kinds of tests should use InVM transport.

We also need to test case where recovery occurs without the server being restarted.

[ataylor]

So, I have changed it a little bit. journal.prepare is now receiving an EncodingSupport, and TransactionHolder is aways holding a byte[] representing the XID. JournalStorageManager later will translate that after load is done.

cheers Clebert.

All these kinds of tests should use InVM transport.

Is InVM working? I kept getting a BufferOverflowException!

java.nio.BufferOverflowException
 at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:165)
 at org.jboss.messaging.core.remoting.impl.ByteBufferWrapper.putBytes(ByteBufferWrapper.java:322)
 at org.jboss.messaging.core.message.impl.MessageImpl.encode(MessageImpl.java:137)
 at org.jboss.messaging.core.remoting.impl.wireformat.SendMessage.encodeBody(SendMessage.java:101)
 at org.jboss.messaging.core.remoting.impl.wireformat.PacketImpl.encode(PacketImpl.java:152)
 at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.doWrite(RemotingConnectionImpl.java:435)
 at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl.access$900(RemotingConnectionImpl.java:161)
 at org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl$ChannelImpl.send(RemotingConnectionImpl.java:825)
 at org.jboss.messaging.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:261)
 at org.jboss.messaging.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:129)
 at org.jboss.messaging.tests.integration.xa.BasicXaRecoveryTest.testBasicSendWithCommit(BasicXaRecoveryTest.java:124)

[ataylor]

We also need to test case where recovery occurs without the server being restarted.

I still have some more tests to write which will include this.

[clebert.suconic]

"timfox" wrote:

This is related to https://jira.jboss.org/jira/browse/JBMESSAGING-1282

What I thought that should be happening, is TransactionImpl aways storing the prepare on disk. That way an empty transaction would be recovered instead of "killing" the whole branch on the TM.

public void prepare() throws Exception
 {
 if (state != State.ACTIVE)
 {
 throw new IllegalStateException("Transaction is in invalid state " + state);
 }

 if (xid == null)
 {
 throw new IllegalStateException("Cannot prepare non XA transaction");
 }

 pageMessages();

// if (containsPersistent)
// {
 storageManager.prepare(id, xid);
// Without the if (contains Persistent)
// }

 containsPersistent = true; // just to make sure commit will also use the storageManager.

 state = State.PREPARED;
 }

I'm not sure if the journal could work with empty transactions, but that should be easily adaptable.

[timfox]

"timfox" wrote:

This is related to https://jira.jboss.org/jira/browse/JBMESSAGING-1282

What I thought that should be happening, is TransactionImpl aways storing the prepare on disk. That way an empty transaction would be recovered instead of "killing" the whole branch on the TM.

My understanding, is that if the transaction branch is empty, then prepare should return return XAResource.XA_RDONLY

(See ServerSessionImpl.XAPrepare)

And then the transaction manager knows not to ever call commit or rollback for that branch.

I however recommend clarify these finer points of XA with a member of the transactions team.

[timfox]

Ah no. We only return RD_ONLY if there the tx is completely empty, including non persistent operations, so yes I think we need to record a blank tx branch in the journal.

[clebert.suconic]

"timfox" wrote:
Ah no. We only return RD_ONLY if there the tx is completely empty, including non persistent operations, so yes I think we need to record a blank tx branch in the journal.

Besides.. if we choose to store a Prepare Record for an Empty Transaction, we would need the commit as we need the CompletionRecord of the Prepare as we need that in order to reclaim files.

So, if you add a PrepareRecord, you need a CommitRecord or you would break reclaiming.

[adrian.brock]

"timfox" wrote:
Ah no. We only return RD_ONLY if there the tx is completely empty, including non persistent operations, so yes I think we need to record a blank tx branch in the journal.

Returning "read only" would be the most optimal thing to do.
You then don't record the prepare in the journal.

However, once you do that the non-peristent message should be removed from memory
or anywhere else you reference it.

If you return "read only" you won't get the commit/rollback notification.

That obviously means non-persistent messages won't get redelivered
once they've tried to be prepared in a 2PC transaction, even if the eventual
outcome of the transaction is a rollback.

That is perfectly spec compliant though since there's no guarantee
the message would get delivered in the first place, let alone redelivered after a rollback. :-)

[timfox]

Returning "read only" would be the most optimal thing to do.
You then don't record the prepare in the journal.

However, once you do that the non-peristent message should be removed from memory
or anywhere else you reference it.

The problem with returning RD_ONLY as I see it, is that if I have a tx branch with a bunch of non persistent sends, and return RD_ONLY then commit will never get called, so those messages won't ever get routed in memory to its subscribers (since routing occurs on commit)

One other possibility is to return OK, but not actually persist the branch if it only contains non persistent operations, then if we subsequently get a commit or rollback for that branch on recovery, we can return OK to the commit/rollback even though we can't find the branch. The tx mgr then thinks the transaction was resolved ok, even though the non persistent messages were lost (which is fine anyway).

[clebert.suconic]

"Tim Fox" wrote:
..we can return OK to the commit/rollback even though we can't find the branch. The tx mgr then thinks the transaction was resolved ok, even though the non persistent messages were lost (which is fine anyway).

If there is no other implications, that's the most optimal solution, as we won't need to record anything on the journal for NonPersistentMessages on that case.

[adrian.brock]

"timfox" wrote:

One other possibility is to return OK, but not actually persist the branch if it only contains non persistent operations, then if we subsequently get a commit or rollback for that branch on recovery, we can return OK to the commit/rollback even though we can't find the branch. The tx mgr then thinks the transaction was resolved ok, even though the non persistent messages were lost (which is fine anyway).

The important part with returning OK is that you need to persist the XID
so you can return it from XAResource.recover() - otherwise you might
confuse the transaction manager.

It doesn't matter whether there is actually anything behind the XID.

http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4154677#4154677

I guess the real problem occurs when you have a non-persistent send()
in a failed transaction commit, i.e.. another branch failed to commit, not you.

The TM then calls later recover() and you return nothing.
The TM would be well within its rights to assume you already committed
(although it could be confused that you lost memory of the XID you said was OK :-)