Transactions in a replicated cache

[jerrygauth]

I'm trying to cache several nodes within a transaction in a replicated cache environment. I'm using the latest JBossCache HEAD code along with a fairly recent build of JBossAS HEAD. The transaction is being executed from a servlet running on one of the cluster nodes.

If I use a non-replicated environment (e.g., INVALIDATION_SYNC), the transaction executes properly and commits or rolls back as expected.

If I use a replicated environment (e.g., REPL_SYNC), the transaction executes properly as long as I configure the cache to use the DummyTransactionManager and I use this manager to execute the transaction from the servlet.

If I get a transaction manager from JNDI (either java:TransactionManager or UserTransaction) and I configure my cache to use JBossTransactionManagerLookup, the transaction always fails during the prepare phase on the remote node. As noted, the same code works properly if the cache isn't replicated. I've attached the relevant stack trace.

I've looked for a JBossCache test that executes similarly (i.e., transaction in a replicated environment) but I've only been able to find one that executes in standalone mode using the dummy manager.

Any ideas on whether I'm missing something here?

ERROR [RpcDispatcher] failed invoking method
org.jboss.util.NestedRuntimeException: Already associated with a tx; - nested throwable: (java.lang.IllegalStateException: Already associated with a tx)
at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:4247)
at org.jboss.cache.TreeCache._replicate(TreeCache.java:3894)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jgroups.blocks.MethodCall.invoke(MethodCall.java:286)
at org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:236)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:618)
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:515)
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:326)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUp(MessageDispatcher.java:734)
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.access$300(MessageDispatcher.java:566)
at org.jgroups.blocks.MessageDispatcher$1.run(MessageDispatcher.java:703)
at java.lang.Thread.run(Thread.java:534)
Caused by: java.lang.IllegalStateException: Already associated with a tx
at org.jboss.tm.TxManager.resume(TxManager.java:660)
at org.jboss.cache.interceptors.TxInterceptor.handleRemotePrepare(TxInterceptor.java:263)
at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:124)
at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:67)
at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:128)
at org.jboss.cache.TreeCache.invokeMethod(TreeCache.java:4241)

[manik]

From Brian's response email:

Copying Manik.

I think this is because JBoss's TxManager.resume(Transaction tx) checks
if there is an existing uncompleted tx bound to the thread, and if there
is throws an exception. It doesn't check if the tx passed as a method
arg is the same as the one already bound to the thread.

Here's the relevant code from TxInterceptor.handleRemotePrepare()

if (ltx == null)
{
ltx = createLocalTxForGlobalTx(gtx); // creates new LTX
and associates it with a GTX
if (log.isDebugEnabled())
{
log.debug("(" + cache.getLocalAddress() + "):
started new local TX as result of remote PREPARE: local TX=" + ltx + ",
global TX=" + gtx);
}
}
else
{
//this should be valid
if (!isValid(ltx)) throw new CacheException("Transaction
" + ltx + " not in correct state to be prepared");
}

//associate this thread with this ltx
txManager.resume(ltx);
if (log.isTraceEnabled()) {log.trace("Resuming existing
transaction " + ltx + ", global TX=" + gtx);}


I think if the last couple lines were inside the preceding else block
the problem would go away.

Good job on this one, Jerry. Clearly we need some integration tests that
exercise JBossCache with JBoss' transaction manager. Not easy in
standalone mode, as JBoss AS is required. The HttpSession replication
tests used to do that, but now they all use BatchModeTransactionManager,
which is a subclass of DummyTransactionManager.


Brian

[manik]

Thanks Brian and Jerry, I'll look into and test a fix for this.

This is what releasing a DR is supposed to check. I expect QA will spot this one as well, when running through the latest AS test suite. :)

[manik]

Ok, I've fixed this in HEAD. Expecting to hear from QA with this problem in 1.3.0.DR1 any minute now! :-) We do need some integration tests, the question is where should such tests reside. I dare say they should be in the AS test suite. And such tests probably already do exist in AS.

I also spotted something else that has changed with this version of JBossCache: with 1.2.4, I could write a test that created 2 replicated cache instances in the same JBossAS instance, and I could run these tx tests. With 1.3.0, I had to fire up 2 separate AS instances, and run one treecache instance in each. The problem was to do with JMX bindings and the new CacheMgmtInterceptors - the JMX server threw up when trying to bind 2 cache instances which were supposed to be in the same cluster since the cluster name was used as the service name. I don't think this is a "real world" problem at all, although I think it may break some AS tests.

[jerrygauth]

The fix seems to be fine. At least my own sample code having transactions in a replicated cache now works. :)

re: the multiple instance issue - I recently modified the interceptor mbean registration code so that the mbean will only be registered if not already registered. So the server should no longer regurgitate but it's still not going to do what you need. It seems like you would need to define two separate instances of the cache having different service names. You can also disable the interceptor mbeans by setting UseInterceptorMbeans to false in the cache's configuration.

[jerrygauth]

I've just tried using two clustered cache instances on a single server and it seemed to run fine. I created the second cache configuration by copying the first and then modifying the service name. The instances were registered properly and replication worked as expected.

[brian.stansberry]

Ok, I've fixed this in HEAD. Expecting to hear from QA with this problem in 1.3.0.DR1 any minute now! :-) We do need some integration tests, the question is where should such tests reside. I dare say they should be in the AS test suite. And such tests probably already do exist in AS.

I don't believe so :( The AS tests I'm aware of relate to HttpSession replication and clustered SSO, and those now use a cache that uses a DummyTransactionManager subclass as its TM. There may be some EJB3 related tests that use JBoss' TM; I'm not sure.

I'll open a JIRA issue for this. It will be an AS issue; I guess the corresponding JBossCache issue is that a release has to pass the unit tests for whatever AS version(s) it's supposed to integrate with. It's great you're already pushing this with 1.3.0, Manik.

[brian.stansberry]

Opened http://jira.jboss.com/jira/browse/JBAS-2750.

[manik]

It's great you're already pushing this with 1.3.0, Manik.

Integration woes, great? I don't think so! :-)