JBCache 2.0.0 / Weblogic 9: Exception while trying to rollba

[spennec]

I'm experiencing a strange problem, with TreeCache seemingly trying to rollback a Weblogic transaction.

Here's the setup:
- Two weblogic 9 servers, running a clustered application.
- Each night, the cached data must be refreshed. Since this process requires other external processes to succeed before running, it is triggered by an external client. It's a simple Java app that calls two methods in a stateless session bean.

The cache reloader client calls one instance. This instance reloads its cache. When a specific event occurs, the second instances catches it thanks to a CacheListener and starts its own reload. We do not use replication for this reloading process since the data that would be transfered at commit-time is too big. While the cache normally uses replications, for daily operations, the reloading puts the cache in LOCAL mode, only for the reloading duration.

The following exception was encountered on the client app that triggered the reloading of the cache. On the servers, there were absolutely no exceptions, and everything went smoothly. All logs show that the two instances reloaded their cached data successfully.

Here is the stacktrace:

###### CLIENT APPLICATION
com.myCompany.commons.exception.SystemException: See nested for more details.
 at com.myCompany.dac.ejb.DacRemoteExceptionInterceptor.handleRemote(DacRemoteExceptionInterceptor.java:55)
 at com.myCompany.dac.ejb.DacRemoteExceptionInterceptor.execute(DacRemoteExceptionInterceptor.java:22)
 at com.myCompany.util.bizdel.BusinessDelegateInvocationContext.proceed(BusinessDelegateInvocationContext.java:46)
 at com.myCompany.util.bizdel.BusinessDelegate$1.invoke(BusinessDelegate.java:286)
 at $Proxy6.resetAndLoadCache(Unknown Source)
 at com.myCompany.dac.cache.DacCacheReloader.main(DacCacheReloader.java:24)

###### CAUGHT EXCEPTION
Caused by: javax.transaction.TransactionRolledbackException: BEA1-2AB0DCA982790F4DB46C: weblogic.transaction.internal.AppSetRollbackOnlyException
 at weblogic.transaction.internal.TransactionImpl.setRollbackOnly(TransactionImpl.java:524)
 at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1172)
 at org.jboss.cache.interceptors.OrderedSynchronizationHandler.beforeCompletion(OrderedSynchronizationHandler.java:84)
 at weblogic.transaction.internal.ServerSCInfo.doBeforeCompletion(ServerSCInfo.java:1150)
 at weblogic.transaction.internal.ServerSCInfo.callBeforeCompletions(ServerSCInfo.java:1128)
 at weblogic.transaction.internal.ServerSCInfo.startPrePrepareAndChain(ServerSCInfo.java:115)
 at weblogic.transaction.internal.ServerTransactionImpl.localPrePrepareAndChain(ServerTransactionImpl.java:1288)
 at weblogic.transaction.internal.ServerTransactionImpl.globalPrePrepare(ServerTransactionImpl.java:2098)
 at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:259)
 at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:228)
 at weblogic.ejb.container.internal.BaseEJBObject.postInvoke1(BaseEJBObject.java:539)
 at weblogic.ejb.container.internal.StatelessEJBObject.postInvoke1(StatelessEJBObject.java:72)
 at weblogic.ejb.container.internal.BaseEJBObject.postInvokeTxRetry(BaseEJBObject.java:374)
 at com.myCompany.dac.svc.ejb.session.DacAdmin_accoim_EOImpl.resetAndLoadCache(DacAdmin_accoim_EOImpl.java:191)
 at com.myCompany.dac.svc.ejb.session.DacAdmin_accoim_EOImpl_WLSkel.invoke(ILweblogic.rmi.spi.InboundRequest;
 Lweblogic.rmi.spi.OutboundResponse;Ljava.lang.Object;)Lweblogic.rmi.spi.OutboundResponse;(Unknown Source)
 at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:548)
 at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:224)
 at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:438)
 at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
 at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
 at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:388)
 at weblogic.rmi.internal.BasicServerRef.access$300(BasicServerRef.java:57)
 at weblogic.rmi.internal.BasicServerRef$BasicExecuteRequest.run(BasicServerRef.java:965)
 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
 at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
; nested exception is:
 weblogic.transaction.internal.AppSetRollbackOnlyException
 at weblogic.rjvm.ResponseImpl.unmarshalReturn(ResponseImpl.java:211)
 at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:338)
 at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:252)
 at com.myCompany.myProject.svc.ejb.session.myProjectAdmin_accoim_EOImpl_921_WLStub.resetAndLoadCache(Unknown Source)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at com.myCompany.util.bizdel.BusinessDelegateInvocationContext.proceed(BusinessDelegateInvocationContext.java:49)
 at com.myCompany.dac.ejb.DacRemoteExceptionInterceptor.execute(DacRemoteExceptionInterceptor.java:20)
 ... 4 more
Caused by: weblogic.transaction.internal.AppSetRollbackOnlyException
 at weblogic.transaction.internal.TransactionImpl.setRollbackOnly(TransactionImpl.java:524)
 at org.jboss.cache.interceptors.TxInterceptor$LocalSynchronizationHandler.beforeCompletion(TxInterceptor.java:1172)
 at org.jboss.cache.interceptors.OrderedSynchronizationHandler.beforeCompletion(OrderedSynchronizationHandler.java:84)
 at weblogic.transaction.internal.ServerSCInfo.doBeforeCompletion(ServerSCInfo.java:1150)
 at weblogic.transaction.internal.ServerSCInfo.callBeforeCompletions(ServerSCInfo.java:1128)
 at weblogic.transaction.internal.ServerSCInfo.startPrePrepareAndChain(ServerSCInfo.java:115)
 at weblogic.transaction.internal.ServerTransactionImpl.localPrePrepareAndChain(ServerTransactionImpl.java:1288)
 at weblogic.transaction.internal.ServerTransactionImpl.globalPrePrepare(ServerTransactionImpl.java:2098)
 at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:259)
 at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:228)
 at weblogic.ejb.container.internal.BaseEJBObject.postInvoke1(BaseEJBObject.java:539)
 at weblogic.ejb.container.internal.StatelessEJBObject.postInvoke1(StatelessEJBObject.java:72)
 at weblogic.ejb.container.internal.BaseEJBObject.postInvokeTxRetry(BaseEJBObject.java:374)
 at com.myCompany.dac.svc.ejb.session.DacAdmin_accoim_EOImpl.resetAndLoadCache(DacAdmin_accoim_EOImpl.java:191)
 at com.myCompany.dac.svc.ejb.session.DacAdmin_accoim_EOImpl_WLSkel.invoke(Unknown Source)
 at weblogic.rmi.internal.BasicServerRef.invoke(BasicServerRef.java:548)
 at weblogic.rmi.cluster.ClusterableServerRef.invoke(ClusterableServerRef.java:224)
 at weblogic.rmi.internal.BasicServerRef$1.run(BasicServerRef.java:438)
 at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
 at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:147)
 at weblogic.rmi.internal.BasicServerRef.handleRequest(BasicServerRef.java:388)
 at weblogic.rmi.internal.BasicServerRef.access$300(BasicServerRef.java:57)
 at weblogic.rmi.internal.BasicServerRef$BasicExecuteRequest.run(BasicServerRef.java:965)
 at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
 at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

The org.jboss lines seem to show that some JBossCache component provoked an Exception while trying to trigger the rollback. Unfortunately there is no message explaining the reason of the problem.

I am also very confused by the fact that there are absolutely no Exceptions on the servers.

- Is is correct to say that JBossCache was trying to mark the weblogic transaction as rollbacked but didn't succeed?
- How can I know which problem lead JBossCache to trigger the rollback?
- Is that a JBossCache bug?
- What is the state of the cache after such an Exception?
- Why is this exception not caused by an Exception on one of the servers? (or at least: why is there no trace of this exception on any of the servers?)

Thanks for your answers,

Sebastien

[manik]

Hmm, assuming the "client" java program does not have an instance of JBoss Cache, the exceptions must be on the server side.

Here is the code that triggers this rollback:

...
try
{
.....
}
catch (Throwable t)
{
 try
 {
 tx.setRollbackOnly();
 }
 catch (SystemException se)
 {
 throw new RuntimeException("setting tx rollback failed ", se);
 }
 if (t instanceof RuntimeException)
 throw (RuntimeException) t;
 else
 throw new RuntimeException("", t);
}

So the exception should be thrown on the server as well. From your logs though, it seems as though tx.setRollbackOnly() triggers an exception and the code above never gets to the point of throwing or even logging the REAL problem.

This is clearly a weblogic-specific issue since this does work correctly on other tested transaction managers.

As a workaround, we would have to log the error before setting the tx to rollback-only to ensure it is logged in such a case. Could you raise a feature request in JIRA for this?

In the meanwhile (if you are happy compiling your own version of JBoss Cache) I'd suggest getting the src code for 2.0.0.GA and making the change yourself to help try and find the problem you're experiencing.

[spennec]

Hi Manik,

Thanks for your answer :)

As you guessed, the client java program does not know anything about JBCache.

I'm going to send this link to our Weblogic-Guru and see with him what can be done.

About the JIRA entry: should I really open a feature request for this? I did it over here http://jira.jboss.org/jira/browse/JBCACHE-1275, but more see the situation as a bug, not a new feature. Anyway I'll let you handle change the category...