UserTransactionImple.abortWithoutAck() throws NPE?

[ruthbd]

When invoking rollback() on a UserTransaction after catching an Exception, the underlying UserTransactionImple throws an NPE when calling abortWithoutAck():

Caused by: com.arjuna.wst.SystemException: java.lang.NullPointerException
 at com.arjuna.mwlabs.wst.at.remote.UserTransactionImple.abortWithoutAck(UserTransactionImple.java:360)
 at com.arjuna.mwlabs.wst.at.remote.UserTransactionImple.rollback(UserTransactionImple.java:153)
 at

This is using JBossTS 3.3.0 (I think) and JBossAS 4.3 EAP.

Could someone help with pointing me in what direction I should be looking to debug this? Thanks!

Regards,
Brice Ruth

[adinn]

Hi Brice. It's not really clear to me where the error is coming from as the exception trace shows the line where the SystemException is created rather than the wrapped NullPointerException which is being caught. What i suggest you do is attach a debugger to your program and add a breakpoint on line 360 (inside the catch). When the break is reached evaluate

 ex.printStackTrace(System.out)

This will display a stack trace for the NPE which should show you the line which is trying to dereference a null pointer. I strongly suspect it will be line 326 indicating that ctx is null (i.e. there is no currently active transaction).

[ruthbd]

I will see if I can capture this ... no problem.

[ruthbd]

... no problem

Famous last words.

But, I got it. Here ya go:

21:33:46,122 ERROR [STDERR] java.lang.NullPointerException
21:33:46,122 ERROR [STDERR] at com.arjuna.mwlabs.wst.at.remote.UserTransactionImple.abortWithoutAck(UserTransactionImple.java:326)
21:33:46,122 ERROR [STDERR] at com.arjuna.mwlabs.wst.at.remote.UserTransactionImple.rollback(UserTransactionImple.java:153)
21:33:46,122 ERROR [STDERR] at com.example.lbpa.core.util.advice.WsTransactionalAttributeSourceAdvice.processException(WsTransactionalAttributeSourceAdvice.java:60)
21:33:46,122 ERROR [STDERR] at com.example.lbpa.core.util.advice.WsTransactionalAttributeSourceAdvice.invoke(WsTransactionalAttributeSourceAdvice.java:39)
21:33:46,122 ERROR [STDERR] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
21:33:46,122 ERROR [STDERR] at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:631)
21:33:46,122 ERROR [STDERR] at com.example.lbpa.web.struts.ext.TransactionalActionServlet$$EnhancerByCGLIB$$c2e4d91b.doSuperPost(<generated>)
21:33:46,122 ERROR [STDERR] at com.example.lbpa.web.struts.ext.TransactionalActionServlet.doPost(TransactionalActionServlet.java:105)
21:33:46,122 ERROR [STDERR] at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
21:33:46,122 ERROR [STDERR] at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
21:33:46,122 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
21:33:46,122 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
21:33:46,122 ERROR [STDERR] at com.example.lbpa.web.filter.JsonWrapperFilter.doFilter(JsonWrapperFilter.java:90)
21:33:46,122 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
21:33:46,122 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
21:33:46,122 ERROR [STDERR] at com.example.lbpa.web.filter.XMLSerializerWorkerFilter.doFilter(XMLSerializerWorkerFilter.java:55)
21:33:46,122 ERROR [STDERR] at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:183)
21:33:46,122 ERROR [STDERR] at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:138)
21:33:46,122 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
21:33:46,122 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
21:33:46,122 ERROR [STDERR] at com.example.lbpa.core.util.hibernate.OracleTraceFilter.doFilter(OracleTraceFilter.java:64)
21:33:46,122 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
21:33:46,122 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
21:33:46,122 ERROR [STDERR] at com.example.lbpa.web.filter.RetrieveWorkerFilter.doFilter(RetrieveWorkerFilter.java:78)
21:33:46,122 ERROR [STDERR] at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:183)
21:33:46,122 ERROR [STDERR] at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:138)
21:33:46,122 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
21:33:46,122 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
21:33:46,122 ERROR [STDERR] at org.securityfilter.filter.SecurityFilter.doFilter(SecurityFilter.java:182)
21:33:46,122 ERROR [STDERR] at com.example.internalsso.exampleSecurityFilter.doFilter(exampleSecurityFilter.java:172)
21:33:46,122 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
21:33:46,122 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
21:33:46,138 ERROR [STDERR] at com.example.lbpa.web.util.CacheControlFilter.doFilter(CacheControlFilter.java:58)
21:33:46,138 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
21:33:46,138 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
21:33:46,138 ERROR [STDERR] at com.example.lbpa.web.util.AjaxNoCacheFilter.doFilter(AjaxNoCacheFilter.java:54)
21:33:46,138 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
21:33:46,138 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
21:33:46,138 ERROR [STDERR] at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
21:33:46,138 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
21:33:46,138 ERROR [STDERR] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
21:33:46,138 ERROR [STDERR] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
21:33:46,138 ERROR [STDERR] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
21:33:46,138 ERROR [STDERR] at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:179)
21:33:46,138 ERROR [STDERR] at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
21:33:46,138 ERROR [STDERR] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
21:33:46,138 ERROR [STDERR] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
21:33:46,138 ERROR [STDERR] at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
21:33:46,138 ERROR [STDERR] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
21:33:46,138 ERROR [STDERR] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:241)
21:33:46,138 ERROR [STDERR] at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
21:33:46,138 ERROR [STDERR] at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:580)
21:33:46,138 ERROR [STDERR] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
21:33:46,138 ERROR [STDERR] at java.lang.Thread.run(Thread.java:595)

I still don't quite grok the internal workings between UserTransaction and the JTA Tx, but it *almost* seems (from incidental log output) that the JTA Tx is rolling back *before* the ut.rollback(). Not sure if that leaves resources in an invalid state when abortWithoutAck() is invoked ...

This would be where my knowledge of how rollback works in a 2PC world is fairly slim.

Thanks,
Brice

[adinn]

Thanks for debugging this, Brice. As I suspected the NPE comes from line 326 which means that the transaction has already been disassociated from the current thread when you call rollback.

Just in case you are unsure where this is happening, I'll put this in context: .at.remote.UserTransactionImple is the class used to implement a WSTX transaction in an XTS client. So, it looks like you have configured Spring to create a WSTX transaction inside one of your web servlets and then, presumably invoked some other web service which has failed, causing Spring to rollback the transaction. If this is so then is this second service using bridging to allow it to access XA resources? Anyway, at the point where the exception occurs Spring has caused rollback on the WSTX transaction in the client but it has already been disassociated from the thread indicating that rollback has already been called.

Firstly, I'll note that I think the WSTX implementation is wrong here because it should not try to dereference a null pointer. This is what is causing the SystemException shown in your trace to be thrown. The WSTX code ought to detect the null pointer and explicitly throw an exception to notify that the TX has already been disassociated.

The obvious candidate for this exception is UnknownTransactionException but throwing that may actually be misleading. UnknownTransactionException is currently thrown when the client thinks it is in a TX and talks to the coordinator but gets a no such TX response i.e. it indicates something is wrong within the WSTX layer. If the transaction has already been rolled back then, arguably, a further attempt to rollback should get a SystemException to indicate that the client is doing something wrong. I'll think about this before I decide what to do but I need more input from you. So, the question is why has your transaction already been rolled back.

Am I right in assuming that your servlet is invoking a second service via the TX bridge.

Have you installed handlers on your client connections to the second service which rollback the WSTX transaction?

If so then you may be confusing Spring because it is expecting to do the rollback. Can you please let me know a bit more about your setup here?

regards,


Andrew Dinn

[ruthbd]

Thanks for your continued help, Andrew.

The way I *think* I have things configured is as such:

1. 
   
2. Request comes into app A, advice around the top-level servlet gets a UserTransaction, then starts a bridge to JTA
   
3. processing continues
   
4. processing invokes a remote web service supporting WS-TX
   
5. in-handler joins coordination context provided in SOAP header and starts a bridge to local JTA (note: service is *optionally* able to join WS-TX, but will always join/create a UserTransaction - it detects if a *remote* coordinator is present and if so, doesn't drive the UserTransaction to completion, rather just calling suspend() if a remote coordinator *is* present)
   
6. if an exception occurs, I *think* Spring on the remote end will mark the JTA transaction as rollbackOnly ... it should notice that the Spring transaction is jca-subordinate to a parent and not try to commit/rollback
   
7. if a remote coordinator is present (as it is in this case), remote service should just suspend() the UserTransaction at its end
   
8. control comes back to "local" application and the remote service has thrown a fault .. local JTA transaction is marked rollbackOnly because an exception in the business layer was detected
   
9. I'm not 100% sure what happens here .. sometimes exceptions in the business layer propagate up to the top-level servlet and we call ib.stop() and ut.rollback() - what I *think* is happening in this case (maybe?) is we call ib.stop() and then ut.commit() and in the commit(), rollbackOnly is detected ... maybe that throws an exception?
   
10. if an exception is detected within the try/catch for starting/stopping/commiting the bridge & usertransaction, we always try to stop() the bridge and rollback() the usertransaction. Since the bridge may have already been stopped, we first check to see if its still available, otherwise we just skip it and try to rollback the usertransaction.
    
    
    I'll grab the WsTransactionalSourceAdvice and post that here, as well as the FaultHandler on the remote end. I'll also try to step through this scenario and clear up the exact behavior in the last couple points above.
    
    

[ruthbd]

OK, here's the advice we use to control everything in the "local" application ...

package com.example.lbpa.core.util.advice;

import org.aopalliance.intercept.MethodInterceptor;
import org.aopalliance.intercept.MethodInvocation;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.jboss.txbridge.InboundBridge;
import org.jboss.txbridge.TxBridgeManager;

import com.arjuna.mw.wst.UserTransaction;
import com.arjuna.mw.wst.UserTransactionFactory;
import com.arjuna.wst.WrongStateException;

public class WsTransactionalAttributeSourceAdvice implements MethodInterceptor {
 private static final Log LOG = LogFactory.getLog(WsTransactionalAttributeSourceAdvice.class);

 public Object invoke(MethodInvocation invocation) throws Throwable {
 UserTransaction ut = UserTransactionFactory.userTransaction();
 InboundBridge ib = null;
 Object obj = null;
 try {
 try {
 ut.begin();
 } catch (WrongStateException e) {
 processException(e, ib, ut);

 // Try to re-establish
 ut = UserTransactionFactory.userTransaction();
 ut.begin();
 }
 ib = TxBridgeManager.getInboundBridge();
 ib.start();

 obj = invocation.proceed();

 ib.stop();
 ut.commit();
 } catch (Throwable t) {
 throw processException(t, ib, ut);
 }
 return obj;
 }

 private RuntimeException processException(Throwable t, InboundBridge ib, UserTransaction ut) {
 WsTransactionalAttributeSourceAdvice.LOG.error("Processing exception " + t.getMessage());
 RuntimeException re = new RuntimeException(t);
 try {
 if (ib != null) {
 WsTransactionalAttributeSourceAdvice.LOG.debug("Stopping InboundBridge");
 try {
 ib.stop();
 } catch (RuntimeException e) {
 WsTransactionalAttributeSourceAdvice.LOG.error(
 "Exception thrown while processing exception " + e.getMessage(), e);
 }
 }

 if (ut != null) {
 WsTransactionalAttributeSourceAdvice.LOG.debug("Rolling back UserTransaction ...");
 ut.rollback();
 WsTransactionalAttributeSourceAdvice.LOG.debug("... done.");
 }
 } catch (Throwable th) {
 WsTransactionalAttributeSourceAdvice.LOG.error("Exception thrown while processing exception " + th.getMessage(), th);
 re = new RuntimeException(th.getMessage(), re);
 }
 return re;
 }
}

[ruthbd]

Here's the Fault handler that's executed on the remote service ...

/**
 *
 */
package org.jboss.wst.contrib.xfire.service;

import org.codehaus.xfire.MessageContext;
import org.codehaus.xfire.handler.AbstractHandler;

import com.arjuna.mw.wst.BusinessActivityManager;
import com.arjuna.mw.wst.BusinessActivityManagerFactory;
import com.arjuna.mw.wst.TransactionManager;
import com.arjuna.mw.wst.TransactionManagerFactory;
import com.arjuna.mw.wst.UserTransaction;
import com.arjuna.mw.wst.UserTransactionFactory;

/**
 * XFireOutHeaderContextProcessor.
 * @author BDR011
 *
 */
public class XFireFaultHeaderContextProcessor extends AbstractHandler {

 /**
 * @see org.codehaus.xfire.handler.Handler#invoke(org.codehaus.xfire.MessageContext)
 * @param context
 * @throws Exception
 */
 public void invoke(MessageContext context) throws Exception {
 if (RemoteCoordinationIndicator.isRemoteCoordinatorPresent()) {
 suspendTransaction();
 } else {
 rollbackTransaction();
 }
 RemoteCoordinationIndicator.flush();
 }

 private void rollbackTransaction() throws Exception {
 UserTransaction tx = null;
 try {
 tx = UserTransactionFactory.userTransaction();

 if (tx != null)
 {
 tx.rollback() ;
 }
 }
 catch (final Exception e) {
 throw XFireInHeaderContextProcessor.processException(e, null, tx);
 }
 }

 /**
 * Suspend the current transaction.
 */
 private void suspendTransaction() throws Exception
 {
 try
 {
 /*
 * There should either be an Atomic Transaction *or* a Business Activity
 * associated with the thread.
 */
 final TransactionManager transactionManager = TransactionManagerFactory.transactionManager() ;
 final BusinessActivityManager businessActivityManager = BusinessActivityManagerFactory.businessActivityManager() ;

 if (transactionManager != null)
 {
 transactionManager.suspend() ;
 }

 if (businessActivityManager != null)
 {
 businessActivityManager.suspend() ;
 }
 }
 catch (final Exception e)
 {
 e.printStackTrace();
 throw e;
 }
 }
}

[marklittle]

"adinn" wrote:

Firstly, I'll note that I think the WSTX implementation is wrong here because it should not try to dereference a null pointer. This is what is causing the SystemException shown in your trace to be thrown. The WSTX code ought to detect the null pointer and explicitly throw an exception to notify that the TX has already been disassociated.

+1

It should throw IllegalStateException

The obvious candidate for this exception is UnknownTransactionException but throwing that may actually be misleading.

See above. IllegalStateException is the better choice because that more closely matches JTA semantics.

UnknownTransactionException is currently thrown when the client thinks it is in a TX and talks to the coordinator but gets a no such TX response i.e. it indicates something is wrong within the WSTX layer. If the transaction has already been rolled back then, arguably, a further attempt to rollback should get a SystemException to indicate that the client is doing something wrong. I'll think about this before I decide what to do but I need more input from you. So, the question is why has your transaction already been rolled back.

I think we should change the signature ;-) Of course that would happen naturally if we supported JAXTX ;-)

[adinn]

It should throw IllegalStateException

Ok, we actually already have a WrongStateException which e.g. is thrown when a begin is attempted and the thread is already associated with a TX. I guess this is the equivalent of JTA IllegalStateException.

I think we should change the signature ;-)

Not just the signature but also the javadoc ;-). It currently says

 /**
 * The rollback operation will terminate the transaction and return
 * normally if it succeeded, while throwing an appropriate exception if it
 * didn't. If there is no transaction associated with the invoking thread
 * then UnknownTransactionException is thrown.
 */

This should probably say IllegalStateException for this case and identify UnknownTransactionException as marking the situation where the coordinator does not know about the existence of the TX. Same applies for commit which is currently documented as follows:

 /**
 * The transaction is committed by the commit method. This will execute
 * the PhaseZero, 2PC and OutcomeNotification protocols prior to returning.
 * If there is no transaction associated with the invoking thread then
 * UnknownTransactionException is thrown. If the transaction ultimately
 * rolls back then the TransactionRolledBackException is thrown. When
 * complete, this operation disassociates the transaction from the current
 * thread such that it becomes associated with no transaction.
 */

n.b. commit does currently detect a null local context and throws UnknownTransactionException.

[adinn]

Ok Brice, thanks for the code. the problem is that your cleanup routine in the advice code can end up calling stop twice or end up calling rollback after a commit has disassociated th etx from the thread. Here is what you should have:

 obj = invocation.proceed();
 try {
 ib.stop();
 } catch (Throwable t) {
 throw processException(t, null, ut);
 }
 try {
 ut.commit();
 } catch (Throwable t) {
 throw processException(t, null, null);
 }
 } catch (Throwable t) {
 throw processException(t, ib, ut);
 }
 return obj;
 }

Notice that an exception in stop passes null instead of ib, avoiding a second call to ib. An exception during commit passes null for both arguments avoiding calls to stop and rollback.

This is because when you call ib.stop the bridge ensures the bridged to transaction is no longer left active so you must not call stop again even it throws an error. The same is true once you call commit. It disassociates the transaction even if if throws an error. So if you call rollback after calling commit it will throw another exception. In the current code it throws a SystemException because of the NPE. But even if we patch this to throw a WrongStateException your code still will not work.

As a general comment I'll note that exception handling during cleanup is very awkward when you have multiple cleanups to do. You have to keep very careful track of which things still need cleaning up and which ones are already cleared by previous cleanup attempts. Your original code was nice and neat but that's a sure sign that something is up because handling cleanup errors is almost always not neat :-)

[adinn]

Ok Brice, thanks for the code. the problem is that your cleanup routine in the advice code can end up calling stop twice or end up calling rollback after a commit has disassociated th etx from the thread. Here is what you should have:

 obj = invocation.proceed();
 try {
 ib.stop();
 } catch (Throwable t) {
 throw processException(t, null, ut);
 }
 try {
 ut.commit();
 } catch (Throwable t) {
 throw processException(t, null, null);
 }
 } catch (Throwable t) {
 throw processException(t, ib, ut);
 }
 return obj;
 }

Notice that an exception in stop passes null instead of ib, avoiding a second call to ib. An exception during commit passes null for both arguments avoiding calls to stop and rollback.

This is because when you call ib.stop the bridge ensures the bridged to transaction is no longer left active so you must not call stop again even it throws an error. The same is true once you call commit. It disassociates the transaction even if if throws an error. So if you call rollback after calling commit it will throw another exception. In the current code it throws a SystemException because of the NPE. But even if we patch this to throw a WrongStateException your code still will not work.

As a general comment I'll note that exception handling during cleanup is very awkward when you have multiple cleanups to do. You have to keep very careful track of which things still need cleaning up and which ones are already cleared by previous cleanup attempts. Your original code was nice and neat but that's a sure sign that something is up because handling cleanup errors is almost always not neat :-)

[ruthbd]

Thanks, Andrew - I'll refactor and see if that takes care of this situation!

[marklittle]

I think either WrongStateException or IllegalStateException. Probably the former until we support JTA wholesale.

[ruthbd]

Andrew,

The change you recommended worked, except that I had to refactor processException() slightly to throw a specific named exception, not just RuntimeException - so that I could check in the outer catch {} if the exception being caught was that one, if so, I just rethrow and don't invoke processException again (otherwise we're back to where we started).

So, in a way, its even less nice and neat now ;-)

I have a follow-up question now on how exactly rollback semantics work between coordinators, but I'll start a new topic on that.