Top level transaction for JBPM Persistence + 'Nested'-Transa

[ralfoeldi]

I have the following problem:

If I have a workflow with two transactable actions executed in a single call and the second one fails the only way to rollback the first action is to rollback the transaction in which both actions were executed.

If token.signal() is called in the same transaction that will persist JBPM state, the rollback would also prevent any changes to the workflow state, i.e. the workflow would be where it started. Even if we would be reacting to failures inside of the workflow (e.g. setting the token in an error-node, etc.) this would never be persisted and is therefore worthless.

Pseudo code a)
tx.begin();
...
processInstance = processDefinition.createProcessInstance();
processInstance.signal();
JBPM ActionA ok;
JBPM ActionB crash => tx.setRollbackOnly();
graphSession.saveProcessInstance(processInstance);
tx.commit();

If we suspend the transaction used to retrieve / instantiate a process instance, start a new 'nested' transaction, commit or rollback this new transaction, resume the old transaction and use that to commit JBPM state changes, everything seems ok except for the fact that the 'nested' transaction could commit and the 'JBPM'-transaction could fail.

Pseudo code b)
tx.begin();
...
processInstance = processDefinition.createProcessInstance();
TransactionManager..suspend();

tx.begin();
processInstance.signal();
JBPM ActionA ok;
JBPM ActionB crash => tx.setRollbackOnly();
tx.commit();
this is where we're not water tight
TransactionManager.resume();
graphSession.saveProcessInstance(processInstance);
tx.commit();

Right now I don't see how a JBPM workflow could safely react to the failure of a transaction in the workflow itself. But then that seems hard to believe as I can't possibly be the only one with this problem. (Its basically a problem of Java not supporting nested transactions, but thats a different thread.)

What would the JBPM 'Best Practice' be for this problem?

[koen.aers]

You might want to implement ActionB with the help of a message queue. Instead of executing the action synchronously, you put a message on the queue in the same transaction as the token signal. This message is read by an MDB that performs the stuff you want to do in a new transaction. If something goes wrong in that transaction, you can rely on a 'compensation handler' to put the token in an error state or do whatever you want to do. Does this help?

Regards,
Koen

[ralfoeldi]

Hi Koen,

that is what I am already doing, at least for all resources that do not support transactions and for a somewhat other purpose.

If ActionA sends an e-mail there is no way to rollback or 'compensate' an e-mail, so I send a JMS Message that will only get sent when and if the transaction commits. I then have the option to let the token wait until the recieving MDB sends the mail and triggers the waiting token - this again by way of JMS Message and MDB so that all this happens in a new transaction without putting recieving message, sending e-mail, triggering token, saving JBPM state all in one transaction.

That does solve a problem, but not my problem.

If I use two transactions as in 'pseudo code b' I have seperated JBPM state from action execution and therefore would be able to persist JBPM state even if I rollback the action transaction.

The problem with two transactions is the 'gap' between the commit of the inner and the outer transaction, the red part of the pseudo code. The inner transaction could commit and the outer transaction could fail and we would have an inconstistent state.

This problem exists in your szenario as well. Whoever does the 'compensation handling' cannot rely on a specific JBPM state because the outer tranasction might not have committed. That would reduce error handling to a all-or-nothing option (think fork, 3 new tokens a,b,c, error in MDB action for c token, 'error handler' tries to do sth with c token, but that does not even exist, because outer transaction commit failed.)

As I wrote in the orginal post this is actually a problem of Java not supporting nested transactions and I don't think my 'pseudo code b' can be made completely fail safe (because of Java).

I was hoping for some other, totally different, JBPM specific(?) approach that would solve the problem. This cannot be the first time this problem came up?

I am fully aware that there is only a very, very small chance of this problem arising, but the purpose of transactions is deterministic behaviour...

Greetings from Switzerland,
the country of banks and insurance companies that do not tolerate 'gaps' in their processes... (my client happens to be one of those 'banks and insurance companies')

Rainer

[koen.aers]

Rainer,

Don't worry, we have banks and insurance companies amongst our customers ;-).
But I think I don't get your point very clearly. Either the transactions are isolated and then you have two different transaction contexts that can commit or rollback independently, or you have one transaction context (maybe but not necessarily with nested transaction contexts) that fails or commits. Those two options are possible with jBPM.
After looking more closely your second scenario, I seems not very logical to me that you want to use a processinstance object in the nested transaction that you created in the outer transaction before this transaction is ever committed. Am I missing something?

Regards,
Koen

[ralfoeldi]

Hi Koen,

glad to here that about the banks and insurances companies. In case this here crashes I'll contact you and we will see if JBPM already has 'nobody ever got fired for hiring IBM'-status :-)

In a certain sense you did understand my problem correctly: if you have two transactions these are isolated. Thats the 'problem' The transaction executing token.signal() could commit, the transaction persisting jbpm could fail.

But putting both in one transaction is not an option. I am forced to use rollback if an ActionB wants to prevent ActionA from doing something. If I only use a single transaction jbpm state would never change if the transaction is rolled back as it would never be persisted.

Using uncommited processinstances? Well, I DO have the java object (processinstance) to work with. The way I understand JBPM persistence is that a bunch of things happen on java objects that only get persisted when the dust settles down. But that doesn't really effect the szenario. It would be the same if I commited the new processinstance, began an new transaction for the actions, commited those and then began another transaction for persisting jbpm state. I guess I would have to do some hibernate magic with reconnecting disconnected objects (I'm not the Hibernate crack...rtfm for me on that one)

Back to the original problem: maybe this problem just isn't solveable and I will have to live with it. It realy boils down to missing nested transactions in Java where the inner transaction could rollback independently but would only commit together with the outer transaction :-(

I was just hoping.... :-)

Thanks for the effort

Rainer

[ralfoeldi]

Hi Koen,

if you're still there and interested (or anybody else) let me put my problem into more generic words. (The 'gap' I mentioned in the first post just blew up full force... so this is - at least for me a very real problem.)

I want to be able to execute Actions (Nodes with ActionHandlers) in a transactional context in which I can rollback these Actions.

If an Action decides to rollback the transaction it might also want to influence the workflow (e.g. ok, I'm failing in whatever I'm doing here, I give up and set token to error state x.)

These changes to the workflow state - although performed in the context of the rolled back transaction - must be persisted (or else it wouldn't make any sense.)

I also want to make sure that the Action transaction is only commited if and when the workflow state changes are committed.

(This is what just blew up in an endless circle where the Actions committed - and subsequently sent mails ... lots of mails :-( - the jbpm workflow state for various reasons did not commit, i.e. starting all over again.)

Ok, maybe I'm repeating myself and should listen to my own "live with it" from the last post, but maybe, just maybe someone out there has an idea?

(Everything described above is up and running even though it took alot of attaching and detaching of process instances from hibernate - you were right on that one, but the final problem of the 'gap' that just blew up remains, so maybe someone has a totally different point of view that might make the problem solveable.)

Greetings from Switzerland (They actually have snow here! Not the non-winter of Germany)

Rainer

[kukeltje]

what were the various reasons the jbpm workflow state did not commit if I may ask? (in the meantime I'm thinking about a possible solution)

[ralfoeldi]

Hi Ronald,

the 'various reasons' for not commiting the workflow state was a mistake on my side. LoggingSession.saveProcessLog() threw a NPE when trying to persist my own impl. of ProcessLog. This was left over from playing around with JBPM a few weeks ago, so I just removed it without debugging that any further.

But the few thousand mails it sent (to my own account :-) made it clear to me, that 'live with it' isn't really an option.

Rainer

[kukeltje]

can you create a small testcase to reproduce it? preferably with logging a record instead of sending an email ;-). With a small description of how to reproduce.

As a possible solution I was thinking of using async functionality. Having an action put something in a queue, have an action mdb pick that up and start working. If it succeeds (or fails) put a message back in the queue and have an 'command mdb' signal the process while setting a process variable or taking a certain transition based on the message content. If you then trust the reliability of JMS wouldn't that solve your problem. Not much could go wrong in jBPM and if something does, you have a rollback, but still have the message in the queue to retrigger the jbpm engine.

Any comments?

btw, doesn't your spamfilter block thousands of messages send from the same address to the same address within a few seconds? Mine does... can't see the problem there :-D

[ralfoeldi]

Hi Ronald,

you're right on track. What you suggest is exactly what I'm doing.

(Before I get into details let me repeat what the overall intention is: I want to be able to rollback the workflow execution while being able to persist workflow state. Maybe I am approaching the problem from a totally wrong perspective.)

I have
- a JbpmMDB that creates and starts proccess instances and signals tokens (depending on the recieved message)
- ActionHandlers that send JMS Messages to other MDB that do something (e.g. send e-mail)
- these MDBs do something and are able to send 'signal' messages back to the JbpmMDB to - if needed - signal a waiting token.

All this is up and running.

The problem arises in the "not much could go wrong in jBPM" - part. If something does go wrong, i.e. the tx does not commit, the message that originally triggered the JbpmMDB is redelivered, but the jms 'action' messages are already commited, sent and there is no way to get them back. Then I have a redelivered message and I start all over again... resulting in thousands and thouands of mails my spam filter does not get because I'm contacting the company smtp from the inside :-(

This is the part marked red in the original post, even though the implementation specifics have changed, that still remains the core of the problem.

Writing a test case is nontrivial, but I guess the only way to describe my problem, even thought this is not an implementation problem - everything works splendid and as expected - but a conceptual one.

Maybe this part of the code gives you an idea. (Some exception handling code and client specific stuff has been removed.)

This method is from the JbpmMDBean and called when a message sent from an external system is recieved requesting us to start a workflow. All MDB are configured to container managed transactions.

The last statement in this method sends the JMS message that triggers the next method.

(This method is ok. No problems and transactions are failsafe.)

 private void onCreate( Message message )
 {
 LOG.debug("create()");

 JbpmSession jbpmSession = null;

 try
 {
 Map map = extractMapMessage(message);

 jbpmSession = jbpmSessionFactory.openJbpmSession();

 jbpmSession.beginTransaction();

 String workflowName = (String) map
 .remove(ActorConstants.JMS_MAP_WORKFLOW_KEY);

 workflowName = StringUtil.isSet(workflowName) ? workflowName : "default";

 GraphSession graphSession = jbpmSession.getGraphSession();

 ProcessDefinition processDefinition = graphSession
 .findLatestProcessDefinition(workflowName);

 ProcessInstance processInstance = processDefinition
 .createProcessInstance();

 // now pass on all the properties we got from the JMSMessage ...
 ContextInstance contextInstance = processInstance.getContextInstance();

 contextInstance.addVariables(map);

 graphSession.saveProcessInstance(processInstance);

 // hibernate seems to need an explicit commit even if running
 // in a transaction context. actual commit is controlled by
 // the app server. (hibernate sourcecode shows that this only forces a
 // flush without any further transaction stuff)
 jbpmSession.commitTransaction();

 // now we still have to send a start Message
 // (the reason for this is basically hibernate and transactions)
 // we want to execute jbpm actions in a different transaction
 // but jbpm needs! a processinstance that is connected via
 // hibernate and that will only work if the processinstance is
 // committed and free. this IS ridiculous but at the moment I don't
 // see any other option

 /*************************************************************************
 * Setup the properties of the JMSMessage
 ************************************************************************/
 Map properties = new HashMap();

 properties.put(ActorConstants.JMS_PROP_ACTOR_KEY,
 JbpmMDBean.JMS_PROP_ACTOR_VALUE);

 properties.put(ActorConstants.JMS_PROP_ACTOR_ACTION_KEY,
 JbpmMDBean.JMS_PROP_ACTOR_ACTION_START);

 properties.put(ActorConstants.JMS_PROP_JBPM_PROCESS_INSTANCE_ID_KEY,
 new Long(processInstance.getId()));

 _jmsSendMessageBean.sendTextMessage("", properties);
 }
 catch (JMSException e)
 {
 LOG.error("ERROR: A JMSException '" + e.getMessage()
 + "' has occurred try to send a 'start' message. "
 + "Rolling back current transaction, this will cause a "
 + "redelivery of the current recieved message");

 _ctx.setRollbackOnly();
 }
 finally
 {
 if (jbpmSession != null)
 {
 jbpmSession.close();
 }
 }
 }

This method is called when the message sent in the method above is recieved and starts the process that was just created.

What we do here is splitt the execution of the workflow and the persistence of the workflow state in two transactions.

The first transaction is the one coming from the container and includes recieving the JMS message and persisting jbpm workflow state.

The second transaction is achieved by a call to _jbpmExecutionBean.startProcessInstance( long ) which is a StatelessSessionBean. This method returns the not yet persisted instance of ProcessInstance in the state after the signal() call returns. The implementation is posted below this method.

If the second transaction (the one in which the workflow executes) commits (sending further JMS messages, doing whatever) but the first transaction fails (the one persisting jbpm state) the message gets redelivered and we have the thousands and thousands of mails....

(This works fine unless the first transaction fails.)

 private void onStart( Message message )
 {
 LOG.debug("start()");

 JbpmSession jbpmSession = null;

...

 processInstanceId = (Long) message
 .getObjectProperty(ActorConstants.JMS_PROP_JBPM_PROCESS_INSTANCE_ID_KEY);

...

 ProcessInstance processInstance = null;

 try
 {
 processInstance = _jbpmExecutionBean
 .startProcessInstance(processInstanceId.longValue());

 jbpmSession = jbpmSessionFactory.openJbpmSession();

 jbpmSession.beginTransaction();

 jbpmSession.getGraphSession().saveProcessInstance(processInstance);

 jbpmSession.commitTransaction();
 }
 catch (ServiceException e)
 {
 // we get here if sth went wrong in the execution bean
 // so that the jbpm state is undefined => rollback
 _ctx.setRollbackOnly();
 }
 }
 finally
 {
 if (jbpmSession != null)
 {
 jbpmSession.close();
 }
 }
 }

This is the method that actually triggers the jbpm workflow execution. No problems here. Works as expected.

 /**
 * Starts the process defined by the given processInstanceId in a new <@link
 * UserTransaction>, i.e. calls <@link ProcessInstance#signal()> on the
 * retrieved instance.<br>
 * <br>
 * This method will start a new <@link UserTransaction>, hook up the <@link
 * ProcessInstance> to a Hibernate Session in this new transaction, call
 * <@link ProcessInstance#signal()>, close the Hibernate Session and then
 * commit or rollback the transaction depending on the Status it is in.<br>
 * <br>
 * The main purpose of this method is to seperate the transaction used to
 * persist JBPM state (this should happen in the calling class) and the
 * transaction used to handle the actions in the JBPM Workflow.
 * <strong>Therefore this method does not persist JBPM state in any active way</strong>
 * although it is possible, that Hibernate in it's magnificent, all knowing
 * glory, might do so on it's own.<br>
 * <br>
 * If <@link ProcessInstance#signal()> throws anything this is regarded as
 * undefined behaviour and a <@link ServiceException> is thrown. In this case
 * the JBPM processInstance is in an undefined state.
 *
 * @param processInstanceId
 * @return
 * @ejb.interface-method view-type="local"
 */
 public ProcessInstance startProcessInstance( long processInstanceId )
 throws ServiceException
 {
 ProcessInstance processInstance = null;

 JbpmSession jbpmSession = null;

 /***************************************************************************
 * Start a new UserTransaction
 **************************************************************************/

 // we only need user transactions if we want to pass them on to
 // the workflow, otherwise we could just use container managed tx
 UserTransaction ut = beginTransaction();

 /***************************************************************************
 * get jbpmSession, get processInstance, set UserTransaction and go...
 **************************************************************************/
 try
 {
 jbpmSession = jbpmSessionFactory.openJbpmSession();

 processInstance = jbpmSession.getGraphSession().loadProcessInstance(
 processInstanceId);

 ContextInstance contextInstance = processInstance.getContextInstance();

 contextInstance.setTransientVariable("UserTransaction", ut);

 try
 {
 /***********************************************************************
 * Workflow triggered here
 **********************************************************************/
 processInstance.signal();
 }
 catch (Throwable th)
 {
 LOG.error(
 "ERROR: eDartsActor JBPM Workflows are expected to NOT throw "
 + "exceptions. This one did so anyway. Please check workflow.",
 th);

 rollbackTransaction(ut);

 throw new ServiceException(th);
 }
 }
 finally
 {
 // we have to explicitly close the jbpmSession or else
 // hibernate won't lett us access the processInstance later on
 if (jbpmSession != null)
 {
 jbpmSession.close();
 }

 handleTransactionEnd(ut);
 }

 // if we get here the calling class gets the processInstance to do
 // whatever it wants to with it, if anything went wrong we never get here
 return processInstance;
 }

Does this do the trick or should I try do draw a few diagramms?

Rainer

[hoogenbj]

Hi,
Apologies for butting in like this, as I've not been following the discussion. However, I did notice you complaining about messages being redelivered. You could get rid of the redelivered messages like this:

if (message.getJMSRedelivered()) {
L.warn("Message was redelivered. Not reprocessing it.");
return;
}

where message is the argument for your MDB's onMessage() method.
Again, sorry if I'm restating the obvious.
Regards
Johan

[ralfoeldi]

Hi Johan,

well butting in on a private discussion is indeed very rude, but I accept your apologies :-)

I'm glad about anybody with ideas!

That could indeed be a solution. Checking for redelivery, taking that as cue for some kind of problem and setting the jbpm workflow to an error node.

The more I think about it, the better I like it. It isn't the 'non plus ultra' solution of transactual integrity and will leave a lot of manual error handling if something does go wrong, but that shouldn't happen to often anyway.

Thank you very much.

Greetings from Switzerland, the land of banks and insurances companies that can now use jbpm in a failsafe transacted way :-)

Rainer