JBPM - concurrent process execution fails

[efip10]

In my deployment, I use a pool of MDBs to start a number of workflows.

The workflows progress from node to node, collecting objects in hibernate session, and at some state they fire a message to (another) queue. After that, the finally { jbpmContext.close() } is reached, and the session begins to flush.

In the meantime, in some other place in the galaxy, another pool of MDBs receives the messages and starts processing on them. First thing that is done is jbpmContext.loadProcessInstanceForUpdate(id); at which state the execution fails with Hibernate exceptions, like

org.hibernate.HibernateException: null index column for collection: org.jbpm.graph.exe.ProcessInstance.instances

and

org.hibernate.LazyInitializationException: illegal access to loading collection

At this stage, I cannot even end or cancel my token, since the processInstance is not loaded correctly.

As a result, my workflow stays in zombie state and actually never ends.

We use JBoss 4.0.5, Hibernate, Spring, MySQL (tried with all isolation levels) and jBPM 3.2.

Any suggestions on how to make sure the processInstance is fully saved to the DB before I read it? No mutexes can help here because of clustered deployment.

Thanks in advance.

[efip10]

Forgot to mention: there are no tasks, joins/forks, or child processes used in my flow. It is really, really simple continuous flow.

[kukeltje]

what in your view is a simple continous flow, is in fact straight trough processing. BUT.... with influences from external systems. This is an issue currently under debate/investigation.

Saving the process before reading it is done when closing the context. So the sooner you do that, the sooner you can read it again in other systems.

[estaub]

I'm not sure, but I think that if you mark your message-firing node as 'asynch="true"', this will force a commit and take care of the problem.

If you happen to be firing the message from the node-entry event, this won't work; you'll need to mark a prior node, perhaps introducing a dummy one just for this purpose.

-Ed Staub

[kukeltje]

Ed,

Will putting an async=true on the node realy force a commit? That has to be documented well then (rollback
's further on in the process a rollback will not be possible then) This is not a problem if the process has to wait for the return value of the async message node.

If it is not saved with async=true and I'm not sure, the result of this async action is to return *before* there is a real save, we get the other 'well known ;-)' exceptions.

[efip10]

Thanks for the replies, guys.

I tried to put async=true on all of my message-firing nodes, and actually that does not help. I can see in the log that context.close() is finished after the message arrives and the instance is loaded.

Except for using a voluntary Thread.sleep(), does anyone have an idea of how I can make sure the process has finished the commit?

[efip10]

FYI guys, in the meantime I have a workaround of using

ctx.getJbpmContext().save(ctx.getProcessInstance())

in my action - before sending a message; all this works only because of @TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED) on MDB, so every save() also commits.

I am not sure whether this might be a solution for everyone (I am sure it wouldn't), so it'd be nice if a call to loadProcessInstance() would not return until the instance is in some steady state.

[estaub]

"kukeltje" wrote:
Ed,

Will putting an async=true on the node realy force a commit? That has to be documented well then (rollback
's further on in the process a rollback will not be possible then) This is not a problem if the process has to wait for the return value of the async message node.

Ronald,

Take a look at Node.enter():

// execute the node
if (isAsync) {
ExecuteNodeJob job = createAsyncContinuationJob(token);
MessageService messageService = (MessageService) Services.getCurrentService(Services.SERVICENAME_MESSAGE);
messageService.send(job);
token.lock(job.toString());
} else {
execute(executionContext);
}

So yeah, I think it leads to a commit.
Unfortunately, as efip10 found out, there's a race condition between the commit and the new job.

-Ed Staub

[koen.aers]

efib10,

can you post the details of your processdefinition and your client code? we are investigating these serious concurrency issues that exist in jBPM at the moment. There are two failing jUnit tests in the org.jbpm.job.executor package (JobLoadJoinTest and JobLoadSubprocessTest). As you know debugging and analysing these type of problems is very painful, so we need all the possible info we can get.

Thanks,
Koen

[estaub]

Koen,

See also http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4008324#4008324
It's the same thing, I think. Unfortunately, no JIRA or unit test there, either.

-Ed Staub

[efip10]

Koen,

Since there's quite a lot of stuff, I'd rather have it sent by email than posted. May I use [e-mail] for that?

Thanks

[koen.aers]

Best is to extract a minimum setup that produces the problem and attach that to the JIRA issue (http://jira.jboss.com/jira/browse/JBPM-983).

Regards,
Koen

[e120274]

I have found a problem with instances of processes. When I start multiple instance of same process, JMSIntegrationService mix the instances inside the Token variable. Two instances started at the same time and in one instance flow did not continue and other instance activity is mixed with current instance process. I have overridden the JmsIntegrationService to get invokes for each activity to handle. Take the token's process instance id and see the problem. Can Jbpm BPEL work with multiple process instances at the same time? It is very important problem I think? Can anyone tried to handle multiple process instances?
Thansk a lot.

[kukeltje]

please use a new topic for this. This topic is on concurrency within one process, not for concurrent instances of a process.

[efip10]

As I have tried to build the minimum setup, I come to think that it is not JBPM problem.

Let's put it this way: I create a row in DB, send a JMS message, and create another row. In the end I commit my transaction.

In the meantime my message is processed, and there is a query to the DB for the rows I have created (but not commited yet - or maybe yes?). Obviously, the same race condition would still hold, no JBPM involved.

Maybe using JMS is not a good idea here, and maybe I should use database as a synchronizer (i.e. write a message to my DB and consume it from there by some other thread) rather than JMS. In this case, all the rows will be in DB.