Concurrency in jBPM4

[roschmel]

I hava a question regarding concurrency in jBPM4:

In jBPM3 when you send multiple signals to the same processinstance to the EJB Command Service (in a possible clustereed environment) you get StaleObjectExceptions beacuse the executions are not synchronized.

Is it planned to address this issue in jBPM4? I am asking because we are building a synchronisation mechansim currently and I need to know whats planned in this area.

AND:
I know StaleObjects are part of the design in jBpm3..... i read all the threads. But is simply does not work in an Integration Environment where you also talk a lot with non transacted or non XA transacted resources.

[tom.baeyens]

depends on what you mean with "addressed" we'll be adding automatic retries in case of SOSE's.

but explain me " But is simply does not work in an Integration Environment where you also talk a lot with non transacted or non XA transacted resources."

what is in that case the behaviour that' you'ld want from jbpm ?

you want your processes to move from one state to the next in a transactional way. in such a state transition,you want to access and update non transactional resources. now what do you want to see happening when an exception occurs ?

[roschmel]

Hi,

let my try to explain my thoughts in short:

1) A lot of external system cannot be rolled back (WebService Calls, Legacy Systems...).
2) To make it worse, there are also a lot of system where a retry causes major issues (think about a Bank Transfer - doing it twice may not be good choice)

So in an ideal world you would exactly know where the process execution has crashed. This would mean committing every little step - but this is not really cool stuff.....

So its first about knowing where the process have crashed - therefore you need logging information - just another topic I am currently discussing in the user forum.

And its about minimizing the chance, that a jBPM Transaction is getting rolled back. It is espacially about not considering a rollback as "a normal path of execution" - a rollback is for really bad things happended and not per design. So I think SOSE and retrying as a designed path of execution is simply wrong.

What I would expect:
If I have a process 1 waiting in state-A and waiting in state-B for the callback from an external system and both of them sending me the signal at the same time, I expect to have them executed just one after the other and I do not expect getting exception and letting the engine doing things twice.

We are currently working on a pessimistic locking feature for this situation. But there are two major issues with it:
1) You have to find out what the shared resource is based on the command you get (and its most of the time the parents- parents- ... processinstance)
2) We are doing it in the CommandService EJB and because its a synchronous method call we have to block the thread until we get the look

[tom.baeyens]

"roschmel" wrote:
So in an ideal world you would exactly know where the process execution has crashed. This would mean committing every little step - but this is not really cool stuff.....

this is what we call a transactional script.
this is where jpdl is exactly suited for. see async="true". you can have the commit after every step.

"roschmel" wrote:
So its first about knowing where the process have crashed - therefore you need logging information - just another topic I am currently discussing in the user forum.

no logging information is irrelevant. you need to map how you want java exceptions to affect the progress of your process.

for each potential failure of non transactional resources you need to decide : either the process continues or the process execution rolls back till the last committed point of execution.

"roschmel" wrote:

And its about minimizing the chance, that a jBPM Transaction is getting rolled back. It is espacially about not considering a rollback as "a normal path of execution" - a rollback is for really bad things happended and not per design. So I think SOSE and retrying as a designed path of execution is simply wrong.

then i think you didn't get it yet.

"roschmel" wrote:

What I would expect:
If I have a process 1 waiting in state-A and waiting in state-B for the callback from an external system and both of them sending me the signal at the same time, I expect to have them executed just one after the other and I do not expect getting exception and letting the engine doing things twice.

this makes sense ! i didn't think if this use case yet.

should be really simple to get this one going. as we already have the exclusive job execution. that means that you can mark async messages or timers as being exclusive.

first, you have to make sure that all the progress in the process is made by the job executor. this would be possible to use the async command service. then you need to make sure that the async command service would set the exclusive flag to true.

then you need to make sure that all other jobs that are created in this process also set this exclusive flag to true.

good point!

we have all the infrastructure in place to make sure that we also cover this use case. i'll keep that in mind and hopefully this is fully documented by jbpm 4.0.0.GA

[roschmel]

Hi Tom,

thanks for your answer....

Unfortunately - based on my knowledge - only the Thread based JobExecutor handles the exclusive Flag but not the enterprise deployment JobListenerBean. So here would the idea which we are working on would fit in. We could use a JBossCache for managing the exclusive locks for jobs. This would mean, that you also have to do the signals using jobs because it would be the only chance the get them - in a way - synchronized.

Currently when you are using async=true in an enterprise deployment you get the transaction demarcation as you desribed but just imagine following situation:
If I have a process 1 going to node A with async=true and going to node B with async=true (after a fork) then both JMS messages will be commited to the queue to the same time and if you have a fast enought server it will execute both in parallel and you will again get a StaleObject in one of them.

So If you want a lot of transaction demarcation you bring yourself in a situation where you increase the chance of StaleObject and therefore for rollback dramatically.

This is why we did not use async=true for transaction demarcation but we are using BMT and built an ActionHandler that we call SavePoint to commit a running transaction but keeping the Session and the JbpmContext open. This is possible with a little configuration work and one dirty code trick in the ActionHandler to handle the autoSaveInstances.

One other comment:

for each potential failure of non transactional resources you need to decide : either the process continues or the process execution rolls back till the last committed point of execution.

This works well for problems in the external resource - of course you can control this quite well. But if pull the plug right after the non transacted resource did its work sucessfully or if you cannot persist the process itself due to any database failure than you must know in a post mortem view, how long the process ran and where it crashed exactly to have the chance to bring the process back to a "normal" running condition.

[camunda]

I didn't get the problem yet. You don't have JTA in place but work in a clustered environment? Or just one resource is non transacted?

I know quite huge systems where jbpm with JMS and "normal" JTA transactions are no problem, even with some rollbacks. Especially with JMS, the message is automatically redelivered and JMS scales quite well, why is that such a big problem?

BTW: Pesimistic locking could be done in jbpm as well by using Hibernate if you really need it.

Normally I don't think baking your own stuff there is a good idea...

[aguizar]

Unfortunately - based on my knowledge - only the Thread based JobExecutor handles the exclusive Flag but not the enterprise deployment JobListenerBean.

It does, starting from version 3.3.0 / 3.2.4. An ExecuteJobsCommand is constructed with the exclusive job IDs and sent to the command queue. The command executor then executes the jobs serially.

[roschmel]

@Alex:
Thanks for the tip - I just had a look at it. It seems quite interesting solution. Two issue are comming to my mind, when I look at the code:

If you have multiple excluive jobs which should be executed concurrently I think you will 1) get stale object exception in the ExecuteJobCommand because the will overwrite the lockowner each other and 2) change the transaction behaviour because you execute multiple jobs with the ExecuteJobsCommand in one transaction. This might have unwanted side effects.

[roschmel]

@Bernd

We have a clustered environment where we have following resources:
1) The database transaction for jbpm to an oracle
2) Another or mapping technology to a legacy database which is transacted but non xa
3) Various WebService calls which are causing operation in external systems and cannot be rolled back
and we might get other stuff to.... (its an EAI scenario). We are using a JTA Manager but we do not use XA. And I know - perhaps an ESB would fit better into it - but for different reasons its currently not an option.

I will try to sum up my issues to make it more clear:

1) The jBPM throws StaleObjectExceptions when one ProcessInstance is getting Signals concurently. When your process is using non transacted (or not XA capable) resources this might cause problems because you cannot just retry easily.

Our solution: We replaced the CommandService Implementation with our own implementation managing clusterwide locks on process instances.

2) When you use non transacted (or non XA capable) resources you might want to have more fine granular transaction demarcation. Therefore you can use async="true" but this will again increase the possiblity for 1) at least with the source up to 3.2.3 (thx alex)

Our solution: We are using BMT and implemented an ActionHandler which can commit the transaction during process execution - but we need a dirty code hack for this.

[camunda]

@roschmel: Okay, now its a bit clearer. Why don't you use hibernate pessimistic locking in this case? And: With async=true, couldn't you get completly rid of the problematic situations if you use it often enough?

Alex wrote:
An ExecuteJobsCommand is constructed with the exclusive job IDs and sent to the command queue. The command executor then executes the jobs serially.

Couldn't this lead to the behavior of a repeating timer creating jobs which cannot be processed as fast as created (assume the system is very utilized)? This is why we normally decided against executing timer stuff over JMS.

Or what do you think?

[roschmel]

I do not use pessimistic locking because it is necassary to synchronize against the parent parent parent.... process instance because a signal anywhere in the process can cause data changes up in the hierachy. So I would have to make a loadInstanceForUpdate on the parent instance.

Using pessimitic locking on the row level causes deadlock situations in the database where orcale throws exceptions then.

I decided to do it with a jboss cache so that i can relase the jbpmcontext during the wait for the lock in order not to hold the the database resources during waiting. But the loadInstnaceForUpdate would definitly be an option I think.