Stateless Process Engine executor threading issues

[jorgemoralespou_2]

Hi,
We've developed an engine for executing process instances. Until know, we only need execution of stateless processes, so we get the root token, and signal on it until reaching the final state. Everything works fine till here. We have a WS facade for starting the Process executions.
The problem is that execution times increase linearly as the number of clients increase, which isn't the desired behaviour.
Everything seems to scale up properly, but for the JBPM engine. Theres is plenty of cpu time left, plenty of threads configured in the AS, plenty of everything, but isn't scaling.
I post you here the execution logic we use so any of you can help us finding a solution.

Thanks

 JbpmContext ctx = null;
 boolean newContext = false;
 try {
 // Fetch JbpmContext
 ctx = getJbpmConfiguration().getCurrentJbpmContext();
 if (ctx == null) {
 ctx = getJbpmConfiguration().createJbpmContext();
 newContext = true;
 }

 // Get process definition
 ProcessDefinition processDefiniton = null;
 if (version == null) {
 processDefiniton = ctx.getGraphSession()
 .findLatestProcessDefinition(processName);
 } else {
 processDefiniton = ctx.getGraphSession().findProcessDefinition(
 processName, version);

 }
 if (processDefiniton == null) {
 throw new ProcessDefinitionNotFoundException(processName,
 version);
 }

 // Create instace, add input params as context variables

 ProcessInstance instance = new ProcessInstance(processDefiniton);
 instance.getContextInstance().setTransientVariable(
 "REQUEST", req);

 // Execute process
 do{
 instance.getRootToken().signal();
 } while (!EndState.class.isAssignableFrom(instance.getRootToken().getNode().getClass()));

 // Get output parameters
 Map<String, Serializable> response = (Map<String, Serializable>) instance.getContextInstance().getTransientVariable("RESPONSE");
 return response;
 } catch (Exception e) {
 throw new JbpmEngineException("Unexpected error", e);
 } finally {
 if (ctx != null && newContext) {
 ctx.close();
 }
 }
 }

[kukeltje]

my first impression is that you do not use jbpm as it was inteded to be used. Holding on to a context blocks the engine for other processinstances to run. Getting a new context each time is not expensive, so get one, signal it, keep the token/processinstance id somewhere and go on with other instances. Once needed, get a context again, use the token/process id to retrieve the instance and signal it.

[jorgemoralespou_2]

Thanks Ronald,
But actually I pasted an old version of the code.

 try {
 // Fetch JbpmContext
 ctx = getJbpmConfiguration().createJbpmContext();

 // Get process definition
 ProcessDefinition processDefiniton = null;
 processDefiniton = ctx.getGraphSession()
 .findLatestProcessDefinition(processName);
 if (processDefiniton == null) {
 throw new ProcessDefinitionNotFoundException(processName);
 }

 // Create instace, add input params as context variables
 ProcessInstance instance = new ProcessInstance(processDefiniton);
 instance.getContextInstance().setTransientVariable(
 "REQUEST", req);

 // Execute process
 do{
 instance.getRootToken().signal();
 } while (!EndState.class.isAssignableFrom(instance.getRootToken().getNode().getClass()));

 // Get output parameters
 Map<String, Serializable> response = (Map<String, Serializable>) instance.getContextInstance().getTransientVariable("RESPONSE");
 return response;
 } catch (Exception e) {
 throw new JbpmEngineException("Unexpected error", e);
 } finally {
 if (ctx != null) {
 ctx.close();
 }
 }
 }

This code, as I think obtains a context, and signals every token in the execution path right through the end. I don get hold to a context.

I'm not sure if your answer is still aplicable. I'll try what you suggest though.

Also, is there any way of not using services like Messaging, Authentication and Logging? I've read that commenting the code in jbpm.cfg.xml should do the work, but it? actually doing all the work but persisting to database.

[kukeltje]

you do complete one instance of a process in one loop, continuously holding on to a context. That is (afaik, but never tried it) blocking.

[jorgemoralespou_2]

You might be right. And that probably explains the behaviour I'm observing, but how can I actually signal a Token if I'm not hold to a Context.
Execution of nodes should occur within a Context, as long as I know, and since I signaling every node in the execution path, the execution get's done right.
For what I understand from the sources, JBPMContext is bound to the execution thread, and for that reason, seems multi-threadable. As long as I instantiate one JBPMContext in a new thread, the one dispatching the http request from my ws faccade, it should scale properly.
Probably there is an issue with such a load for Hibernate, or something else that escapes my eyes.
Also, I don't want to persist results from executions, or at least only the minimal required, the process instance result, but I don't see any way of disabling this, or any other superflual services for my needs.

[kukeltje]

Yes, execution of nodes should occur in a context. For that a jBPM context is used. It does not mean it has to be the identical, same context. That is why I said you have to hold on to a tokenid or processid to use that to get to the token/process when opening a new context. Look at the examples, testcases etc...

And yes, openening a new context in a new thread should scale (at least I hope so).... disabeling real persistency is simple by using an in-memory hsqld database and deleting the processinstance once it completes. If you do use a database, tuning of indexes etc... should indeed be done. jBPM only delivers minimal tuning..

[jorgemoralespou_2]

Hi Ronald,
I still don't get the point. Our executions are in such a way, for now, that once a token is signaled, it must continue through the end, that's why we get hold on to the context. There is no state saving till the end of the execution, because there isn't any waiting state. I don't see any clue in the examples or in any other document you pointed me.
Any change we made so far, leaves us at the same point, double client load, double response time, and plenty of free cpu.
Is there any concurrency issue or synchronicity issue regarding JBPM's context?

[kukeltje]

Is there any concurrency issue or synchronicity issue regarding JBPM's context?

Not that I know of if it is in another thread, but I don't know everything :-)....

Can you create a minimal testcase (no web, just standalone unittest) with real multithreading, that demonstrates the problem.

btw, do you call any external services synchronously? if so, how long do they take to process?

[jorgemoralespou_2]

Hi Ronald,
We're trying to isolate the problem and work a testcase that runs outside the AS. It 's no easy job, due to high integration with the AS., but I'll try to post it here when we have it.

btw, do you call any external services synchronously? if so, how long do they take to process?

Sure we call external services. In fact our system is acting as a small orchestration software, but that is the point on running it inside a multithreaded AS. I expect it to scale on load properly, not double. The external systems are also low on load, so the problem is internal to our system.

[kukeltje]

I also tried contacting one of the core developers to get a statement on this... I also already told him you use it in a kind of orchestration way.

The reason I asked is if you use forks, and want a kind of parallel processing, you have to use async actions in the nodes, otherwise all 'legs' of a fork will be processed sequentially.

And that an AS is multithreadded does not mean everything is....

[jorgemoralespou_2]

"kukeltje" wrote:
I also tried contacting one of the core developers to get a statement on this... I also already told him you use it in a kind of orchestration way.

Thank you very much for your interest.

"kukeltje" wrote:

The reason I asked is if you use forks, and want a kind of parallel processing, you have to use async actions in the nodes, otherwise all 'legs' of a fork will be processed sequentially.
And that an AS is multithreadded does not mean everything is....

We don't use forks. Right now, only nodes, decisions and exception handlers.
I have already read about the forking issue, but that is not our case.
As soon as we have a real testcase, I'll post it here.

[jorgemoralespou_2]

Hi Ronald, I'm posting here the code.

Web service

@WebService(name = "ProcessTestService", targetNamespace = "http://test/ws", serviceName = "ProcessTestService")
@SOAPBinding(style = SOAPBinding.Style.RPC)
public class ProcessTestService {

 public String testInTransaction(String param1) throws Exception {
 Map request = new HashMap();
 StatelessEngine engine = new StatelessEngine();
 Object o = engine.executeProcessInstance("test-som-process", request);
 return "result";
 }

 @WebMethod
 public String test(String param1) throws Exception {
 String retValue = null;
 UserTransaction tx = getUserTransaction();
 tx.begin();
 try {
 retValue = testInTransaction(param1);
 tx.commit();
 } catch (Exception e) {
 tx.rollback();
 throw e;
 }
 return retValue;
 }

 private UserTransaction getUserTransaction() {
 UserTransaction ut = null;
 try {
 InitialContext ctx = new InitialContext();
 ut = (UserTransaction) ctx.lookup("UserTransaction");
 } catch (NamingException e) {
 appLog.error("Error obtaining the UserTransaction", e);
 }
 return ut;
 }
}

The engine

import java.util.Map;
import org.jbpm.JbpmContext;
import org.jbpm.graph.def.ProcessDefinition;
import org.jbpm.graph.exe.ProcessInstance;
import org.jbpm.graph.node.EndState;
import com.hp.som.bpm.SomJbpmEngineConfigurationMBean;
import javax.naming.Context;
import javax.naming.InitialContext;
import org.jbpm.JbpmConfiguration;

public class StatelessEngine{

 public Object executeProcessInstance(String processName, Map request) throws Exception{
 JbpmContext ctx = null;
 try{
 ctx = getJbpmConfiguration().createJbpmContext();

 // Get process definition
 ProcessDefinition processDefiniton = ctx.getGraphSession().findLatestProcessDefinition(processName);
 if (processDefiniton == null) {
 throw new Exception("Process definition not found");
 }

 ProcessInstance instance = new ProcessInstance(processDefiniton);
 instance.getContextInstance().setTransientVariable("REQUEST", request);

 // Execute process
 do{
 instance.getRootToken().signal();
 }while (!instance.getRootToken().hasEnded());

 // Check that execution is on a end state. if not, throw exception
 if (!EndState.class.isAssignableFrom(instance.getRootToken()
 .getNode().getClass())) {
 throw new Exception("Process finished not in an End Node");
 }

 Object response = instance.getContextInstance().getTransientVariable("RESPONSE");
 return response;
 } finally {
 if (ctx != null) ctx.close();
 }
 }


 private static JbpmConfiguration conf = null;

 protected JbpmConfiguration getJbpmConfiguration() throws Exception {
 if (this.conf == null){
 JbpmConfiguration config = null;
 String jndiName = null;
 Context ctx = new InitialContext();

 // Get jndi name for jbpm configuration
 jndiName = "java:/jbpm/JbpmConfiguration";
 config = (JbpmConfiguration)ctx.lookup(jndiName);
 this.conf = config;
 }
 return this.conf;
 }

}

The test process

<process-definition
 xmlns=""
 name="test-som-process">
 <start-state name="start">
 <transition name="" to="node1"></transition>
 </start-state>
 <end-state name="end">
 </end-state>
 <node name="node1">
 <action class="com.hp.som.test.TestActionHandler"></action>
 <transition name="" to="node3"></transition>
 </node>
 <node name="node3">
 <action class="com.hp.som.test.TestActionHandler"></action>
 <transition name="" to="node4"></transition>
 </node>
 <node name="node4">
 <action class="com.hp.som.test.TestActionHandler"></action>
 <transition name="" to="node5"></transition>
 </node>
 <node name="node5">
 <action class="com.hp.som.test.TestActionHandler"></action>
 <transition name="" to="node6"></transition>
 </node>
 <node name="node6">
 <action class="com.hp.som.test.TestActionHandler"></action>
 <transition name="" to="node8"></transition>
 </node>
 <node name="node8">
 <action class="com.hp.som.test.TestActionHandler"></action>
 <transition name="" to="end"></transition>
 </node>
</process-definition>

And finally test handler

public class TestActionHandler implements ActionHandler {
 public void execute(ExecutionContext context) throws Exception {
 try {
 // Create a process logger whith process name
 customExecute(context);
 // The action have been executed correctly
 context.getToken().signal();
 } catch (Exception e) {
 e.printStackTrace();
 throw e;
 }
 }

 protected void customExecute(ExecutionContext context) throws Exception {
 String param = "Do nothing important";
 }

}

When you load test the web service, you can see how response time doubles as client doubles, but cpu stays low.

[kukeltje]

Thanks, but... this is not something I can run out of the box. I need a webcontainer (that I have) , webservice loadtestclient (strange word, but I don't have it) etc...etc...etc... so unfortunately it's not usable for me this way (it's not a unit test). Do you think you can still create a unit test? (actionhandler as an inner class etc..) process as string internally etc.. that way I just have one class and run it...

[jorgemoralespou_2]

I'll try to post it tomorrow morning. Thanks.

[tom.baeyens]

jorgemoralespou_2,

you can use jBPM persistent or transiently. so you have to decide if you want persistence first. you always need a jbpmContext, but in case you don't need persistence, you can just parse the process from a file and create instances with the ProcessInstance constructor (all inside one or more jbpm context blocks)

apart from that, you should not require an engine that drives executions through the process. if you need to feed an external trigger, it means that your process behaves as a wait state. so it's a matter of not using wait states to avoid the external signals. if you create a new process instance and signal it, it will run until the end in one go if there are no waitstates inside the process.