jBPM 6 and Quartz timer : Timer jobs do not restart automatically after server crash/restart

[anindyas79]

Hello

I am trying out the new jBPM v6 and with Quartz scheduler enabled. I followed this http://mswiderski.blogspot.in/2013/09/jbpm6-samples-with-runtimemanager.html and then subsequently followed the projects https://github.com/mswiderski/jbpm-examples/tree/master/jbpm6/jbpm-sample-cdi and https://github.com/mswiderski/jbpm-examples/tree/master/jbpm6/jbpm-sample-cdi-services/src/test/java. I am able to run process with SingleKSession as well as KSessionPerProcessInstance strategies. I have included a intermediate timer node which fires @15sec interval and then waits on a join node. When I create the proces for the first times the timer works nicely. But if I restart the server the timers are not activated by default. It is only after when if I create some another process or some another type of process, basically the runtime needs to be build up again.

Its a valid use case, it is highly possible that after server restart no user would be doing anything, the server would just run idly and the timer should keep on firing. Think of this timer as some kind of reminder job.

Any suggestion or workaround would be really helpful.

Thanks

Anindya

[swiderski.maciej]

I assume this is you custom application so you need to ensure that when your application starts runtime manager is initialized. You don't need to initialize any RuntimeEngine or do any operations on it, just make sure RuntimeManager is created. That will trigger anything that is active on that runtime manager.

HTH

[anindyas79]

Yes, its a custom solution. I think you mean to say something like this https://github.com/mswiderski/jbpm-examples/blob/master/jbpm6/jbpm-sample/src/main/java/org/jbpm/examples/ProcessEngineService.java

However, I am wondering if this can be done the CDI way? How do I make the application load the RuntimeManager eagerly at the start of the application. Attached are two files, one ApplicationScoped producer and the other is an EJB. I have tried injecting the RuntimeManager in ApplicationScopeProducer but it fails with recursive call error.

I have attached the two files. It would be helpful if you could take a look and suggest a way out.

Thanks

Anindya

[swiderski.maciej]

what you could try is to invoke any method on RuntimeManager in your ejb's post construct method. Like for example call runtimeManager.toString() that should enforce CDI to bootstrap all instances required by that bean. Assuming ejb beans are constructed on application startup.

Another thing I noticed is that you use bean managed transaction then you should always getRuntimeEngine and at the end disposeRuntimeEngine on runtimeManager object to clean up properly. If you would be in container managed transaction environment then runtime manager registers tx synchronizations to the clean up for you in right time whenever you call getRuntimeEngine.

HTH

[anindyas79]

OK...as you suggested I have created an EJB with @ Singleton and @ Startup annotation so that the bean starts as soon as application boots and injects the RuntimeManager. It indeed starts the Quartz scheduler. But it has the drawback I have to have an EJB as @ Singleton, I was hoping to make all EJBs @ Stateless and no @ Singleton. Now I shifted to CMT so that I dont need to dispose the runtimeengine manually. Attached is KSessionPerProcessInstanceCMTProcessBean.java, its a very simple file just calling basic APIs. But as soon as I boot up this EJB I get an unncecessary session disposed warning in the server log pls refer to server-log-1.log

The process has one start node, two human tasks node for john and mary resp. and an end node. Now after the sever boots up with this warning, I call the startProcess method twice, no issues, I can see two processes getting created. Bus as soon as I hit the approveTask method for john (basically I want to complete the two tasks for john) I get  ERROR [org.jboss.as.ejb3] (http-localhost/127.0.0.1:8080-1) javax.ejb.EJBTransactionRolledbackException: This runtime is already diposed, pls refer to server-log-2.txt

It would be really helpful if you could review the attached files. I cannot find a web application example with these different strategies. I am trying to build an webapp to showcase all the new features of jbpm6 e.g. multiple KSession strategies, timer persistence across clusters etc. The poc is based on the https://github.com/jsvitak/jbpm-6-examples/tree/master/rewards-basic I have taken that as the starting point. I am using jbpm 6.0.0.CR2

Thanks

Anindya

[anindyas79]

Hi Maciej

I debugged the CR3 version of source code and I found the reason of this bug. When I am creating the runtimeengine for the first time with an empty context we call the getRuntimeEngine(Context<?> context) method which internally calls the saveLocalRuntime(contextId, runtime); So the internal threadlocal map now has 1 entry in the map [{emptycontext: runtime@ad5379}]

public RuntimeEngine getRuntimeEngine(Context<?> context) {

        Object contextId = context.getContextId();

        KieSession ksession = null;

        Integer ksessionId = null;

        if (contextId == null || context instanceof EmptyContext ) {

            ksession = factory.newKieSession();

            ksessionId = ksession.getId();                

        } else {

            RuntimeEngine localRuntime = findLocalRuntime(contextId);

            if (localRuntime != null) {

                return localRuntime;

            }

            ksessionId = mapper.findMapping(context);

            if (ksessionId == null) {

                throw new SessionNotFoundException("No session found for context " + context.getContextId());

            }

            ksession = factory.findKieSessionById(ksessionId);

        }

        InternalTaskService internalTaskService = (InternalTaskService) taskServiceFactory.newTaskService();

        configureRuntimeOnTaskService(internalTaskService);

        RuntimeEngine runtime = new RuntimeEngineImpl(ksession, internalTaskService);

        ((RuntimeEngineImpl) runtime).setManager(this);

        registerDisposeCallback(runtime, new DisposeSessionTransactionSynchronization(this, runtime));

        registerItems(runtime);

        attachManager(runtime);

       saveLocalRuntime(contextId, runtime);

        ksession.addEventListener(new MaintainMappingListener(ksessionId, runtime));

        return runtime;

    }

protected void saveLocalRuntime(Object processInstanceId, RuntimeEngine runtime) {

        if (processInstanceId == null) {

            return;

        }

        Map<Object, RuntimeEngine> map = local.get();

        if (map == null) {

            map = new HashMap<Object, RuntimeEngine>();

            local.set(map);

        }

        map.put(processInstanceId, runtime);

    }

Now as soon as the process starts because of this event handler the saveLocalRuntime(contextId, runtime); is called again, now the contextid is the new processInstanceId. Hence in the threadlocal map we got two entries now  [{emptycontext: runtime@ad5379}, {newprocinstid: runtime@ad5379}]

@Override

        public void beforeProcessStarted(ProcessStartedEvent event) {

            mapper.saveMapping(ProcessInstanceIdContext.get(event.getProcessInstance().getId()), ksessionId); 

            saveLocalRuntime(event.getProcessInstance().getId(), runtime);

        }

Now when the runtime is getting disposed due to the transactionsynchronization the removeLocalRuntime(RuntimeEngine runtime) is called which iterates over the map values, if a match is found with the runtime then it collects the key and  breaks out of the loop. This is the problem. Because now in this map we have two keys with the same runtime and not one key. So, we cannot break here, we need to collect all the keys to be removed. Since the key "EMPTYCONTEXT" is getting removed as the runtime is diposed, next time when I call perProcessInstanceManager.getRuntimeEngine(ProcessInstanceIdContext.get(pid)); in the approve task method I get the disposed runtime from the local map and hence the bag. It should have been removed from the map and should have been fetched from the database

So, either the removeLocalRuntime is to be fixed to remove all keys corresponding to a particular runtime or the saveLocalRuntime is to be fixed i.e. override the EMPTYCONTEXT key with the new processinstanceid key after the process created successfull. I think the 1st option is safe.

protected void removeLocalRuntime(RuntimeEngine runtime) {

        Map<Object, RuntimeEngine> map = local.get();

        Object keyToRemove = -1l;

        if (map != null) {

            for (Map.Entry<Object, RuntimeEngine> entry : map.entrySet()) {

                if (runtime.equals(entry.getValue())) {

                    keyToRemove = entry.getKey();

                    break;

                }

            }

            map.remove(keyToRemove);

        }

    }

[anindyas79]

Hi Maciej

I made changes to the removeLocalRuntime as below to dispose of the multiple keys from the local map pertaining to a runtimeengine. This fixed the session already disposed error that I was getting when I called the approvetask method. Also I took the lastest code from github, I see that there are changes in the init method, that fixes the boot up session disposed warning message.

I have attached the changed file...you can see the changes marked as "Changed by Anindya". Please review.

// Changed by Anindya

    protected void removeLocalRuntime(RuntimeEngine runtime) {

        Map<Object, RuntimeEngine> map = local.get();

        List<Object> keysToRemove = new ArrayList<Object>();

        //Object keyToRemove = -1l;

        if (map != null) {

            for (Map.Entry<Object, RuntimeEngine> entry : map.entrySet()) {

                if (runtime.equals(entry.getValue())) {

keysToRemove.add(entry.getKey());// Changed by Anindya

                    //break;

                }

            }

// Changed by Anindya

            for(Object keyToRemove : keysToRemove) {

map.remove(keyToRemove);

            }

        }

    }

[swiderski.maciej]

thanks for detailed analysis. I'll be looking at this later today and let you know. If you see this as a bug please file jira issue for this so I can fix that as part of it.

Cheers

[swiderski.maciej]

alright, issue is clear now. Problem is caused by EmptyContext usage here as it does return the value for contextid and thus get's stored into local cache of runtimes. You can resolve that locally by using ProcessInstanceIdContext.get() when starting a process instead of EmptyContext.get which in fact is the recommended way. I'll fix the saveLocalRuntime to make sure only proper entires will be cached.

Thanks for reporting this!

[anindyas79]

Thank you Maciej. I went through the fix you made, downloaded the latest code, and tested. Yes, it indeed works.

[swiderski.maciej]

glad to hear that, thanks a lot for confirming this!

[aibeke]

Hi Maciej.

In our project uses a "Per-request strategy". But when the server is restarted flies timers.

After reading the discussion I realized that for an easy solution to change the strategy to "Singleton strategy" and thus solved the problem with the timers.

Question: What is affected and what are we facing? If sent to the production server, nothing will break?

[swiderski.maciej]

in general it's not recommended to use per request with timers, if you have timers in your process then use per process instance as it provides similar results when it comes to performance and isolation. Singleton is shared across all process instances of given deployment (runtime manager) and thus can lead to bottle neck with high volume of concurrent requests.

when it comes to move to production, better to separate, these that are currently running with per request, let them complete or abort them and then use new deployment with changed strategy