job executor

[tom.baeyens]

FYI: i've started on the new job executor implementation based on Guillaume's ideas. just so that you know what i'm working on. i don't know yet when the first commit of this will occur.

each message or timer will result in a job being inserted in the job table. this will be done in the default implementations of the message service and the timer service

the job executor will manage 1 dispatcher thread and a nuber of job executor threads.

the dispatcher thread will scan the job table and lock one (or more in case of exclusive) jobs. then the job ids will be handed over to one of the threads in the job executor thread pool.

when a job is acquired, the lock timeout will be calculated and stored in the job record. that way, the dispatcher thread can look for all unacquired jobs or jobs for which the lock time has expired. default lock time is set to 30 mins.

the dispatcher thread also will have a backoff policy in case of exceptions. meaning if an exception occurs during job locking (most likely unavailability of the db or an optimistic locking failure in a cluster) then the dispatcher thread will use incremental delays inbetween attempts until a certain maximum. so the dispatcher thread will check for jobs every 10 seconds, but in case of repeating exceptions, this will back off to e.g. one attempt every 5 mins.

[jeffj55374]

Thanks for the info, Tom.

We are currently using JBPM in a data warehousing ETL application.
Based on this experience we have discovered a number of useful enhancments that I think are general purpose in nature.

Job Executor Wish List

Ability to plug-in or in some fashion allow the application to sort the jobs and allow the application to priortize jobs. Currently jobs are executed in the order they are created. In our application it is better to select jobs based on the order in which the process instances where created so that selection is biased to run jobs associated with the oldest process instance first.

It would be nice to be able to tune the number of job executor threads dynamically while the system is running. This is useful when nodes have extensive data processing to do before moving on to the next node. i.e. an action that is processing intensive with durations in the many minutes or hours range.

Any thought to using the java.util.concurrent.ThreadPoolExecutor as the basis for the pool of JBPM JobExecutorThreads?

Have setters for all the JobExecutor variables so that Spring can be used to manage the instance.

Thanks,
Jeff

[tom.baeyens]

"jeffj55374" wrote:

Job Executor Wish List

Ability to plug-in or in some fashion allow the application to sort the jobs and allow the application to priortize jobs. Currently jobs are executed in the order they are created. In our application it is better to select jobs based on the order in which the process instances where created so that selection is biased to run jobs associated with the oldest process instance first.

controlling sequence of execution is hard. especially i don't see how we could do that in a clustered/multithreaded environment

but we are grouping the queries in hibernate mapping files so that they can be customized. so if you can accomplish your use case with updating a query, then it should be easy to do so

"jeffj55374" wrote:

It would be nice to be able to tune the number of job executor threads dynamically while the system is running. This is useful when nodes have extensive data processing to do before moving on to the next node. i.e. an action that is processing intensive with durations in the many minutes or hours range.

that is available on API level. just a matter of exposing it through the web console UI.

"jeffj55374" wrote:

Any thought to using the java.util.concurrent.ThreadPoolExecutor as the basis for the pool of JBPM JobExecutorThreads?

good suggestion. i used some classes from concurrent util, but didn't look at th eThreadPoolExecutor yet. i'll definitely look at it asap. thanks !

"jeffj55374" wrote:

Have setters for all the JobExecutor variables so that Spring can be used to manage the instance.

is done in the jbpm.3 code. so that should be available in the next release. it would be good if you could double check the current HEAD of cvs if it has all the accessors that you need.

[tom.baeyens]

I'm looking at the java.util.concurrent.ThreadPoolExecutor and i'm not able to get the bahaviour i want out of it.

There is one thread acquiring jobs and dispatching their jobDbids to a pool of executor threads. The dispatcher thread is the producer thread. There is a thread pool of consumers that have to take a jobDbid and execute the job.

I want a throttle control mechanism. Meaning that if all the consumers are busy, the producer thread should wait until consumer threads become available. That prevents that a lot of jobs get acquired but the dispatcher, before they can be executed by job executor threads.

The only thing that provides throttle control in the thread pool executor things is ThreadPoolExecutor.CallerRunsPolicy. But in that policy, the throttle mechanism is not good. When no threads are available, the dispatcher thread is used to execute a job. It effectively reduces throttle, but in the following situation, it doesn't use the full capacity: suppose that all 5 threads are busy when the dispatcher thread tries to offer a new job. Then the dispatcher thread will start to execute that job itself. Suppose that executing the job takes 3 seconds and that immediately after the threads were unavailable, they all finish, then the 5 threads will be idle until the dispatcher thread finishes execution of the job 3 seconds later.

It seems to me that writing a RejectedExecutionHandler that implements the desired behaviour is more complex then just using a simple SynchronousQueue and manage the thread pool myself.

Any suggestions are appreciated.