-
1. Re: job executor
jeffj55374 Nov 26, 2007 1:35 PM (in response to tom.baeyens)Thanks for the info, Tom.
We are currently using JBPM in a data warehousing ETL application.
Based on this experience we have discovered a number of useful enhancments that I think are general purpose in nature.
Job Executor Wish ListAbility to plug-in or in some fashion allow the application to sort the jobs and allow the application to priortize jobs. Currently jobs are executed in the order they are created. In our application it is better to select jobs based on the order in which the process instances where created so that selection is biased to run jobs associated with the oldest process instance first.
It would be nice to be able to tune the number of job executor threads dynamically while the system is running. This is useful when nodes have extensive data processing to do before moving on to the next node. i.e. an action that is processing intensive with durations in the many minutes or hours range.
Any thought to using the java.util.concurrent.ThreadPoolExecutor as the basis for the pool of JBPM JobExecutorThreads?
Have setters for all the JobExecutor variables so that Spring can be used to manage the instance.
Thanks,
Jeff -
2. Re: job executor
tom.baeyens Nov 26, 2007 4:03 PM (in response to tom.baeyens)"jeffj55374" wrote:
Job Executor Wish ListAbility to plug-in or in some fashion allow the application to sort the jobs and allow the application to priortize jobs. Currently jobs are executed in the order they are created. In our application it is better to select jobs based on the order in which the process instances where created so that selection is biased to run jobs associated with the oldest process instance first.
controlling sequence of execution is hard. especially i don't see how we could do that in a clustered/multithreaded environment
but we are grouping the queries in hibernate mapping files so that they can be customized. so if you can accomplish your use case with updating a query, then it should be easy to do so"jeffj55374" wrote:
It would be nice to be able to tune the number of job executor threads dynamically while the system is running. This is useful when nodes have extensive data processing to do before moving on to the next node. i.e. an action that is processing intensive with durations in the many minutes or hours range.
that is available on API level. just a matter of exposing it through the web console UI."jeffj55374" wrote:
Any thought to using the java.util.concurrent.ThreadPoolExecutor as the basis for the pool of JBPM JobExecutorThreads?
good suggestion. i used some classes from concurrent util, but didn't look at th eThreadPoolExecutor yet. i'll definitely look at it asap. thanks !"jeffj55374" wrote:
Have setters for all the JobExecutor variables so that Spring can be used to manage the instance.
is done in the jbpm.3 code. so that should be available in the next release. it would be good if you could double check the current HEAD of cvs if it has all the accessors that you need. -
3. Re: job executor
tom.baeyens Dec 1, 2007 6:41 AM (in response to tom.baeyens)I'm looking at the java.util.concurrent.ThreadPoolExecutor and i'm not able to get the bahaviour i want out of it.
There is one thread acquiring jobs and dispatching their jobDbids to a pool of executor threads. The dispatcher thread is the producer thread. There is a thread pool of consumers that have to take a jobDbid and execute the job.
I want a throttle control mechanism. Meaning that if all the consumers are busy, the producer thread should wait until consumer threads become available. That prevents that a lot of jobs get acquired but the dispatcher, before they can be executed by job executor threads.
The only thing that provides throttle control in the thread pool executor things is ThreadPoolExecutor.CallerRunsPolicy. But in that policy, the throttle mechanism is not good. When no threads are available, the dispatcher thread is used to execute a job. It effectively reduces throttle, but in the following situation, it doesn't use the full capacity: suppose that all 5 threads are busy when the dispatcher thread tries to offer a new job. Then the dispatcher thread will start to execute that job itself. Suppose that executing the job takes 3 seconds and that immediately after the threads were unavailable, they all finish, then the 5 threads will be idle until the dispatcher thread finishes execution of the job 3 seconds later.
It seems to me that writing a RejectedExecutionHandler that implements the desired behaviour is more complex then just using a simple SynchronousQueue and manage the thread pool myself.
Any suggestions are appreciated.