I was writing in an email to someone how the job executor worked. So I thought i leverage that effort in the docs and here in the forum.
Here's how the job executor works in a nutshell:
Jobs are records in the database. Jobs are commands and can be executed. Both timers and async messages are jobs. For async messages, the dueDate is simply set to now when they are inserted. The job executor must execute the jobs. This is done in 2 phases: 1) a job executor thread must acquire a job and 2) the thread that acquired the job must execute it.
Acquiring a job and executing the job are done in 2 separate transactions. A thread acquires a job by putting its name into the owner field of the job. Each thread has a unique name based on ip-address and sequence number. Hibernate's optimistic locking is enabled on Job-objects. So if 2 threads try to acquire a job concurrently, one of them will get a StaleObjectException and rollback. Only the first one will succeed. The thread that succeeds in acquiring a job is now responsible for executing it in a separate transaction.
A thread could die inbetween acquisition and execution of a job. For clean-up of those situations, there is 1 lock-monitor thread per job executor that checks the lock times. Jobs that are locked for more then 30 mins (by default) will be unlocked so that they can be executed by another job.
The required isolation level should be set to REPEATABLE_READ for hibernate's optimistic locking to work correctly. That isolation level will guarantee that
update JBPM_JOB job set job.version = 2 job.lockOwner = '192.168.1.3:2' where job.version = 1
will only return result 1 row updated in exactly 1 of the competing transactions.
Non-Repeatable Reads means that the following anomaly can happen: A transaction re-reads data it has previously read and finds that data has been modified by another transaction, one that has been committed since the transaction's previous read.
Non-Repeatable reads are a problem for optimistic locking and therefor, isolation level READ_COMMITTED is required if you configure more then 1 job executor thread.