Just updating with what I am doing about this.
Looking into the problem, there are really 2 problems I am trying to resolve:
1. Creating new transaction on activities
2. Customize exception handling
Creating new transaction per activity achieved by setting continue="async" on the activity. The only problem with it is if you have a fork, and than activities within the fork has continue="async", sometimes execution will get stuck at join. There is obviously some sort of race condition there, and, when this happens, in the database I can see both lines of executions after fork being stuck with "inactive-join" state.
Customizing exception handling. Two choices here, one being JBPM native - setting parameters for retry interceptor tag. Unfortunately it does not work for me for the following requirements:
1. Schedule of retries for me is different from formula JBPM provides
2. I should be able to manually retry failed activity per user request
3. I should be able to ignore failed activity (let execution to proceed) per user request
So, I made my activities implement ExternalActivityBehaviour. In the case of exception it does not propagate exception to JBPM, but calls "waitForSignal", than creates persisted Quartz trigger to send "retry" signal to itself in calculated interval. In "signal" method if signal is "retry" it will redo the activity, if signal is "ignore" it takes default transition. I can probably make it more generic by actually providing the name of transition to take, but I don't have any use for it now.
I still need to add some synchronization magic between executing failed activities by trigger and by user request.