meeting context

[tom.baeyens]

Agree, it's API stuff. But I was referring to this

Questions were more about declaring vars, typing them, giving default values etc

My concern was more about granularity of the discussion. I.e. Thomas reply is more likely what I consider an appropriate level to have a discussion at the team meeting.
Does that make sense?

In the first couple of years as we build jBPM, we had a lot of "can you do this process construct?" type of questions. We realized that by overwriting the Node's execute method, we could just implement any behaviour we want in plain Java. The focus then quickly shifted to the runtime data structure (the tree of executions). As only that becomes the limiting factor of what kind of process constructs the engine can handle.We also realized that this way, we can implement any process language as it is just a set of process construct implementations.

Next came pluggable aspects. Swimlanes in jPDL don't make sense in BPEL. Even other process languages that include task management (like bonita) have other interpretations of process roles. Correlation sets only make sense in BPEL. So it must be possible to plug these kinds of aspects on top of the core execution data.

And the last factor was the different environments: BPEL runs in an ESB-like environment (based on WSDL services). jPDL runs in any kind of Java environment and wants to bind to different transaction demarcation technology. Pageflow executions must be able to persist themselves in the http session, and so on.

These 3 aspects
* node pluggability
* pluggable aspects
* embeddability
lead to the design goals of the PVM based approach to jBPM 4.

So IMO it makes sense to see if the BPMN constructs can be build on top of the PVM. But I'm pretty confident that is going to be possible. That is not where the challenge is IMO.

We'll be pursuing following process languages:
* jPDL
* BPEL (Bull)
* XPDL (Bull)
* Pageflow (SEAM)
* Thread Control Language (TODO)
* Executing BPMN directly from the STP-BPMN file format could be another potential candidate.

In the context of those target languages, I don't think that a dogmatic adoption of BPMN has the answers. We should look at the executable process parts of BPMN and see to what extend it makes sense to align jPDL with BPMN. Then we should use the BPMN figures in the designer where appropriate.

The real challenge with BPM and the process languages that we target is to identify the common parts and define how we're going to add the process language specific parts. The PVM is focussed on the common parts, but with a clear strategy on how all of those specific process language features can be build on top. In terms of the API, we are currently far enough advanced to see that a common API is possible that is shared between all these process languages.
This means one API for all kinds of workflow.

Another challenge is the impact of persistence. The more you try to hide that aspect, the more it pops up in unexpected places. Over the last years, we found out that the clue has been to align jBPM's persistence with the user's domain model persistence. That is why embeddability is such a crucial part of what we do. By making the embeddability of jBPM easy into a Java project, we make our own market. A real discussion point in this sense could be JPA. As more projects start with non-hibernate based JPA persistence, we loose a big part of our embeddability.

[heiko.braun]

These 3 aspects
* node pluggability
* pluggable aspects
* embeddability
lead to the design goals of the PVM based approach to jBPM 4.

Good point. Let's talk about design goals, but in the context of the upcoming meeting. AFAIK the meeting is about the API and CTS, right?
Then we should put the focus on questions related to that domain. Thomas came up with a list of basic concepts that he would like to see addressed in the API:

- ProcessEngine, Process, ProcessDefinition
- StartEvent (None, Signal, Message)
- Task (None, Send, Receive)
- EndEvent (None, Signal, Message)
- Gateway (Inclusive, Exclusive, Parallel)
- Process, Activity Properties
- Process, Activity Assignments
- Signal, Message
- SequenceFlow

This however is not married to BPMN it merely borrows terminology. Why?
Becasue BPMN terminology is very precise and avoids redundancy.

All of these items do have associated semantics and it should be our first task to make sure everyone has same understanding of what this actually means. This will predict the implementation that follows.

Starting off with the jbpm4 code base and API doesn't work because a successful discussion with all participants requires a certain level of abstraction. We need to talk about what should be there instead of what is possible with the current implementation.

Once we agree on how these basic concepts relate to each other
(and I hardly believe that we manage to so in 3 days) then we could move on to topics that build upon that:

Process dialects
- handling multiple process dialects
- extending process dialect elements
- extending the core engine capabilities

Embeddability
- transactions
- persistence
- security

[tom.baeyens]

AFAIK the meeting is about the API and CTS, right?

That is a part of it. The first goal is to synchronize the whole team on the goals and strategy of jBPM 4 so that we're all aiming for the same thing.

- ProcessEngine, Process, ProcessDefinition
- StartEvent (None, Signal, Message)
- Task (None, Send, Receive)
- EndEvent (None, Signal, Message)
- Gateway (Inclusive, Exclusive, Parallel)
- Process, Activity Properties
- Process, Activity Assignments
- Signal, Message
- SequenceFlow

We can compare those concepts to jPDL, see how it is different. If the BPMN concept is different, we can see to what extend we want to support it in the jPDL language.

This however is not married to BPMN it merely borrows terminology. Why?
Becasue BPMN terminology is very precise and avoids redundancy.

Existing jBPM/jPDL terminology is even more precise as that is executable. BPMN terminology is only described in words in the spec.

All of these items do have associated semantics and it should be our first task to make sure everyone has same understanding of what this actually means.

The whole team already knows the exact semantics of jPDL. We should go over the BPMN concepts and see how we want to support this in jPDL.

Starting off with the jbpm4 code base and API doesn't work because a successful discussion with all participants requires a certain level of abstraction.

i don't get this argument.

We need to talk about what should be there instead of what is possible with the current implementation.

agreed!

Once we agree on how these basic concepts relate to each other
(and I hardly believe that we manage to so in 3 days) then we could move on to topics that build upon that:

Process dialects
- handling multiple process dialects
- extending process dialect elements
- extending the core engine capabilities

Embeddability
- transactions
- persistence
- security

I don't see how we can split the discussion like that.


I realize that for us, jBPM 3 to jBPM 4 is in incremental change, whereas for you and Thomas it is a lot of concepts and design decisions in one package. Every decision can be challenged. Even the decisions that potentially take away our foundations and cause us to review all the design decisions again. So far I have seen a lot of new and good ideas, but none of those challenge the basic fundamentals of the approach we took.

So starting jBPM 4 development from scratch is something that I do not consider opportune. This can, of course, change when our approach is challenged (technically, conceptually or in any other way).

[camunda]

For me, the most important point is really to sync everybody, so that we know in which direction we are heading (with jBPM4, PVM, BPEL, API-Proposal, BPMN, ...).

And I would like to discuss some of the proposals made here. For me, the strength of jBPM is simplicity, but beeing powerful at the same time. A lot of projects I have been in or at least know embed jBPM in their architectures/applications. For me it is crucial to conserve this! Some discussions or proposals in the direction BPMN/API were too abstract and theoretical in my opinion. jBPM has a long history, a bunch of experience and had some nice pragmatical approaches. That should be considered when discussing jBPM 4. There is a reason why the people like jBPM :-)

BPMN is indeed an important notation. But I would see the way to go more in a BPMN -> jPDL or a BPMN -> PVM mapping. The latter may be already close to an Executable BPMN. By the way: Executable BPMN is still heavily under research and not yet usable in practice. Good to keep up to date, but really rely on it completely is not the right track in my eyes. And at least it should be done during a real life project, not developed in the lab! My vote: The core engine should kept a simple state machine as it is now.

And I think these thoughts, where I know we have different opinions on the table ;-), need to be discussed first in the meeting, since that's the basis for everything else!

[kukeltje]

AFAIK the meeting is about the API and CTS, right?

That is a part of it. The first goal is to synchronize the whole team on the goals and strategy of jBPM 4 so that we're all aiming for the same thing.

Yes, the api(s) (and cts) should be the result of this

But regarding the supported languages... xpdl is with bull, as is bpel. They already decided on their own api correct? Mainly for backwards compatibility if i'm correct (and maybe the timing). How much are we going to take those into account?

- ProcessEngine, Process, ProcessDefinition
- StartEvent (None, Signal, Message)
- Task (None, Send, Receive)
- EndEvent (None, Signal, Message)
- Gateway (Inclusive, Exclusive, Parallel)
- Process, Activity Properties
- Process, Activity Assignments
- Signal, Message
- SequenceFlow

We can compare those concepts to jPDL, see how it is different. If the BPMN concept is different, we can see to what extend we want to support it in the jPDL language.

Jeff deLong already did some work. I suggest that that is "verplicht leesvoer" (required reading material) for the meeting. But also the other way around. jPDL might have concepts that do not fit in BPMN. Extending it might be needed then.

I'm currently also looking at the bpmn spec in more detail then I ever did and I more and more come to the conclusing that if we focus on the basic concepts to much, we might have to change the api again if more complex functionality is needed (intermediate events? and the corresponding triggers) but if someone proves me wrong, I'll be the first to admit that

This however is not married to BPMN it merely borrows terminology. Why?
Becasue BPMN terminology is very precise and avoids redundancy.

Existing jBPM/jPDL terminology is even more precise as that is executable. BPMN terminology is only described in words in the spec.

http://www.brsilver.com/wordpress/subscribers-only-2/three-levels-of-process-modeling-with-bpmn/

See level 3

Once we agree on how these basic concepts relate to each other
(and I hardly believe that we manage to so in 3 days) then we could move on to topics that build upon that:

Process dialects
- handling multiple process dialects
- extending process dialect elements
- extending the core engine capabilities

Embeddability
- transactions
- persistence
- security

I don't see how we can split the discussion like that.

Me neither, can you elaborate?

One thing I have not heard people mention but what is a major issue (imo) is backwards compatibility or processdefinition upgrades etc (do we discuss those as well?).

[kukeltje]

I second Bernd... with an addition

That does not mean we cannot leverage some of bpmn
- terminology (hmmm)
- different out of the box supported gateways (nothing more then jpdl3++)
- supporting cancelling
- graphical notations (visibility of timers, cancellation, events)

And if we agree on those, we can see how they best fit in an api

[camunda]

Jeff deLong already did some work. I suggest that that is "verplicht leesvoer" (required reading material) for the meeting.

Can you send a link what you mean exactly? Thanks...

[kukeltje]

He did some research info mapping bpmn onto jpdl. I thought you already received this pdf but i'll forward it this afternoon.

[kukeltje]

Info should ben into, stupid iPod touch completion

[heiko.braun]

... can you elaborate?

Let's take a different angle: This is not about the process definition language. (It's going to be pluggable anyway, right?)
Building an API doesn't necessarily mean throwing away what's there, it could as well mean refactoring the existing one.

The API will have core elements and terminology. Each core elements has associated semantics which lead to implementation detail. We want the API to be correct, easy to use and easy to learn.

The process of finding these core elements (or core concepts) and establishing names for them is important to get one step closer to 'correctness'.

The well known list of 'basic concepts', which Thomas suggested, is merely a collection of core elements that will lead to both a meaningful set of API that can represent of large number of use cases to begin with, but at the same time covers critical areas of the engine implementation. At least that's the idea. If the list is incomplete or the terminology is somehow misleading then make suggestions how it could be changed.

Again this doesn't necessarily mean we ditch the existing jBPM4 code base, it's merely a process of verification.
If the current code base would already have a precise and 'to the point' API, then we wouldn't need to do it.

Here's how I can picture the meeting:
We go through the list of core elements and for each of those we'll first discuss correctness and importance of that element in a BPM world and then secondly match it with jBPM4 elements. Either the BPMN expression is wrong to begin with or unnecessary for jBPM4 (no use case) then we can ditch it right away. But if it turns out to be important then it will either have a representation in jBPM4, just differently named or it requires changes to the code base.

But we'll start with a small set, and then gradually increase to the API until it is feature complete. In the beginning, the API would not allow you todo everything jBPM4 is currently capable of, but sooner or later it will.

[kukeltje]

Let's take a different angle: This is not about the process definition language. (It's going to be pluggable anyway, right?)

Ok, maybe we already look at it differently here. Executing process languages is the thing that jBPM (ok PVM) does for you and supporting that should be exposed by the API.

Pluggable yes, but you cannot drop-in replace jpdl for bpel or xpdl :-) and in each of these executable languages you can do things you cannot do with the other.

But instead of going into details, I'll read on first, maybe we agree more than I upfront think ;-)

Building an API doesn't necessarily mean throwing away what's there, it could as well mean refactoring the existing one.

The API will have core elements and terminology. Each core elements has associated semantics which lead to implementation detail. We want the API to be correct, easy to use and easy to learn.

The process of finding these core elements (or core concepts) and establishing names for them is important to get one step closer to 'correctness'.

Establishing names... great (read my lips: no-more-mapping).
finding common elements or more complex elements (like different gateways) and use those in jbpm... ehh... jpdl, great, but that is process language. The gateway (decision in jpdl) is not something I've ever used from 'the' api...

Basic things I did do with 'the api' are

- start a process
- end a process
- signal process to continue (either via default or supplied transition)
- set a variable
- read variable
- get a tasklist
- end a task
- get basic properties of a task/node/.... in a process instance (e.g. task.hasEnded())
- ...

Although... working on tasks is already more complex. In jpdl you can have variables with a task scope, or on the process level. It has to be clear how to do that and should be possible via an api. In howfar this is possible in all process languages I'm not sure, so should this be generic? Or put in a more process language specific api, e.g. on that extends the basic api?

The well known list of 'basic concepts', which Thomas suggested, is merely a collection of core elements that will lead to both a meaningful set of API that can represent of large number of use cases to begin with, but at the same time covers critical areas of the engine implementation. At least that's the idea.

You mean language implementation instead of engine implementation right? To me an 'engine' is the implementation of a language on the pvm. Regarding the number of use cases I wonder if this is from a theoretical of practical view. As Bernd points out the flexibility of jBPM/JPDL and the way you can use the current api is one of its usp`s

If the list is incomplete or the terminology is somehow misleading then make suggestions how it could be changed.

Terminology is imo certainly not misleading, but the collection of concepts is, well not misleading, but maybe to theoretical, or as mentioned earlier something that belongs in a process language

Again this doesn't necessarily mean we ditch the existing jBPM4 code base, it's merely a process of verification.

That is what, to a great extend, has been done the last years, hence we are now where we are with the api and language constructs. Verification against real live usecases and not just theoretical usecases (workflow patterns) or a non-excutable, visual definition (bpmn)

That BPMN is eventually becoming more popular, the alignment of BPMN 2.0 with XPDL and a usable (free) editor, the time is now indeed right to leverage it more, but *without* losing the flexibility.

If the current code base would already have a precise and 'to the point' API, then we wouldn't need to do it.

The 'current, 3.x' api has grown, and once you know it (it is not hard to learn if you use it) is not to bad. It's not as to the point as it could be but that is something Tom was/is working on for jBPM4 (afaik) before this common api discussion started.

Here's how I can picture the meeting:
We go through the list of core elements and for each of those we'll first discuss correctness and importance of that element in a BPM world and then secondly match it with jBPM4 elements. Either the BPMN expression is wrong to begin with or unnecessary for jBPM4 (no use case) then we can ditch it right away. But if it turns out to be important then it will either have a representation in jBPM4, just differently named or it requires changes to the code base.

Now I'm kind of lost... you start with saying it is about the api and not the process language, and now it is mostly the language (elements are for me language constructs)

What I think is more important is what we want to do with the (basic) api. Three things (quickly from my head, not complete maybe)

- work on individual process instances

- signalling
- setting properties
- reading properties (

- working on (a set) tasks

- getting a tasklist
- ending a task
- setting variables/propperties/...

- advanced things

- get a tasknode in a process-definition and create a task on it
- ending specific tasks on other nodes if something happens
- ...

Why these advanced things? Because up to now you could not do everything in the jpdl language (but also were not limited to it!!!!) but had to write some small (often reusable) javacode to achieve something (see my milestones blog).

Agreeing on a basic api for the first two items will not be hard at all with using BPMN terminology. Extending the language or providing additional node implementations via javacode so more bpmn elements are provided out of the box is also something we can agree on

The third item has a lot to do with the process language. In my milestone example I explicitly create a task from another node. In another upcomming example, I send the other node an event/signal/trigger (yes, we should standardize here ;-)) so it can create the task itself. Probably fits better in BPMN, but are these kinds of events supported? Currently looking into that.

Maybe we should have two api's per 'language', one basic that is common across process/workflow languages and an advanced one specifically for a language. For jpdl, the advanced api can be the jbpm api that is currently there. I'm just not sure though how pageflow, threadcontrol will fit in thebasic api (terminology wise) if that is a goal at all.

But we'll start with a small set, and then gradually increase to the API until it is feature complete. In the beginning, the API would not allow you todo everything jBPM4 is currently capable of, but sooner or later it will.

Two things:
- Feature complete by the GA release? That is sooner than later.
- jBPM4 currently is not capable of 'anything' (ok exaggerated a little). The (new(er)!!) api`s are (were?) under development, JPDL4 is under development (maybe new language constructs like support for different gateways out of the box)

Two things I cannot stress enough:
- backwards compatibility needs to be there....
- do not underestimate the current flexibility of jBPM (jpdl) and how it is used in other products

Good discussion btw... lets keep it going, maybe the november meeting will be short then and we'll have lots of time to get used to kasteelbier ;-)

[heiko.braun]

Now I'm kind of lost... you start with saying it is about the api and not the process language, and now it is mostly the language (elements are for me language constructs)

Interesting question. Maybe we should elaborate on the question how far the actual PDL influences the design of the API. This will probably reveal that different API's are required:

- One for building a process model from XML descriptors
(Similar to what's in the PVM already)
- One for invoking on it (Client API)
- One for extending the engine capabilities (i.e. new node types)
- One to solve integration problems (TX, persistence, etc)

But to answer your question, I'd say that core elements are not language constructs. They are building blocks that predict core runtime capabilities, independent of the actual PDL.

Let's look at a Parallel Gateway for example. To me, this is a core element. But it's semantics are not clear. Does a parallel split require a subsequent join or can both path of execution end differently? The answer to this question has a technical impact on the implementation (i.e. threading, transactions) and thus an impact on the API.

Why an impact on the API? Because if we chose real parallel execution (open graph, independent path's of execution) then client 'signal' (jBPM3) doesn't make sense anymore. Actually it wouldn't even work because the client will not be in control of execution anymore.

I believe that there is a small set of core elements, which are not related to the final PDL that, once we got it right, can be used to assemble concrete engines (jBPM4+jpdl) leveraging building blocks.

[kukeltje]

Now I'm kind of lost... you start with saying it is about the api and not the process language, and now it is mostly the language (elements are for me language constructs)

Interesting question. Maybe we should elaborate on the question how far the actual PDL influences the design of the API. This will probably reveal that different API's are required:

- One for building a process model from XML descriptors
(Similar to what's in the PVM already)
- One for invoking on it (Client API)
- One for extending the engine capabilities (i.e. new node types)
- One to solve integration problems (TX, persistence, etc)

+10
I like the use of jaxb as Thomas is doing. Not fully sure why, but it feels right.

But to answer your question, I'd say that core elements are not language constructs. They are building blocks that predict core runtime capabilities, independent of the actual PDL.

To some extend you are correct, but if you search the net for problems regarding mapping bpmn to bpel or xpdl (improved with bpmn 2.0 and xpdl 2.0) it is not that easy and so they are not independent.

Let's look at a Parallel Gateway for example. To me, this is a core element. But it's semantics are not clear. Does a parallel split require a subsequent join or can both path of execution end differently?

Afaik, but could not find it directly, BPMN is based on structured graphs so a split requires a corresponding join.

In JPDL 3 it is fully clear, you need a fully nested fork-join pair. This differs from the statement in the workflow patterns for parallel split:

"www.workflowpatterns.com" wrote:
These branches may or may not be re-synchronized at some future time.

The number of cases that I've came across in the forum where people did

 fork 1
 / \
 | fork 2
 | / \
 join 1 |
 \ /
 join 2

Are rare and could in most cases simply be modeled in another way. The reason jpdl requires this are simple. The implementation of a join is more simple then. This has been discussed several years ago, so unless bpmn requires otherwise, the decision for this is simple I think ;-)

The answer to this question has a technical impact on the implementation (i.e. threading, transactions) and thus an impact on the API.

It indeed influences the transactions (see the issues with the join in jira) and to a lesser extend threading (it was always stated that jBPM was not a thread-control-engine), therefore the async functionality to leverage e.g. jms

Why an impact on the API? Because if we chose real parallel execution (open graph, independent path's of execution) then client 'signal' (jBPM3) doesn't make sense anymore. Actually it wouldn't even work because the client will not be in control of execution anymore.

For that one parallel gateway you are correct, but again, the fork is not something that I ever signalled from the client api, so no change in that area. For each of the legs in a fork (weren't they called teeth ;-)) there can be wait-states again (activities) that need to be signalled

I believe that there is a small set of core elements, which are not related to the final PDL that, once we got it right, can be used to assemble concrete engines (jBPM4+jpdl) leveraging building blocks.

But those are building blocks behind the scenes, reusable to create a new PDL. Sounds nice, JPDL and e.g. an executable BPMN can leverage both, but that is not related (as you also state above, right?) to the client-api and imo that should be the most stable, simple to use etc...

Great.... I get the feeling we are getting somewhere

[kukeltje]

"kukeltje" wrote:

Now I'm kind of lost... you start with saying it is about the api and not the process language, and now it is mostly the language (elements are for me language constructs)

Interesting question. Maybe we should elaborate on the question how far the actual PDL influences the design of the API. This will probably reveal that different API's are required:

- One for building a process model from XML descriptors
(Similar to what's in the PVM already)
- One for invoking on it (Client API)
- One for extending the engine capabilities (i.e. new node types)
- One to solve integration problems (TX, persistence, etc)

+10
I like the use of jaxb as Thomas is doing. Not fully sure why, but it feels right.

But to answer your question, I'd say that core elements are not language constructs. They are building blocks that predict core runtime capabilities, independent of the actual PDL.

To some extend you are correct, but if you search the net for problems regarding mapping bpmn to bpel or xpdl (improved with bpmn 2.0 and xpdl 2.0) it is not that easy and so they are not independent.

Oops, forgot to correct this statement after reading (and answering) the last statement in your previous post.

I tend to agree with your statement, it all came down to terminology again ;-)

Implementing an xml based language is then creating a xml binding and in that re-use or extend the building blocks (core elements, base classes for activities in the pvm) to use the pvm api

[heiko.braun]

Afaik, but could not find it directly, BPMN is based on structured graphs

Well, we just borrow the terminology (i.e. parallel gateway) and need think about what it means for the implementation. This is exactly the kind discussion I'd like to have.

In JPDL 3 it is fully clear, you need a fully nested fork-join pair. This differs from the statement in the workflow patterns for parallel split:

Exactly, hence the need for verification. What you said about previous discussions is something I don't agree with at all:

This has been discussed several years ago, so unless bpmn requires otherwise, the decision for this is simple I think ;-)

Let's put it the other way around: Why do all the vendors support workflowpatterns.com [1] and BPMN [2] opposed to jbpm?

[1] http://www.workflowpatterns.com/vendors/
[2] http://www.bpmn.org/BPMN_Supporters.htm#current