Design Time Governance - Workflow Triggers from events vs. queries

[eric.wittmann]

The current version of Design Time Governance (version 1.3.0) continues to use polling of the S-RAMP repository to find artifacts to govern.  The way this works is that the user/admin of dtgov must configure a set of Workflow Triggers in the dtgov UI.  Each trigger is made up of the following meta-data:

• Name
• Description
• S-RAMP Query
• Workflow ID
• Workflow Properties

A typical example of a Workflow Trigger might be:

• Name:  Web App Deployment Lifecycle
• Description:  Lorem ipsum... etc
• S-RAMP Query:  /s-ramp/ext/JavaWebApplication
• Workflow ID:  org.overlord.dtgov.DeploymentLifecycle
• Workflow Properties: 
  • DeploymentUrl:  ...
  • NotificationUrl:  ...

This simply means that whenever an artifact matches the query, a new instance of the DeploymentLifecycle workflow will be created.

For a future version of Design Time Governance (dtgov 2.0) we are moving away from this polling approach and embracing an event approach.  The S-RAMP repository (and other interesting sources) will fire events that DTGov will listen to.  DTGov can respond in various ways - one of these is to create a workflow.

As a bridge to get from where we are to where we are going, we are considering replacing just the polling mechanism but keeping everything else the same.  This means that instead of performing queries we will instead need to catch events and respond appropriately, even though the *configuration* of the Workflow Triggers remains query-based.  The consequence of this is that we will need a way to determine whether a given artifact *would have been returned* as part of the result set of a given query.  In other words we need to be able to answer the question:  Does this artifact match this query?

Does this seem like a valuable capability for S-RAMP to have generally?  Is this a reasonable direction to take in the short term?

[objectiser]

A couple of solutions come to mind, to help transition:

1) This one depends on how the events are produced - if SRAMP still needs to issue a query, to create the events, before they are pushed out - then the source query expression could be included in the event definition and match with the query expression configured in the DTGov event receiver. However this means that both SRAMP and DTGov need to be configured with matching queries.

2) If change events are just pushed from SRAMP, with no concept of being pushed as part of a 'category' (i.e. query expression), then I think the DTGov query becomes a predicate over the event, used to determine if it triggers the workflow. So the event definition would include some standard fields, but also adhoc properties, and the query field would define an expression that would access that information to provide a boolean result.

My current preference is (2) - if this was implemented, then sramp query expressions would imply polling (so the existing mechanism could remain supported for a while), otherwise it will be a locally evaluated expression applied to the received event? Although adding a 'trigger type' field may make things more explicit.

[eric.wittmann]

brmeyer and I were discussing this more on IRC today and I think I have another transitional approach we could consider.  DTGov could do this whenever an event was received:

Get the UUID of the artifact from the event

Execute each configured workflow trigger query, but augment the query's predicate with "@uuid = UUID"

We could probably do some basic initial matching of the event to the query (e.g. if the query includes a model and/or type we can reject a query based on that information from the event).

For example, if the s-ramp query for a given workflow trigger is configured as:

/s-ramp/ext/JavaWebApplication

Then we would augment that to become:

/s-ramp/ext/JavaWebApplication[@uuid = 'UUID-GOES-HERE']

Another example - if the query is more complex such as:

/s-ramp/ext/JavaWebApplication[@customProperty1 = 'some-value' or @customProperty2Exists]

Then it would become:

/s-ramp/ext/JavaWebApplication[(@customProperty1 = 'some-value' or @customProperty2Exists) and @uuid = 'UUID-GOES-HERE']

We can actually do this kind of query manipulation easily enough using the s-ramp xpath query parser.  I already have POC code that does this.

This would make the queries very targeted (going after a single artifact by its UUID) and very fast.  If the artifact matched the query, we would then issue a followup query to see if a workflow had already been created.  But that query should also be very fast and targeted.

I think this improves the scalability a lot and we should consider getting this into DTGov 1.3.