Proposal for DTGov 2.0

[eric.wittmann]

We have struggled a bit to define precisely what DTGov is and what we want it to be.  Currently the project allows users to create instances of custom BPMN2 workflows whenever an artifact in the S-RAMP repository is added or changed in some definable way.  The only vertical solution atop this concept currently supported out of the box is Deployment Lifecycle Management.

This approach is fine, as far as it goes.  However, it's not clear that there is a tremendous amount of value provided by the lifecycle management workflow, given other systems designed specifically for this space.

Going forward, we are considering how DTGov might be broadened and focused at the same time.  HA!  What do I mean by that?  Well, we would like DTGov to be more broad in the sense that it can "consume" events from more systems than simply S-RAMP.  So we would like to be able to listen for changes in other systems like bug tracking systems, source code versioning systems, etc.  On the other hand, we are considering constraining DTGov's functional responsibility to focus on the definition and evaluation of design time policies.  So rather than simply triggering an arbitrary workflow whenever something changes, instead we want to have a formal definition of a Design Time Policy.  The UI can then be focused on how to define these policies and then monitor how they are evaluated and applied.

Hopefully the following information will make this more clear.  Any thoughts/feedback always encouraged and welcome!

Terms & Definitions

• Policy Definition: the data model used to define a design time governance policy.  The definition must include information about the following concepts:
• Policy Information Point (PIP): a configured design time governance event source.  Examples include S-RAMP, git, JIRA, etc.  These are read-only sources of governance information.
• Policy Decision Point (PDP): some piece of logic that determines whether the policy has been violated.  Information from the PIP(s), along with other external sources, are used to make this determination.
• Policy Enforcement Point (PEP): a configured endpoint that can be used to affect some sort of governance change.  For example, cancelling a git Pull Request would be considered a policy enforcement point.
• Policy Notification Channels: configured notification channels - sends policy violation information to users.  Examples include twitter, email, SMS, etc.
• Policy Actions: set of asynchronous actions to take when a policy is violated.
• Policy Waivers: a role could be provided allowing someone a chance to waive the policy for this particular instance/violation.

Concepts

• Define governance event sources (JIRA, git, S-RAMP, etc) as the Policy Information Points.
• Actions can be any sort of asynchronous thing - workflow, script, etc…
• Notifications is a top level concept, along with Actions.
• Correlation - we need a way to correlate individual policy violations together into sensible groups (e.g. the “Project” concept already suggested).  This way users can see a stream of related policy violations.
• Wiring together disparate endpoints is easy enough.  But the gap that DTGov fills is the context and interactions useful for Governance

Tooling

• UI overlay on top of the endpoints to create Policy Definitions.
  • Select one or more PIP, with a “selector” of some kind
  • How to use multiple related PIPs?  CEP of some kind?
  • Define a set of Policy Notifications
    • Support for multiple notification channels (email, SMS, twitter)
    • Template based?  How to handle different channel limitations/formats?
  • Define a set of Actions
    • What information is available to be passed to the action(s)?
    • Actions must be async - this is currently handled poorly in DTGov 1.0.
• The UI should be able to display a stream of governance activity with various filters, including at least “project” and “type”.
• Admin user responsible for defining/designing the policies
  • All aspects of each policy definition should be configurable via the UI (see Concepts above)
  • Policies are named for easy consumption in projects (by non-admin users)
• Business users will create Projects, define roles, choose/configure policies which should be evaluated for a project
• Nested projects - allow projects to be nested, allowing this simple concept to model more complicated project configurations.

Technologies

• Camel: Message transformation and routing framework with a large variety of endpoint components (http://camel.apache.org/components.html).
  • Event consumers
  • Notification producers
  • “Action” producers
  • Existing components to use:
    • Amazon-*
    • Dropbox
    • File
    • FTP, FTPS, SFTP
    • GMail
    • SMTP
    • IRC
    • Twitter
  • New components to be created and contributed back to Camel:
    • GitHub
    • JIRA

[jorgemoralespou_2]

Hi,

I'm happy to hear that DTGov is still moving. Regarding technologies, I have always thought that, if some of the concepts that you present here in this document can be solved using "camel" for EIP, rules/cep for evaluating the policies and jBPM was already used to some sort of governance for deployment and integrations with S-RAMP, why not the core could be "SwitchYard"?

I know that this is a very big thing, but maybe makes sense as DTGov and SwitchYard right now can go very much by the hand. (Just a thought!!!)

I like the ideas that you present here, and probably, with more use cases description will be easier to see what are the best fit for each and every part.

One comment, though, is that I do not like that a PDP determines if a policy is being "violated" (this sounds very much like an error), but rather "activated", "triggered" or any other non similar word. Just a matter of taste :-D

[objectiser]

Hi Jorge

I'm not sure what value using Switchyard as the core would have? Switchyard adds a 'services' layer above camel, to leverage its connectors and components, and we would be looking to provide a governance layer that does the same. Could you provide an example of how Switchyard would be of use?

I agree that DTGov and SwitchYard will be used together - but equally DTGov could be used with many other runtime environments.

I also agree with your last point - I don't think we should actually pre-judge what a policy is doing - and therefore not just determine if some condition has been violated or not. Possibly it would be better to use a similar concept to Eclipse markers - when a policy is triggered, it will perform its task and the outcome of that will potentially be to manipulate markers (or issues) associated with the artifact and event source, and recorded against the project. These markers would have a type, severity, priority, etc. The policies could create new markers, read existing ones to update them or remove them if the policy determines they are no longer relevant.

[jorgemoralespou_2]

Hi Gary,

My point in using SwitchYard is in the integration it provides with rules and bpm processes for the creation of rules/workflows for governance. I know it is like using a sledgehammer to crack a nut, but my point was on that I think that using Workflows for some decisions, and rules for others is good, and AFAIK camel does not integrate well with this. But I'm not an expert. And I meant not to expose SwitchYard to users, but to leverage SwitchYard as the runtime engine for the services, and maybe creating this type of governance services on top of SwitchYard. But yeas, you have the point on lightweightness and dependencies, and to be able to run DTGov with independence of any runtime, so a way should be embedding a library, so camel seems a good option.

I tend to think that we should limit very much what is exposed to the clients, providing meaningful data model, and good APIs and utilities, so they do not need to go deeper than expected. At the end, if the tool is too difficult to be used, it won't be used (at least not by so many people). There lays the success of others, in the ease of use, even if it is less powerful. I have seen that the kind of people that has to use this kind of tools, so defining governance, are not usually very skilled as developers.

[objectiser]

Hi Jorge

I believe someone has already worked on a camel connector for interacting with jBPM, so the plan was to make use of that to invoke workflows - and also provides a good migration path from DTGov 1. We would probably need to implement a drools component.

Agree on keeping it simple and hiding camel - that will be the aim - although I think it would be good to do some early prototypes of just the runtime side of DTGov 2, with hand coded camel, to demonstrate a few meaningful use cases.

[jorgemoralespou_2]

Hi Gary,

AFAIK camel-jbpm component has many issues when running on Karaf, as it seems there are many dependencies not very OSGi friendly like "kie-services-client having a dependency on HornetQ". Not sure on how much/many, and if the upcoming jbpm-client in next release solves this issues, as I guess SwitchYard will run into the same kind of issues/problems. Just something to be warned of. So, I guess that if there is problems with camel-jbpm there will also be with SwitchYard, so again, I have no point ;-)

Doing early prototyping has proven to me very much effective. And having as many use cases covered (or documented) as well.