S-RAMP Event Model

[eric.wittmann]

Description

In version 0.6 of S-RAMP we will be supporting a true (JMS) event mechanism that DTGov will leverage.  This will replace the current DTGov polling approach to finding changes in the repository.

Objective

This discussion is intended to derive the data model that s-ramp should use when emitting events, along with the types of events.

Strawman Proposal

Event Type:  New Artifact

Primary event key is the artifact UUID.

The event data model should include *all* s-ramp meta-data, including:  1) standard properties 2) custom properties 3) classifiers 4) relationships?? (not sure about #4)

Sample JSON data model:

{
  "properties" : {
    "uuid" : "12345",
    "model" : "wsdl",
    "artifactType" : "WsdlDocument",
    "name" : "example.wsdl",
    "description" : "Something here.",
    "createdBy" : "ewittman",
    "version" : "1.0",
    "createdTimestamp" : "2014-01-01T11:33:17Z",
    "lastModifiedTimestamp" : "2014-01-01T11:33:17Z",
    "lastModifiedBy" : "ewittman",
    "targetNamespace" : "urn:org.example.wsdl/example.wsdl",
    "custom-property-1" : "foo",
    "custom-property-2" : "bar"
  },
  "classifiers" : [
    "http://www.jboss.org/overlord/deployment-status.owl#DevPass",
    "http://www.jboss.org/overlord/deployment-status.owl#QaPass"
  ]
}

Event Type:  Updated Artifact

Primary event key is the artifact UUID.

The event model will contain the *changes* to the artifact's meta-data, including:  1) standard properties 2) custom properties 3) classifiers 4) relationships?? (not sure about #4)

Sample JSON data model:

{
  "old-properties" : {
    "name" : "example.wsdl",
    "description" : "Something here.",
    "lastModifiedTimestamp" : "2014-01-01T11:33:17Z",
    "lastModifiedBy" : "ewittman",
    "custom-property-2" : "bar"
  },
  "properties" : {
    "name" : "example-updated.wsdl",
    "description" : "Something NEW here.",
    "lastModifiedTimestamp" : "2014-02-10T11:33:17Z",
    "lastModifiedBy" : "tbrady",
    "custom-property-2" : "new-value-here"
  },
  "removed-classifiers" : [
    "http://www.jboss.org/overlord/deployment-status.owl#DevPass",
  ],
  "added-classifiers" : [
    "http://www.jboss.org/overlord/deployment-status.owl#QaFailed"
  ]
}

Event Type:  Deleted Artifact

Primary event key is the artifact UUID.

No other event data is necessary.

[objectiser]

Good start, although I would like to widen the scope to cater for other event sources (e.g. git, etc). So a couple of comments:

1) I think we need to separate fixed element names from properties, which will be generally be source specific. So in terms of the "add" example above, "name" would be at the top level, and "classifiers" would be a nested element under "properties".

{

  "type" : "addArtifact",

  "name" : "example.wsdl",

  ....

  "properties" : {

    .....

    "custom-property-2" : "bar",

    "classifiers" : [

      "http://www.jboss.org/overlord/deployment-status.owl#DevPass",

      "http://www.jboss.org/overlord/deployment-status.owl#QaPass"

    ]

  }

}

2) Something like UUID is a specific way of locating the artifact in sramp, but is not generic. Changing to an event based model also changes the nature of the relationship between producer and consumer - i.e. previously sramp knew the event source as it pulled the data, but with the event driven mechanism, DTGov won't necessarily know the event source. So I think the events need to include top level fields for (a) event source - a way to know which system produced the event, and (b) how to locate the artifact within that event source - which in the case of sramp would be the UUID, but for GIT it would be a repo relative path.

Was wondering whether we could use 'project' instead of event source, and have the git repo, etc. locations defined in the project definition - however that would require the event sources to have knowledge of the 'projects', which I don't think would be right. So I think the only option is for each event to identify its event source in such a way that DTGov could then access that event source to gain further information or perform actions (e.g. reject PR).

3) In terms of the event types, there will be artifact related ones (e.g. add, update, remove) but we should also consider other event types that may be of interest - for example, DTGov may initiate tasks to be performed, which would best be done asynchronously - so possibly taskStarted, taskFailed and taskCompleted events? These are not important initially, so can be fleshed out later, but just wanted to introduce the concepts for discussion.

[objectiser]

Sorry forgot to say - in the modified event layout above I've separated out 'name' , but not artifactType. Was thinking that it may be better to apply DTGov policies based on the extension contained in the name, as it reduces the work required on the event publisher side to do artifact classification? So any event source could create events for "abc.wsdl" and it would apply the same policies.

Thoughts?

[brmeyer]

> So I think the events need to include top level fields for (a) event source - a way to know which system produced the event, and (b) how to locate the artifact within that event source - which in the case of sramp would be the UUID, but for GIT it would be a repo relative path.

+1.

> 4) relationships?? (not sure about #4)
Eric, what exactly did you have in mind there?  Does DTGov currently do anything with the relationships when polling?

[eric.wittmann]

I was assuming that we could have different payloads depending on the *type* of the JMS event.  So the format of the event body would be specific to the event type.  Is this not how JMS apps are typically designed?  Do they instead create a common event model and then all events regardless of type conform to that same format?

[eric.wittmann]

> 4) relationships?? (not sure about #4)
Eric, what exactly did you have in mind there?  Does DTGov currently do anything with the relationships when polling?

Not out of the box, no.  But there's nothing stopping a user from configuring this sort of thing in DTGov:

/s-ramp/ext/MyType[customRelationship[@property1 = 'some-value']]

At the very least, I imagine it would be nice to know whether an artifact has any relationships of a particular type as part of the event.  Also for Update events it's possible that the *only* change to an artifact is to create or break a relationship.  I would think we'd want an event fired for that, and include the relationship information.  The question is how much information to provide and in what format.

[objectiser]

Eric Wittmann wrote:

I was assuming that we could have different payloads depending on the *type* of the JMS event.  So the format of the event body would be specific to the event type.  Is this not how JMS apps are typically designed?  Do they instead create a common event model and then all events regardless of type conform to that same format?

Usually there has to be a common understanding between the publisher and subscriber, both in terms of the category of message (e.g. object, text, stream, bytes, map) as well as the actual format of the message payload - although the consumer could be fairly dynamic to determine both, or use header properties which may more explicitly provide the information.

However I think for our purposes it is better if the backend event framework uses a standard (json) event model, and then where necessary we do mapping from external event sources.

Just to clarify, when you say "type" of the JMS event - what "type" are you referring to? do you have examples?

[eric.wittmann]

Sure, I agree that the publisher and subscriber must have a common understanding of the event model.  I also agree that all events consumed by DTGov should probably be JSON as the event payload.

My understanding of your suggestion is that every event, regardless of type, would correspond to the same JSON Schema Type (if such a thing existed).  So that schema type would need to be generic enough to support all possible kinds of events.  Yes?

I guess I was thinking that each type of event could have its own JSON Schema Type, and therefore each schema type could be specific to the event type in question.

When I say event type, I logically mean the following (for example and not exhaustive):

• sramp:artifactCreated
• sramp:artifactUpdated
• sramp:artifactDeleted
• sramp:ontologyCreated
• sramp:ontologyDeleted
• git:commit
• git:pullRequest
• jira:issueCreated
• jira:issueUpdated
• jira:issueResolved
• jira:issueClosed
• jira:commentAdded

Within JMS I assume that each of the above is a possible value of this:

http://docs.oracle.com/javaee/5/api/javax/jms/Message.html#getJMSType()

I could be way off base.

Getting back to the message payload, let's pretend we were going to have the payload be XML instead of JSON.  I might then have one or more XSD file that defined the message payload format for each of the above message types.  So that the payloads would be specific to the type of event being fired, and the producer and consumer would each know what to send/receive.

[objectiser]

No, wasn't thinking about a generic schema type - was thinking about having an inheritance hierarchy where a particular field will be used to define the type. So use jackson's inheritance type mapping capability rather than having to explicitly specify the type as a header property. Haven't actually ever used the JMSType header property - and it seems a strange extension, referring to a repository of type definitions, so may not be implemented in a consistent way across JMS providers.

So essentially each 'event type' would have a different schema.

In terms of the event types - I would prefer to avoid reference to a particular target type, such as git or sramp - as it makes adding new event sources more difficult. We should keep the types generic, e.g. artifactCreated, artifactUpdated (which could apply to sramp update or git commit/PR - may be other properties can help the notification module understand whether the event type is associated with git or sramp, and therefore how to handle notifying the user in the appropriate way).

[brmeyer]

I think we might be talking about 2 different models.  There seems to be 2 distinct buckets, especially considering the recent Camel discussions:

1.) Events emitted by S-RAMP.  The JSON payload should be unique to S-RAMP.

2.) A common data model that's purely internal to DTGov.  Any incoming event source (S-RAMP, GitHub, JIRA, etc.) would first be transformed to the DTGov data model, then routed & acted upon.

Gary, correct me if I'm wrong, but you're mainly describing #2.  For #1, I was assuming something more along the lines of Eric's ideas, including the use of the JMSType header.