Revised use case for creating and governing JBOSS ESB based

[jervisliu]

Hi, posted below is a revised use case for creating and governing JBOSS ESB based web service (EBWS). What I have to confess is that I do not have a lot experiences on using JBOSS products to develop real projects in a real production environment. The use case I posted below can be very wrong. That is reason why I asked for everyone's input. These use cases will be served as a PRD to decide what features we want to include for our first release of SOA Repo. So please, your comments and suggestions will be highly appreciated.

Use Case : Creating And Governing JBOSS ESB Based Web Service (EBWS)

Step 1: Creating a service.
Repository Explorer -> Services -> Create New Service -> Add JBOSS ESB based Web Services

For version 1, we provide options of creating following service types:
a. SOAP service(=EBWS)
b. JBOSS ESB service
c. Generic service.

In this use case, we create a JBOSS ESB based Web Service(EBWS) named AirlineBookingService.


Step 2: Viewing and editing the service from Service Editor.

Once the service is created successfully, double click the AirlineBookingService icon from Repository Explorer -> Services -> AirlineBookingService. This will bring out a service editor for AirlineBookingService. The service editor is consisted with following sections:

a. "Artifact Summary" section
b. "Artifact MetaData" section
c. "Artifact LifeCycle" section
d. "Artifact Dependencies" section.
e. "WhatDependsOnMe" section.
f. "Comments" section.
g. "History" section.

At the very top of Service editor is the "Artifact Summary" section, it contains information as below:

a. The feed: The feed icon provides an Atom/Pub subscription to this artifact's version tree.
b. Description: click the "edit" icon, you will be able to edit the description field.
c. Type: For this use case, the type is JBWS.
d. Version: Every change to the artifact will make a new version.
e. Perm link: This link is a permanent link to this particular version of service. For example, a Perm link to version 3 of test.wsdl is "http://localhost:8080/guvnor-web/repository/jbossesb/test.wsdl?version=3". A Perm link to the latest version of test.wsdl is "http://localhost:8080/guvnor-web/repository/jbossesb/test.wsdl". Clicking the Perm link will download the actual artifact (through HTTP Get, served by Atom/Pub server).
f. "New Version" button: Clicking the "New version" button will bring up a wizard to upload a new version of artifact.
g. "Delete" button: Clicking the "delete" button will delete the current version of artifact from the version history tree.

NOTE: The downloading function (Perm link) and New version function can only apply to these artifacts that have concrete representation formats, for example a xsd file or .esb file. Not sure what should return if a user click the Perm link of AirlineBookingService as its just a container or holder, there is nothing concrete to download.

Step 3: Add artifacts that the service may depend on – Add XSDs
Repository Explorer -> Schemas -> Create New Schema -> Add New XSD
In this use case, we upload three XSD files called AirlineBookingInMessage.xsd, AirlineBookingOutMessage.xsd and AirlineBookingFaultMessage.xsd.


Step 4: Add artifacts that the service may depend on – Add WSDL file
Repository Explorer -> Contract -> Create New Contract -> Add New WSDL

When a JBOSS ESB based Web Service gets deployed, a WSDL file is generated. Save the WSDL file on local disk (name this file AirlineBookingService .wsdl) then upload AirlineBookingService .wsdl into the repository.

NOTE. For known artifact types, we can extract metadata automatically during the uploading process. For example, when AirlineBookingService .wsdl gets added into repository, its port name, service name, port type etc will be extracted from the WSDL file and added as metadata automatically.


Step 5: Add artifacts that the service may depend on – Add a document.
Repository Explorer -> Document -> Create New Document -> Add New Document
User can attach a document to the service at any stage.

In this use case, we upload a Word file called AirlineBookingMessagePRD.odt.


Step 6: Add artifacts that the service may depend on – Add arbitrary files.
Repository Explorer -> Generic artifacts -> Create New Artifact -> Add New Artifact
User can add any arbitrary type files into the repository.


Step 7: Add dependencies.
7.1 Add AirlineBookingInMessage.xsd as AirlineBookingService's dependency: Open AirlineBookingMessage.xsd editor, copy the URL from perm link. Open AirlineBookingService editor, add a new dependency whose URL is the perm link of AirlineBookingMessage.xsd
7.2 Add AirlineBookingService .wsdl as dependency
7.3. Add AirlineBookingMessagePRD.odt as dependency


Step 8: Add MetaData

8.1 From Repository Explorer -> Administration -> MetaData Types, add a new Metadata type called ReleaseForVersion whose type is metadata_type_multi_value.
8.2 From Repository Explorer -> Services -> AirlineBookingService, add a new metadata type named ReleaseForVersion which we created in step 8.1 to AirlineBookingService.
8.3 Edit the value of ReleaseForVersion metadata from AirlineBookingService Service editor.


Step 9: Manage the lifecycle.

9.1 From Repository Explorer -> Administration -> Life Cycles, add a new lifecycle type called MyLifeCycle. MyLifeCycle will have three phases: "ServiceOrientiedDesign; ServiceOrientiedTest; ServiceOrientiedProduction"
9.2 From AirlineBookingService service editor, choose the lifecycle type MyLifeCycle which we created in step 9.1 as the lifecycle type for AirlineBookingService.
9.3 Move the lifecycle phase forward from ServiceOrientiedDesign to ServiceOrientiedTest.


Step 10: Add a comment
The comments section is a JIRA comment like area.


Step 11: Versioning/History
Evey change to AirlineBookingService will make a new version on its version tree. This History section lists all available version names, checkin comments and the link for this specific version. Click the version link, will bring out a service editor for that specific version (though all the information on that service editor will be read only)


Step 12: Search/Query


Step 13: Atom/Pub support.
REST support has been built into the core of SOA Repository. For every feature that is available from GWT GUI, it is also available from Atom/Pub and JAX-RS (JSR-311) interface. For example, the perm Link on Service editor ( eg, http://localhost:8080/guvnor-web/repository/jbossesb/test.wsdl?version=3)
is actually served by the atom/pub server over Http transport instead of using GWT RPC.


Step 14: Deploy AirlineBookingService to a testing or production environment.

SOA Repository does not provide any tools or helps to actually develop a JBWS, and it is not designed to do this. A service developer can always develop and build the service from command line using whatever editor and building tool or from an IDE eg JBOSS Eclipse developer studio. Lets take JBOSS Eclipse studio as an example, the interactions between Eclipse and SOA Repository is as below:

1.Java source code of AirlineBookingService is created and developed through Eclipse.
2.Jboss-esb.xml file and other configuration files that are needed by JBOSS ESB service are also generated and updated through Eclipse.
3.AirlineBookingInMessage.xsd and other xsd files are created by service developers.
4.AirlineBookingMessage.wsdl file is generated by ESB server during the deployment process. This wsdl file is saved to local file system manually by developers.
5.All other files that are needed by JBOSS ESB service are also maintained from Eclipse.
6.A service called AirlineBookingService is created in SOA Repo. Its nothing but an empty holder.
7.Jboss-esb.xml, AirlineBookingInMessage.xsd, wsdl files etc are added into SOA Repo from Eclipse project (or from the local file system). In SOA Repo, these files are called service artifacts, and they are associated with AirlineBookingService. From this point on, the SOA Repo is the "single source of truth" for these artifacts (eg, Jboss-esb.xml, AirlineBookingInMessage.xsd etc). Of course, it is possible that these files are also stored in a source control system like SVN. But the SOA Repo is the only authority for publishing the actual content of these files as well as any other information associated with these files.
8.If the AirlineBookingInMessage.xsd file is updated later on by developers using Eclipse, the developer need to upload the new version of AirlineBookingInMessage.xsd file to SOA repository. This will create a new version of AirlineBookingInMessage.xsd in SOA Repo.
9.Other developers can download AirlineBookingInMessage.xsd from SOA repository to their Eclipse project or local file system. If they change the xsd file, they need to upload it back to SOA Repo as well.
10.So the most common interaction between SOA Repo and Eclipse (or the local file sytem where the service project stays) is the sync of files between SOA Repo and Eclipse.
11.When the service project is ready for testing or production, developer generates a .esb deployment bundle using ant or from Eclipse.

NOTE: once .esb file is generated, I am not sure if we want to add this .esb file to the repository. The .esb file can be one of artifacts that the service depends on.

12.At the moment, there is no function offered in SOA Repo to do the deployment and un-deployment.

[jeffdelong]

I think we should promote the idea that there is a high level description of a service (i.e., its requirements), and then there are the design and implementation details. When a business analysts / soa architect identifies the need for a service, they don't necessarily know that it is a JBossESB service (or even a Web service) at that point. But they still want to capture the requirements for the service, and put the service under lifecycle management. Design and implementation details can then be added later.

So instead, I see the steps more like the following:

Step 1: Create a service.

Repository Explorer -> Services -> Create New Service

name it AirlineBookingService, put it under lifecycle management, add requirements artifacts, move to the service-oriented analysis phase. Add additional artifacts, move to service-oriented design phase.

Step 2: Create a service implementation beneath the Service

Repository Explorer -> Services -> Create New Web Service or
Repository Explorer -> Services -> Create New ESB Service or
Repository Explorer -> Services -> Create New ESB Web Service or
...

Now pick up with your Step 2.

[jeffdelong]

With regard to your Step 2, we might want to distinguish between the version of a file within a service and the version of the service itself. See Step 11. We need to think though service versioning much more.

[jeffdelong]

With respect to Step 3 and 4, while it makes perfect sense to have XSDs uploaded to a common area of the repository, I am not sure that this is true for WSDLs. XSDs can be shared by many services (imported into their WSDL), but a WSDL defines a specific service.

So I would expect

Repository Explorer -> Contract -> Create New Contract -> Add New WSDL
to only be applicable in the context of a given service (indeed in the context of a given Service Implementation - see by earlier response).

[jeffdelong]

With respect to Step 9 Manage the Lifecycle, really this is something that already should have been done.

I would like to see the software provide an out-of-the-box Lifecycle called the Red Hat SOA Development Lifecycle. It would consist of the following phases:

Business Analysis, Service-Oriented Analysis, Service-Oriented Design, Service Development, Service Testing, Service Deployment, Service Retirement.

This lifecycle is the core of our SOA workshop and associated consulting services, and the repository will play a key role in managing this process.

Customers should of course be able to create their own lifecycles (as you have indicated).

[jeffdelong]

One other feature I would like to see is the ability to publish WSDL locations for Web services to a registry, in particular the jUDDI registry. While this is not the same as service deployment (which I think you explained very well is accomplished outside the Service Repository), it is an important function that can be automated by the Service Repository by using standard UDDI APIs.

[jervisliu]

"jeffdelong" wrote:
With respect to Step 9 Manage the Lifecycle, really this is something that already should have been done.

I would like to see the software provide an out-of-the-box Lifecycle called the Red Hat SOA Development Lifecycle. It would consist of the following phases:

Business Analysis, Service-Oriented Analysis, Service-Oriented Design, Service Development, Service Testing, Service Deployment, Service Retirement.

This lifecycle is the core of our SOA workshop and associated consulting services, and the repository will play a key role in managing this process.

Customers should of course be able to create their own lifecycles (as you have indicated).

The preview version of SOA Repository does come with an out-of-the-box lifecycle called DefaultLifecycle. But the Red Hat SOA Development Lifecycle sounds much more "advanced", I will make this one the default one instead. With the preview version, when a service or artifact gets created, its lifecycle type is "un-selected". A user can then open the service editor, choose a lifecycle type, set the inital phase accordingly. A different approach is to set the lifecycle type to Red Hat SOA Development Lifecycle by default, and set the initial phase to Business Analysis. A user can always use the service editor to choose a different lifecycle later on.

[jeffdelong]

I think the lifecycle should apply to the entire service. I.e. when the user creates a service, they are asked which lifecycle process to apply to this service. At this point the service enters Business Analysis. The user then adds artifacts to the repository at different phases of the lifecycle.

An additional (future) feature would be for there to be a list of artifacts that are required at each phase of the lifecycle, and the ability for the user to check these off as they are added. Then when the user wants to say they have completed a phase, the system would consult the list and tell them if everything was completed.

[jervisliu]

"jeffdelong" wrote:
I think the lifecycle should apply to the entire service. I.e. when the user creates a service, they are asked which lifecycle process to apply to this service. At this point the service enters Business Analysis. The user then adds artifacts to the repository at different phases of the lifecycle.

Agree. The user will be asked to fill in lifecycle info (what lifecycle type to use, which lifecycle phase to use as the initial phase) from "New Service" wizard.

How about artifacts lifecycle? Are they going to have lifecycle at all? If the artifact is not shared (i.e., it belongs to one service only), then I see no problme for this artifact to join the service lifecyle. If the artifact is a global one like an xsd file, I am not sure how we can apply lifecycle to it as various services might be under different phases or even using differnt lifecycle types.

"jeffdelong" wrote:

An additional (future) feature would be for there to be a list of artifacts that are required at each phase of the lifecycle, and the ability for the user to check these off as they are added. Then when the user wants to say they have completed a phase, the system would consult the list and tell them if everything was completed.

Looks like this is a job can be done by policy enforcement. SOA Repo should allow users writing their own policies then associate policies with different lifecycle phases. But I have to admit that I have no idea yet at the moment what kind of technologies this policy enforcement framework can be based on. If anyone has any suggestions, please do advise.

[jeffdelong]

I don't believe an artifact has a lifecycle. Rather an artifact is created as part of a lifecycle. E.g. an XSD is a requirement of the service-oriented design phase of the Red Hat SOA Development lifecycle.

I don't think we want to apply a lifecycle to an XSD. I don't think we would talk about the requirements for an XSD, the design of an XSD, the development of an XSD, etc.

So in my opinion lifecycle applies to services. In general we need to place the emphasis on the services, then the supporting artifacts.

I think the description of Guvnor on the web page is incorrect as well. It says:

Overlord project for "Governance" requires a repository to store artifacts - this is Guvnor ! The repository is a versioned store for all "SOA" related artifacts (files) that are needed by runtime and design time systems. For instance, storing configuration files, editing them, and then managing their lifecycle through various interfaces (web, Eclipse) is what Guvnor is all about.

I believe it should say:

Overlord project for "Governance" requires a repository to store and manage the lifecycle of services in support of service discoverability, a key aspect of service oriented architectures. This is Guvnor! The repository is a versioned store for services and all service-related artifacts (files) that are needed by runtime and design time systems. Storing descriptions of services, along with their metadata, contracts, configuration files, etc., and then managing the lifecycle of these services from analysis through deployment, and allowing access to this information through various interfaces (web, Eclipse) is what Guvnor is all about.

The emphasis should be on services.

[jervisliu]

I don't believe an artifact has a lifecycle. Rather an artifact is created as part of a lifecycle. E.g. an XSD is a requirement of the service-oriented design phase of the Red Hat SOA Development lifecycle.

I don't think we want to apply a lifecycle to an XSD. I don't think we would talk about the requirements for an XSD, the design of an XSD, the development of an XSD, etc.

So in my opinion lifecycle applies to services. In general we need to place the emphasis on the services, then the supporting artifacts.

How about WSDL file? the design of WSDL interface does evolve along with lifecycles. The creation, editing and validation of a WSDL happens outside the SOA Repository (using whatever tools users prefer like XMLSpy, JBOSS Developer studio etc), but when the change has been done, a new version of WSDL has to be uploaded back to SOA Repo, right? In this case, we do need to know the status of this WSDL file, is it ready for production or still under testing stage.

[jervisliu]

"jeffdelong" wrote:
With respect to Step 3 and 4, while it makes perfect sense to have XSDs uploaded to a common area of the repository, I am not sure that this is true for WSDLs. XSDs can be shared by many services (imported into their WSDL), but a WSDL defines a specific service.

So I would expect

Repository Explorer -> Contract -> Create New Contract -> Add New WSDL
to only be applicable in the context of a given service (indeed in the context of a given Service Implementation - see by earlier response).

Potentially the SOA Repo will need to support an unlimited list of artifact types. Some artifacts might be shareable, some are not. And I guess most of time whether or not an artifact is sharable is not decided by us, but by the user who is using SOA Repo. We need a unified way to handle this from the SOA Repo GUI.

But I dont have the answer yet ;-)

[jeffdelong]

Let's say that in our Lifecycle process the creation of the WSDL is part of service-oriented design (never mind for now that in the ESB environment we auto-generate the WSDL once the service is created). Keep with your scenario and assume the WSDL is created outside the repository, and uploaded as an artifact during service-oriented design. It may get updated during this phase as part of the design effort, and its version number should increase, along with metadata about who uploaded the new version, wehn, etc.

Now at some point the developer says the design of this service is complete, and moves the service to the next phase, service development. The WSDL might change again, perhaps now the concrete binding information is added to the WSDL, for example. Again the new version is updated.

However it is not the WSDL that is ready for production, or in testing, but rather it is the SERVICE that is ready for production or in testing. The WSDL is just one artifact of the service. An important one, as it is what the client bind to, but it is not the service. The service is ready to move from testing to production deployment when ALL of the requirements of the testing phase have been completed, which includes more than just the WSDL.

Now let's go to a more interesting challenge. Someone comes along and decides that the service needs to evolve. Now we need to create a new Version of the entire service (including its WSDL), and manage this new service through the service development lifecycle. This could involve a change to the WSDL (or not depending on how good a job we did designing it in the first place). Of course if the WSDL did change, then the contract has changed, as we need to understand how to best deploy this new version of the service since it may not be consumable without changes to its clients.

This brings up another interesting feature for a service repository, and that is the ability to keep track of clients / other services that consume a service, so we know who changing a service will impact.

[jervisliu]

"jeffdelong" wrote:
Let's say that in our Lifecycle process the creation of the WSDL is part of service-oriented design (never mind for now that in the ESB environment we auto-generate the WSDL once the service is created). Keep with your scenario and assume the WSDL is created outside the repository, and uploaded as an artifact during service-oriented design. It may get updated during this phase as part of the design effort, and its version number should increase, along with metadata about who uploaded the new version, wehn, etc.

Now at some point the developer says the design of this service is complete, and moves the service to the next phase, service development. The WSDL might change again, perhaps now the concrete binding information is added to the WSDL, for example. Again the new version is updated.

However it is not the WSDL that is ready for production, or in testing, but rather it is the SERVICE that is ready for production or in testing. The WSDL is just one artifact of the service. An important one, as it is what the client bind to, but it is not the service. The service is ready to move from testing to production deployment when ALL of the requirements of the testing phase have been completed, which includes more than just the WSDL.

Hi Jeff, thanks for the detailed explanation. I am convinced. Here is the JIRA: https://jira.jboss.org/jira/browse/GUVNORSOA-43

"jeffdelong" wrote:

Now let's go to a more interesting challenge. Someone comes along and decides that the service needs to evolve. Now we need to create a new Version of the entire service (including its WSDL), and manage this new service through the service development lifecycle. This could involve a change to the WSDL (or not depending on how good a job we did designing it in the first place). Of course if the WSDL did change, then the contract has changed, as we need to understand how to best deploy this new version of the service since it may not be consumable without changes to its clients.

This brings up another interesting feature for a service repository, and that is the ability to keep track of clients / other services that consume a service, so we know who changing a service will impact.

With the preview version of SOA Repository, two sections on service editor are related to dependency: dependency section and whatDependsOnMe section. In this case, whatDependsOnMe can be used to track and analyse the potential impact of a change. There are two scenarios here as far as I can see:

1. With explictly defined dependency: Presume we have a, b, c services, they all depend on inmessage.xsd. The user has to open service editor for service a,b,c then add a dependency to inmessage.xsd explictly. Once this is done, inmessage.xsd's service editor will display the URL of a,b,c under the whatDependsOnMe section. This information will help to analyse the potential impact if inmessage.xsd is changed.

2. With implicit dependency: clients / other services that consume a service falls into this category. If the "clients / other services" are not defined as an entry in SOA Repo, I dont think there is much we can do. At most we can add an entry under whatDependsOnMe to say there is a client called blabla depends on me. The blabla is nothing but a string, might be the name for that client. But there is no way we can use this entry to navigate to the actual client itself.

[objectiser]

Just to check - so if the lifecycle is just associated with the service, will the versions of the dependent artifacts be "locked in" when the service changes to the next lifecycle phase?

For example, if a XSD is directly or indirectly a dependency of a service, and the service moves from development to test, we want to make sure the artifacts associated with the service are stable - so if someone then uploads a new version of the XSD, then service in test is not affected.

Or possibly this should be offered as an option when changing lifecycle phase?