Deploying Overlord to Fabric

[objectiser]

Work is currently under way to support the Overlord sub-projects on Fuse/Karaf. This will result in the creation of features.xml definitions to enable the appropriate bundles to be installed into Fuse/Karaf.

We also want to enable these Overlord sub-projects to be deployed to Fabric as various profiles. This thread is intended to discuss ideas on how this should be achieved.

Profiles are explained here: https://access.redhat.com/site/documentation/en-US/Red_Hat_JBoss_Fuse/6.1/html/Fabric_Guide/files/Profiles.html

The maven plugin for fabric is explained here: https://access.redhat.com/site/documentation/en-US/Red_Hat_JBoss_Fuse/6.1/html/Fabric_Guide/files/F8Plugin.html

It appears that profiles can be defined in a hierarchy. So we may want to have an 'Overlord' profile to include aspects common to all overlord sub-projects, and then have the other profiles as child profiles to that. It may also be appropriate to have DTGov as a child of the SRAMP profile.

[virchete]

Hi guys,

I would like you to take a look to this presentation:

http://fabric8.io/presentations/devnation-2014-intro/index.html

The doc of  Red_Hat Jboss_ Fuse (previous links) is not very helpful. With this presentation (for fabric rookies like me) it is very helpful, because give a good overview of the product.

As the conversations I had yesterday with James Strachan, we can take a look to http://fabric8.io The doc of this site is much more complete than the one provided by Red Hat Fuse.

The version that is used in Fuse 6.1 is 1.0.X. The version that the doc of http://fabric8.io references is the 1.1.X . As James told me the main difference between this 2 versions is that in the 1.1.X it is possible to deploy Fabric as independant product in different Web Servers. It is not oblied to be deployed in Fuse.

Now here I explained my global overview of what we have to do (not entering in technical details):

What is a profile? A profile contains common properties, or in general resources for one container. The resources (what they call Features), can be property files, war, jar... whatever.

As Fabric is a distributed platform, it can not reference to artifacts in the file system. It has to reference to an URL. Fabric has good cooperation with Maven.

There are different ways for deploying the artifacts as features of a Profile:

• Fabric maven plugin
• fabric:watch maven url repository

For me the nicer one is to specify the maven urls where fabric can take a look to deploy the artifacts (include features in the profile).

During the installation of Fuse we need to make the following steps:

1. Install Fuse 6.1.
2. Configure Fuse with some containers (for instance 2 containers).
3. Create an overlord-profile. When we talk about profile( it is very well explained in the presentation), it means all the common resources.

There are 2 task in overlord-fuse-fabric-integration:

• deploy the overlord products in the fabric.
  • Use fabric:watch to check the deployments in the Maven URL. Here we can watch in the different product maven path urls(dtgov url fuse war, s-ramp fuse war url ...)
• New Fabric Fuse Deployer. This deployer is gonna act exatly like a maven deployer. The only point here is that the maven url (groupId/artifact...) need to be one that is defined in fabric as fabric:wath

What I am gonna do right now? First try to create the profiles in fabric and deploy one product (s-ramp right now, because it has a war that can be deployed in Fuse) from my local maven repository.

For the new Fabric Fuse Deployer, the only point here is to create a new bpmn that in the deployment phase deploy a Fabric Fuse Artifact as target.  The Deployer code is nothing relevant, because I am thinking to inherit from the MavenDeployer.

[objectiser]

Thanks for gathering more information.

In terms of the versions of fabric, although the docs on the fabric8.io site may be better, we need to make sure any work we do conforms to version 1.0.x (i.e. Fuse 6.1) for now.

Would prefer to keep this thread just for discussion of deploying Overlord to fabric - I think the Fuse/Fabric deployer discussions should be separate (or possibly on the jira: [DTGOV-144] Support fabric as a deployment type - JBoss Issue Tracker).

Although SRAMP and DTGov may just be wars, they have a number of third party dependencies that also need to be installed. Similarly RTGov defines a number of features, not just a war. So would be better if the profile could reference the relevant features, rather than direct refs to wars.

[virchete]

One error I found in karaf console of fuse 6.1

Could not acquire port lock. Karaf command line · Issue #1258 · fabric8io/fabric8 · GitHub

Steps:

• Download Fuse
• execute fuse/bin/karaf
• fabric:profile-create --parents feature-cxf gs-cxf-base
• profile-create --parents gs-cxf-base gs-cxf-01
• profile-create --parents gs-cxf-base gs-cxf-02
• fabric:container-change-profile child1 example-camel-hello
• fabric:container-list

After this I receive the error:

fabric:container-list

[id] [version] [connected] [profiles] [provision status]

Error executing command: Could not acquire port lock

[virchete]

As conversation with James, it is not good idea to use fabric:watch, because it is used for development purposes:

[13:12] <jstrachan> cool; note that fabric:watch is only really needed for developing snapshots really
[13:12] <jstrachan> if a profile changes to use a different version; it auto-updates already
[13:13] <jstrachan> e.g. if you release 1.0.2 and were on 1.0.1; just change a reference to the new version in a profile & it auto-redeploys

We need to use a features.xml

[13:15] <jstrachan> you only need a features file if you've multiple resources you wanna group together into a single thing; if its just a war, there's no need - you can refer to it as a single entry in the profile

[13:16] <virchete> but the war
[13:16] <virchete> has other dependencies
[13:16] <virchete> that should be included
[13:16] <jstrachan> so its not a stand alone war?
[13:16] <virchete> no
[13:16] <virchete> it has some provided dependencies
[13:17] <jstrachan> ok if you wanna add some bundles too along with the war then you need a features file yeah

There are different ways of  deploy the features in to the fabric:

https://access.redhat.com/site/documentation/en-US/Red_Hat_JBoss_Fuse/6.0/html/Getting_Started/files/Develop-Feature.html

One could be copy the features.xml in the fabric root container/deploy folder

Another could be to deploy the features.xml in a maven repository and then Fabric would read from the repository.

[eric.wittmann]

For what it's worth, S-RAMP has a features.xml file which is generated during the maven build and attached as a resource to the s-ramp-distro-fuse61 module:

https://github.com/EricWittmann/s-ramp/blob/master/s-ramp-distro/s-ramp-distro-fuse61/pom.xml#L216-L328

This is how I have been installing s-ramp into Fuse:

    features:addurl mvn:org.overlord.sramp/s-ramp-distro-fuse61/0.4.0.Final/xml/features
    features:install -v s-ramp

[objectiser]

I think making use of the features.xml is good, as it means we can test everything just based on installing the features.xml directly. Having fabic pull it from nexus will also be good - however the profile will also need to include other things (e.g. property files). Do we know where these will be retrieved from?

[virchete]

Thanks Eric,

That is an amazing job. I am focusing right now in the deployment from the Deployer...

I created a post in the mailing group of Fabric8:

https://groups.google.com/forum/#!topic/fabric8/KjxKWsB2Cjc

It is easy to deploy from karaf console and maven. But our requirement is to deploy from Java side. It is specified in the Post I created my doubts.

[virchete]

Hi guys,

Here you can see the current status of the fuse fabric integration:

https://github.com/fabric8io/fabric8/issues/1322

It is useful, because I also included all the commands I use to create a profile, to assign a profile to a container...

The current status is:

-S-ramp-server is deployed. But authentication is not working... using the user included in the etc/users.properties.

-S-ramp-ui is not getting deployed.

[virchete]

Another question:

https://github.com/fabric8io/fabric8/issues/1323

This is related on how to create a new profile. As conversation with James Strachan for creating a profile is not just to create a new folder in the fabric fuse installation:

-We should clone the local git repository included on fabric fuse.

- Create a new folder with the profile name.

-Include there the properties fiiles and common stuff.

-Push and restart the fabric.

But as normal, this is not working. It is not possible to access the local git repository...

[eric.wittmann]

FYI I'm still working on authentication.  I'm going to be renovating how the IDP works so that it is more consistent across platforms and so that it works in Fuse.  I'll be doing the same with the SPs (the UI web apps).

Additionally, s-ramp won't work in Fuse right now because of the datasource issue.

I don't know why s-ramp-ui isn't deploying.  But I'll be circling around to that at some point - once the IDP is working.

[virchete]

Hi,

Well the status is good right now.

I can access s-ramp-ui. The authentication is failing.

I receive when deploying s-ramp-ui-war an exception trace about IDP:  https://gist.github.com/dvirgiln/3c0108041682f05c0202

[eric.wittmann]

You shouldn't expect the IDP to work in Fuse until I've had a chance to commit my changes.  We upgraded picketlink and broke the Fuse distro.  I'm working on revamping the IDP and SP code now.

[virchete]

Thanks Eric!

Another problem in the integration with fuse fabric: the profiles properties files are not getting filtered.... I know is not nice. I was expecting them to be filtered. Here is the conversation with James. As he is suggesting we should include a dependency with fabric8-api library and call a service method to get the properties file filtered.

Also they are not using CDI for injecting services. He told me they are using Osgi Declarative Services... so more complexity. I think Eric has some suggestions.

[17:25] <jstrachan> I tend to use this helper method https://github.com/jstrachan/fabric8/blob/micro-service/fabric/fabric-zookeeper/src/main/java/io/fabric8/zookeeper/utils/ZooKeeperUtils.java#L359-359

[17:26] <jstrachan> not ideal; the substitution APIs could be a little friendlier; there's also this one: https://github.com/jstrachan/fabric8/blob/micro-service/fabric/fabric-api/src/main/java/io/fabric8/api/FabricService.java#L35-35

[17:27] <virchete> then, just to clarify jstrachan, you are saying that when a profile is added to a container the property resolvers are not applied to the files contained in the profile?

[17:28] <virchete> because then it will couple our code with specific Fabric code, that is not the ideal

[17:28] <virchete> for sure

[17:28] <jstrachan> if you lookup a properties object in a profile its usually substituted via this method: https://github.com/jstrachan/fabric8/blob/micro-service/fabric/fabric-api/src/main/java/io/fabric8/api/Profile.java#L115-115

[17:28] <-- chirino_m (~[e-mail]) ha dejado este servidor (Quit: Computer has gone to sleep.).

[17:28] <jstrachan> fabricService.getCurrentContainer().getOverlayProfile().getConfiguration("myPropertiesFileNameWithoutDotProperties")

[17:30] <virchete> But how to read a property files using fabric

[17:30] <virchete> I do not know about your code internally

[17:30] <virchete> what I have

[17:30] <virchete> is an URL

[17:30] <virchete> new URL("profile:s-ramp-ui.properties");

[17:31] <virchete> then you are telling me that if i want to filter this file

[17:31] <-- jrawlings (~[e-mail]) ha dejado este servidor (Quit: My MacBook Pro has gone to sleep. ZZZzzz…).

[17:31] <virchete> i can not access the URL using commons-configuration library

[17:31] <virchete> because it will take the plain file

[17:32] <jstrachan> we have APIs in fabric8 to do the replacement for you; if you don't wanna use fabric8's APIs am not sure how best to help really

[17:33] <virchete> but then

[17:33] <virchete> I need to include the fabric8 dependency

[17:33] <virchete> in our project

[17:33] <jstrachan> we could maybe try add some kinda fabric8 plugin to expose things as system properties / env vars for you?

[17:33] <virchete> i do not know james

[17:34] <virchete> you know the problem what is

[17:34] <virchete> it is

[17:34] <virchete> we need to use

[17:34] <jstrachan> (though ordering can be an issue; what if you start up before we've populated the system properties etc)

[17:34] <virchete> the version that is in fuse 6.1

[17:34] <virchete> is complex..

[17:34] <jstrachan> all the APIs I showed above are in 6.1

[17:34] <virchete> and then

[17:34] <virchete> is there any solution for filtering the properties file

[17:34] <virchete> depending the container

[17:34] <virchete> for the 6.1 version?

[17:35] <jstrachan> the links above?

[17:35] <virchete> ahh oki

[17:35] <virchete> but to use

[17:35] <virchete> the fabric 8 api

[17:35] <virchete> could you tell me which dependency

[17:36] <virchete> i need to add to our project?

[17:36] <virchete> maven dependency

[17:37] <jstrachan>         <dependency>

[17:37] <jstrachan>             <groupId>io.fabric8</groupId>

[17:37] <jstrachan>             <artifactId>fabric-api</artifactId>

[17:37] <jstrachan>         </dependency>

[17:39] <virchete> thanks James

[17:39] <virchete> on monday i ll try it

[17:40] --> dstanley2 (Adium@nat/redhat/x-uetxfyxpfpwnyljv) se ha unido a este canal.

[17:41] <virchete> jstrachan I imagine

[17:42] <virchete> making an @Inject of the service

[17:42] <virchete> will make the injection correctly isnt it?

[17:42] <-- dstanley1 (dstanley1@nat/redhat/x-tfldpnxrlqofnxqm) ha dejado este servidor (Ping timeout: 258 seconds).

[17:42] <jstrachan> we don't use CDI usually we use "OSGi Declarative Services"

[17:43] <jstrachan> if you copy this project https://github.com/jstrachan/fabric8/tree/micro-service/fabric/fabric-cxf-registry - and this class: https://github.com/jstrachan/fabric8/blob/micro-service/fabric/fabric-cxf-registry/src/main/java/io/fabric8/cxf/registry/FabricCxfRegistrationHandler.java#L78-78

[17:43] <jstrachan> you'll have something thats a service thats registered into fabric/osgi and gets FabricService injected

[17:44] <jstrachan> you can delete all the code but bindFabricService / unbindFabricService and the @Activate / @ Deactivate methods

[17:45] <virchete> aha

[17:45] <virchete> i thougth would be simpler!

[17:45] --> gtully (~gtully@95.44.247.235) se ha unido a este canal.

[17:45] <virchete> i am giving you ideas

[17:45] <virchete> for next releases

[17:45] <virchete> to make it simpler

[17:46] <virchete> i think the automatically filter of profile files

[17:46] <virchete> would be very nice

[17:46] <jstrachan> chat to tdiesler and iocanel; they're looking at simplfiying it

[17:46] <jstrachan> having an @Injection of profile data would be nice