Preferred approach to Switchyard multiple environment properties?

[dustin.barlow]

I have been experimenting with using Maven profiles and filters to replace property values in the switchyard.xml file.

For example, I declare a property:

<sca:property value="${some.property}" name="someProperty"/>

I can then refer to it in any of my project's Bean classes by using:

@Property(name = "someProperty")
private String someProperty;

Here is the build section of the pom.xml I'm using to do key=value substitution.

<build>

        <filters>
            <filter>src/main/filters/properties/${env}.properties</filter>
        </filters>

        <resources>
            <resource>
                <directory>src/main/resources</directory>
                <filtering>true</filtering>
            </resource>
        </resources>

        <testResources>
            <testResource>
                <directory>src/test/resources</directory>
                <filtering>true</filtering>
            </testResource>
        </testResources>

        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.3.2</version>
                <configuration>
                    <source>1.6</source>
                    <target>1.6</target>
                    <debug>true</debug>
                    <showWarnings>true</showWarnings>
                    <showDeprecation>true</showDeprecation>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.switchyard</groupId>
                <artifactId>switchyard-plugin</artifactId>
                <version>${switchyard.version}</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>configure</goal>
                        </goals>
                    </execution>
                </executions>
                <configuration>
                    <scannerClassNames>
                        <param>org.switchyard.transform.config.model.TransformSwitchYardScanner</param>
                        <param>org.switchyard.validate.config.model.ValidateSwitchYardScanner</param>
                    </scannerClassNames>
                </configuration>
            </plugin>
        </plugins>
    </build>

When I run "mvn clean install -P dev", the ${env}.properties portion of the Maven filter file will be replaced with "dev.properties" (NOTE: for brevity I left out the <profile> section which sets the ${env} variable to "dev" based on the -P value passed in).

In the dev.properties file, I have a list of all the key=value pairs that I want to set during the build.  Maven then recursively scans all the files in src/main/resources and substitutes any ${keys} with whatever values I have set in my src/main/filters/properties/dev.properties file.

The issue with this approach I've run into is that the Switchyard plugin kicks off AFTER the Maven filter logic and whatever replacement Maven did on the target version of switchyard.xml that gets bundled in the jar file is overwritten by the Switchyard build plugin and as a result all the key=value substitutions are lost.

If I comment out the Switchyard build plugin and run it again, then the resulting target switchyard.xml file has the correct key=values replaced.  I assume that the Switchyard build is there for a reason (it was added by the Switchyard tooling) so commenting it out makes me a bit nervous as to what unknown side effects this may introduce.

So, what is the preferred way to do this type of multiple environment property management in Switchyard?  I really like using the Maven profile filters approach because it's easy to do and I can manage everything in source control.  However, I'm open to different solutions.

[jorgemoralespou_2]

Hi Dustin,

One drawback of using this approach is that if you ever share your generated artifacts (using dtGov is one way) you may be providing artifacts with incorrect values set up.

Instead I would suggest you to provide a module (dynamic or static) with a properties file with the provided configuration. This module has same problems with sharing, but, it is easier to maintain (single place for lookup).

Also, another option, and preferred by me, could be to externalize configuration in a cache, so the Property could get injected via a Produces that gets the value from an Infinispan cache. This cache can be handled separately.

From my point of view, going the maven way should be avoided as much as possible due to artifacts being promoted, etc... If you decide to do it, at least, use classifiers and generated different artifacts per environment/configuration.

[trohovsky]

Hi Dustin,

you are hitting the issue [SWITCHYARD-1501] Allow switchyard-plugin (config Mojo) to apply Maven resource filtering - JBoss Issue Tracker. Fortunately a workaround for that exists. It is mentioned in the JIRA. I think many people (including me and my workmate) have hit the issue and its priority should be raised. Please vote for it.

Commenting out the Switchyard plugin is not a good idea, because then you will loose validation of the switchyard.xml as well as generation of Transformers and Validators to the switchyard.xml.

Tomas

[dustin.barlow]

Re Jorge,

Thank you for the suggestions.  I'll certainly consider them.

Luckily in my case, this is strictly an internal project that won't be distributed outside of the company I work for.  If it were, I certainly wouldn't do it this way.  I would externalize anything that is environment specific.

Having said that, I don't really follow your suggestion of providing a separate module with a properties file and what the advantage of that is over maven's resource filtering.

Re Tomas,

Thanks for the link.  I also voted for it since it should be fixed even if it isn't considered the optional way to do environment property switching.

The project I'm currently using doesn't do any transformers or validators so that's why I considered commenting it out initially.  But you're right that I shouldn't have to do that or couldn't if I did start needing a transformer or validator.

I am still interested in hearing from others on how they handle multiple environment properties when it comes to Switchyard.  It is a common thing we all have to do in our various situations so it would be good to have some good discussions around it.  I really didn't see much or couldn't find much in the Switchyard docs on this topic.

Dustin

[jorgemoralespou_2]

Hi Dustin,

With maven resource filtering you end up with an artifact that has in its content's switchyard.xml with property for env dev. If you publish this artifact to an artifact repository, or use it in another environment it will have the variables inlined, so it will get previous environments values. If you want to have a new artifact for, let's saye stage environment, you need to rebuild with -P stage to have such an artifact. So this way, you will have to either, rebuild per environment, or build for every environment and have a classifier for the generated artifact that denotes the target environment.

So you are impacting the "logic" artifact.

There is a way of providing a module, either static (in modules dir) or dynamic (in deployment) that externalizes the values for configuration. This module should only have a properties file, or many, with the properties and values per environment.

This is just an artifact for configuration so you don't change/modify artifact with the logic.

It is a matter of, should your artifacts be built once and only once, and be sealed so you are sure you had same logic in all your environments?

Hope it is more clear now. 

[dustin.barlow]

Hi Jorge,

Thank you for the clarification.  Yes, I'm aware of the caveats as you describe of having environment specific artifacts produced by using maven resource filtering. 

Where I work, we debated this versus doing similar externalizing techniques as you describe--which basically meant the server environment had to be pre-provisioned with whatever properties each application/artifact being deployed requires at run-time.  Since these are all internally deployed projects within our controlled environments, we opted for the convenience and traceability via source control of having everything in the artifact that the artifact itself needs to run.

We do limit the filtering to just environment specific things like endpoint URLs, directory paths, database URLs, etc.  We do not use them to switch logic within code.  So really the argument against the "logic" being different didn't apply directly in our case.  One could make the argument that even if you externalized these settings and you accept the premise of impacting the "logic", you still are impacting the logic with externalized properties just in a different way.

We also produce environment specific builds using Jenkins matrix builds where Jenkins automatically puts each environment artifact produced from the same source tree checked out of source control into a separate Jenkins workspace named after the environment it is being built for.  This ensures that each target environment does have the same exact "logic" built into the artifact because it is built from the same source tree.  Each environment build produced by Jenkins can then be automatically deployed to the appropriate environment and needs no outside dependencies (env, module, cache, etc.) previously setup other than the target default container itself.

I do like the externalized module idea as sort of a middle ground, but I would still have to build and manage multiple modules to deploy so it really doesn't prevent the multiple builds part completely.  It certainly would require fewer builds due to environment properties not changing that often. 

We basically just opted against having a single artifact that we push around the enterprise in exchange for having everything self contained.  So far it hasn't really caused us any more or less problems than had we externalized the settings to either the server or container environment.  In the past, we did have issues where developers would go into a particular environment and manually change settings and since that was not under source control, nobody really knew who did it or why.  Now, if you change something in a particular environment, it has to be checked in and then you do have a record of it being changed and why.

One question for you or perhaps anyone else is there are certain things in Switchyard configuration like outbound service/reference calls using promotions that I'm not sure how you'd externalize those properties since those seem by their very nature embedded in the Switchyard configuration.  How would you go about externalizing some of those type of Switchyard configurations that are environment specific?

Dustin

[jorgemoralespou_2]

Hi Dustin,

Yes, you are right, it is a matter of knowing the caveats. If it works for you, then it is a fine solution. On last comment on why I prefer not to embed configuration such as endpoints inside the generated artifact, is that if you start creating dependent projects, and then deploying in ear files, it will end up being a maven configuration hell. But apart from that, and more if you have it under control, it is fine.

One question for you or perhaps anyone else is there are certain things in Switchyard configuration like outbound service/reference calls using promotions that I'm not sure how you'd externalize those properties since those seem by their very nature embedded in the Switchyard configuration.  How would you go about externalizing some of those type of Switchyard configurations that are environment specific?

I do not fully understand what you mean by this. If you mean making properties for things like implementation class and the stuff that goes coded into switchyard.xml, can be done as seen in documentation (example Property overriding):

Properties - SwitchYard - Project Documentation Editor

In this case, you can proceed the same way as you have already done.

Hope it helps,

Jorge

[dustin.barlow]

Hi Jorge,

Sorry I wasn't more clear.  However, you did answer my question though about run-time switching of Switchyard configuration (without using Maven resource filters).  So thanks for the clarification.  I had seen a hard-coded URL in the switchyard-quickstart-http-binding example on the http binding elements of the sca:reference tag and assumed that I somehow would have to replace that URL at build time.  I didn't connect the fact that Switchyard would also resolve ${variables} at run-time even for that type of configuration, which makes perfect sense now that I understand what is going on.

This is slightly off topic, but your comment about using EAR files prompts me to ask how you handle 3rd party dependencies, datasource definitions, JMS queue configuration, etc. that one would traditionally include in the EAR?

I have been using a Switchyard jar project embedded in an EAR just so I could add whatever 3rd party libraries that the Switchyard project needed to run.  Do you just push these dependent libraries out onto the server itself or are you using OSGi or similar techniques to accomplish the same?

Thank you again for your time and comments.  They have caused me to reconsider some of the ways I've been doing things.

[jorgemoralespou_2]

Hi Dustin,

Depending on what it is, it goes to the ear, or outside.

My rule is:

• If it is environment specific don't put it into EAR file (so generated artifact doesn't have to change).
• If it is application specific, put it into the EAR.

What is environment specific:

• Datasource definition
• Endpoints, environmental properties (timeouts, ...)
• JMS configuration
• Libraries shareable (modules, either static or dynamic)

What is application specific:

• Persistence unit definitions
• JMS queues definitions
• LIbraries that are not shareable (no modules defined, or no modules created)

Hope it helps,