11 Replies Latest reply on Jun 10, 2012 2:06 PM by erhard

    Generated config considered harmful?

    kcbabo

      Figured I would throw this subject out there and see what kind of arguments we can get in.  At present, we support the generation of configuration based on annotations in the user's project structure.  This means that the effective configuration of an application consists of two pieces:

       

      1) The "source" config.  This is the SCDL specified directly in src/main/resources/META-INF/switchyard.xml.

      2) The "generated" config.  This is what we generate off of annotations in the project.

       

      At build time, #1 and #2 are merged to become the effective configuration of the application.  This config is placed in target/classes/META-INF/switchyard.xml.

       

      The above model works fine, but there are a few cases where I think it's suboptimal:

      • Quickstarts/Examples - I believe that users want to see the "whole picture" when they look at an example.  As things stand now, someone looking at a quickstart needs to build the quickstart and then know to look in target/classes instead of src/main/resources to see the effective configuration for the application.
      • Tooling - our experience with the Eclipse application modeler suggests to me to that having two separate configurations will create no end of confusion in the tooling.  If the application model includes only user config, then you won't see generated stuff, but everything in the picture is editable.  If the application model includes generated config, then you 'see' the entire application, but certain pieces of the model cannot be edited. 

       

      I like the idea of generated config, but I'm beginning to think it's counterproductive in the cases outlined above.  Therefore, I suggest we switch to a model where tooling generates user configuration for everything.  I also suggest we update all of our quickstarts to include a complete user configuration.  Keep in mind, that generated config is still available as an option, but we won't default to that in our quickstarts or our Eclipse/Forge tooling.

       

      Thoughts?  ;-)

        • 1. Re: Generated config considered harmful?
          jeffdelong

          You probably wrote this after answering yet another email from me where I looked at the switchyard.xml and could not find the bean services and forgot about the generated one in the target directory. This is indeed confusing. And hand editing switchyard.xml is also challenging (as evidenced by the number of emails from me on how to add interfaces, possible options, etc.). Right now forge only goes so far, and even knowing the right set of forge commands is not easy. You can list them, but you can't tell the options that go with them, e.g., type xml on the camel-service create. I think that tooling (in particular Eclipse based graphical tooling) is the way to generate the configuration, and that there should be a single switchyard.xml which includes generated elements, e.g., bean service configuration.

           

          The complete switchyard.xml should still be visible in a source panel, as it is often interesting to view the configuration model. And there may be times when a user has to modify it though I would not want to have to do that, as it carries the risk of breaking something. This is similar to the BPEL editor. I can see the BPEL xml, but I never modify it directly.

          • 2. Re: Generated config considered harmful?
            rcernich

            I know I'm just asking for the big hurt...

             

            I really like the way the system works today.  That said, I think there are areas where it could be improved, but I wouldn't advocate for ditching it altogether.

             

            As for the highlighted shortcomings, in my opinion, those are simply problems that need to be solved.  The tooling should be able to use both files when rendering SwitchYard project details.

             

            The first issue can be accommodated in several ways, e.g. a project explorer extension, similar to other JEE project types.  The project explorer content can be used as an access point for opening the application's configuration.

             

            As for the second issue, the source file can be hidden from the user so that they only have access to the "merged" content.  The editor may become more complicated, but I think that's a problem worth solving.

             

            One of the things I like about the current mechanism is that it is relatively easy to get up and running without having to edit a configuration file.  While more sophisticated tooling may ease the burden, I'm going to guess any tooling solution will not be as easy as adding @Reference to a field in your Java component.  Having said that, I would actually advocate for more sophisticated scanners that would automatically add references for other implementations types (e.g. Camel, Rules, etc.) and that would scan native files (e.g. scan bpmn instead of annotations in supporting classes).

             

            I think it may be beneficial to take a step back and look at what exactly you are creating and how you are creating it.

            • Components - may be defined in switchyard.xml or through a scanner (e.g. @Service).
            • Component references - for some implementation types it must be defined in switchyard.xml, for others a scanner can generate them (e.g. @Reference)
            • Policies - may be defined in switchyard.xml or through a scanner.
            • Composite service/reference - must be defined in switchyard.xml.

             

            If we walk that list, I think for most of the component related items, the natural place to specify the information is in the implemetation itself.  Think about your workflow if there were no annotation scanner available for @Reference or @Service.  You would need to switch back and forth between two editors (Java and SwitchYard) to specify that information.  Then, if you forgot, your application would be broken (yes, a validator would catch this, but...).  We'd be adding a lot of clicks and creating more configuration work for the developer.

             

            All in all, I don't think we should shy away from this just because it makes developing the tools harder.  The better the tools, the better the developer experience, the better for SwitchYard.

             

            Just my two cents.

            • 3. Re: Generated config considered harmful?
              kcbabo

              @Jeff : it's definitely not just you. :-)  I think your recent experience has definitely brought more attention to the issue, but this has been something that's been simmering in the background for a while.  We're not far from a preview of the Eclipse-based editor and I'm pretty confident that we will be able to pull everything together there.  Forge and annotation generation are useful tools, but they are pieces of a solution vs. a solution in and of themselves.

              • 4. Re: Generated config considered harmful?
                jeffdelong

                As I have been finding out working on a PoC is that it is not really easy to get up and running without having to edit a configuration file. In fact all of the applications I have created (three so far, fourth on the way) require editing of the configuration file. Forge only goes so far, and annotations are only available for bean services. Rule services, bpm services, bpel services, and camel services all require the configuration file to be edited. And bean services will rarely exist on their own. Most real world services use complex xml messages that require xsl transformation prior to invoking any Java business logic.

                 

                I don't currently add the annotations to the Java classes. The Java classes are generated by forge with their annotation included (at least the @service annotation). In an Eclipse environment I would expect the same to happen, that the tooling would generate the bean service class with whatever annotations are required. This leaves me as a developer to add the business logic.

                 

                I also don't agree that specifying Policies in the implementation is really best either. I think policies should be added outside the implementation, as they are often not the responsibility of the implementation developer, and can vary based on where the service is deployed. I also don't think the interface or binding should be specified in the implementation. I.e, if I wanted to tie my service binding to soap, I could just use JSR-181 annotations. The benefit of the Switchyard approach is that this is external to and not the concern of the implementation.

                 

                It is not clear to me how scanners would work for some of the implementation types. A scanner could work on a rule file, however Drools has many different ways to author rules.

                 

                I will also repeat that I think it is a good thing to be able to view the entire application (and all of its configuration) both graphically as well as in the switchyard.xml. That some of the configuration information was generated from annotations in the implemention class is fine, as long as I don't have to open the Java class to understand them.

                • 5. Re: Generated config considered harmful?
                  kcbabo

                  @Rob : Good feedback.  Just to be clear, no one is saying we should "ditch" existing functionality.  I mentioned in my original post that config generation is useful and that it should remain an option no matter what we decide to do with the tooling and examples.

                   

                  I don't believe the first issue is addressed by a project navigator.  I believe the visual application model should represent the complete configuration of the application, not a combination of the visual model and a project navigator.  The project navigator is definitely useful, I'm just saying that it does not obviate the need for a complete application view in the visual model.

                   

                  Re: your other comments, I want to reiterate that I'm only suggesting behavior for examples and tooling-generated projects.  If you're using the SCA visual representation to create a service component, then you don't have to mess with XML at all.  It's no different from annotation-driven generation in that respect, except it generates configuration at creation-time instead of build-time.

                  • 6. Re: Generated config considered harmful?
                    kcbabo

                    Jeff DeLong wrote:

                     

                    I also don't agree that specifying Policies in the implementation is really best either. I think policies should be added outside the implementation, as they are often not the responsibility of the implementation developer, and can vary based on where the service is deployed. I also don't think the interface or binding should be specified in the implementation. I.e, if I wanted to tie my service binding to soap, I could just use JSR-181 annotations. The benefit of the Switchyard approach is that this is external to and not the concern of the implementation.

                     

                    Yep.

                     

                    I will also repeat that I think it is a good thing to be able to view the entire application (and all of its configuration) both graphically as well as in the switchyard.xml. That some of the configuration information was generated from annotations in the implemention class is fine, as long as I don't have to open the Java class to understand them.

                     

                    Agreed.  Ultimately, we are saying that there are multiple ways to get this done.  I believe that examples and tooling-driven application development should put everything in the source configuration.  That will work best for the types of people that tend to use examples and tooling, IMHO.  For users that like to go straight to the source editor and drive configuration from annotations and the like, that will still work too.  Developers have choices when they develop these applications and can go with the approach which works best for them.

                     

                    The end goal is that users *never* have to edit XML directly.  I'm not saying they can't if they want to, but we should never force a user to go there.

                    • 7. Re: Generated config considered harmful?
                      rick_wagner

                      Hi All,

                       

                      I'm glad to see we're putting some brain power into these questions.  This goes straight to the heart of usability, and in the long run Simplicity is going to win the game.

                       

                      I don't have any critiques of the current setup, because I am woefully ignorant of the current state of things.  (I regret this.)  Time just doesn't allow it.

                       

                      But I would like to throw in a few salient thoughts:

                      1)  Dev sites regularly post articles showing how to configure Camel in either Java or Spring dialect in clear and concise ways.  It's not a direct competing product, but close enough that we probably ought to be in the ballpark usability-wise.

                      2)  The Mule guys have been putting up 'Mule School' articles that show drop-dead simple drag-and-drop examples through tooling.  Again, I'd think we should regard this as our bogey.  They really do make it look easy.

                       

                      So.... if we're requiring hand-editing of something, it's to our disadvantage if it's not as easy as Camel.  And if there are things being generated behind the scenes, I'd say that's fine so long as nobody has to *ever* look at them.  (But if somebody does-- like support folks-- then I hope we log the heck out of what's configured at runtime so it's easy to see what's been gen'ed without having to dig through arcane artifacts.)

                       

                      My .02 worth.  Keep up the good work, please make it Simple....

                       

                      Rick

                      • 8. Re: Generated config considered harmful?
                        kcbabo

                        Good points, Rick.  Simple is definitely a goal here, but another goal is to have the model scale from basic example applications to complex enterprise applications.  Keep the feedback coming as it really helps us stay focused on the usability front.

                         

                        cheers,

                        keith

                        • 9. Re: Generated config considered harmful?
                          rcernich

                          I suppose you could argue either way, but I think things like component references are probably more easily defined within the context of the component itself (e.g. @Reference for a bean service, SwitchYard service in BPMN, etc.).  I'm not sure what the benefit is to defining component references in the SCA editor if they are not actually used by the component.  Moreover, I think it ends up being an extra step to have to define the reference manually.  I'm sure we could have a validator that generates an error if a reference is not defined in the SCA model, the user would then have to fix the problem.  If we're going to do that, why not generate the config to begin with?

                          • 10. Re: Generated config considered harmful?
                            kcbabo

                            It could definitely be argued either way and the good news is that we're not shutting the door on any one option.

                             

                            The benefit of defining references in the model vs. part of the implementation is that references are an abstract concept that transcends implementation types.  Could we wire up scanners/generators for every implementation type to produce the same thing at the end of the day?  Yeah, we could do that.  But I don't see it offering a significant advantage over creating the reference directly in the tooling.  I also do like the idea of warnings for missing/unused references in the model mainly because it's a check that something is unaccounted for.  In other words, the expected design of the application does not match the implementation.

                            • 11. Re: Generated config considered harmful?
                              erhard

                              Maybe one should question how much configuration is necessary at compile/package time at all. A very nice feature in Camel is that you can do something like this:

                               

                              {code}

                                        public void configure() {

                               

                                                  RouteDefinition distributor = from("switchyard://MyService");

                                                  for ( int i = 0; i < 5; ++i ){

                                                            String destination = "file://output" + i;

                                                            distributor = distributor.to(destination);

                                                  }

                                        }

                              {code}

                               

                              Since I don't have to configure the "file://..." endpoints anwhere at compile-time I am very flexible in defining the routes at runtime (startup) within my code.

                              I just started with switchyard, but it looks like it would not be possible to use Switchyard references instead of the Camel file-endpoints, since I have to define all references in switchyard.xml? When it gets more complicated, I would probably end up writing my own configuration generator and maybe I even need different deployments in different (staging-) environments?