1 2 Previous Next 29 Replies Latest reply on Mar 1, 2007 7:30 AM by dmarchant

    Configuration - normalization

    kurtstam

      Hi guys,

      Currently we have 2 config files for each esb node (server). The jbossesb-listener.xml and the jbossesb-gateway.xml.

      1. Firstly, this can result in a LOT of files if you have a few nodes.
      2. And secondly there is a lot of duplication in these files.

      I'm proposing to normalize the configuration. Basically I want to front the collection of these files with 1 master file from which we can generate the config files for each esb node. Later we can offer a bottum approach too, if we think this is needed.

      The point is that the user can debug the configuration by just looking at one file. For this file we can build an xsd (which is not possible for the individual files at the moment). The format of this file will facilitate the generation of wizzards, as they can build on organized information. Also we can easiliy generate a global deployment picture from this file.

      Here is an example XML:

      <jbossesb>
       <hosts>
       <host name="filebank" dnsName="localhost"/>
       <host name="jmsbank" dnsName="localhost"/>
       <host name="loanbroker" dnsName="localhost"/>
       <host name="jms-provider" dnsName="localhost"/>
       </hosts>
       <servers>
       <server name="fileBankServer" host="fileBank" appserver="jboss-4.0.3SP1">
       <properties>
       <property name="java.naming.provider.url" value="localhost:1099"/>
       <property name="java.naming.factory.initial" value="org.jnp.interfaces.NamingContextFactory"/>
       <property name="java.naming.factory.url.pkgs" value="org.jboss.naming:org.jnp.interfaces"/>
       </properties>
       </server>
       <server name="inhouse-jboss-4.0.4" host="jms-provider" appserver="jboss-4.0.4">
       <properties>
       <property name="java.naming.provider.url" value="localhost:1099"/>
       <property name="java.naming.factory.initial" value="org.jnp.interfaces.NamingContextFactory"/>
       <property name="java.naming.factory.url.pkgs" value="org.jboss.naming:org.jnp.interfaces"/>
       </properties>
       </server>
       <server name="loanbroker-listener" host="loanbroker" appserver="jbossesb"/>
       </servers>
       <buses>
       <bus name="bank-jms-channel" server="fileBankServer" resourceType="QUEUE" userName="" password=""/>
       <bus name="bank-ftp-channel" server="inhouse-jboss-4.0.4" resourceType="FTP" userName="kurt" password="secret"/>
       </channels>
       <buses>
       <service name="filebank-gateway" category="gateway" server="loanbroker-listener" description="This listener picks up files deposited by the fileBank"
       class="org.jboss.soa.esb.FileBankGateway">
       <listeners>
       <listener description="A File-Based listener" bus="bank-ftp-bus"/>
       <listener description="A JMS-Based listener" bus="bank-jms-bus"/>
       </listeners>
       <actions>
       <action name="TestDefaultRouteAction" process="route" class="org.jboss.soa.esb.actions.CbrProxyAction" service-category="MessageRouting"
       service-name="ContentBasedRoutingService" />
       </actions>
       </service>
       </services>
      </jbossesb>
      


      Yes I posted this xml before, but this time I replaced 'channel' by 'bus' as Mark seems to like this better :). I'm sure the Brits amongst us will want to spell buses with 3 s'es.

      Note that I'm proposing this for GA. After GA our configuration may change again (however this file may stay the same, and we could just change the xslt script to break it up).

      --Kurt

        • 1. Re: Configuration - normalization
          kconner

           

          "kurt.stam@jboss.com" wrote:
          I'm proposing to normalize the configuration. Basically I want to front the collection of these files with 1 master file from which we can generate the config files for each esb node.


          In the long run I believe we will need a more dynamic mechanism for configuration but for now I believe this is the right thing to do.

          If we can do this in time for the GA then I believe we should. It looks like it would simplify the creation of the ESB configuration and allow an easier transition for the wizards. This can only help the adoption of the ESB at this time.

          "kurt.stam@jboss.com" wrote:
          I'm sure the Brits amongst us will want to spell buses with 3 s'es.

          Nope, two s'es are fine :-)


          • 2. Re: Configuration - normalization
            marklittle

            It looks fine and we should definitely proceed with this for the GA. It will change once we move towards the container approach and I'd like to be able to offer service drop-ins for the ESB: individually deployable entities. This may mean that there's no such thing as a configuration "file" as we know it, but that's something for post GA.

            One thing though:

            <buses>
             <bus name="bank-jms-channel" server="fileBankServer" resourceType="QUEUE" userName="" password=""/>
             <bus name="bank-ftp-channel" server="inhouse-jboss-4.0.4" resourceType="FTP" userName="kurt" password="secret"/>
             </channels>
            


            Do I get points for spotting the deliberate mistake ;-)?

            • 3. Re: Configuration - normalization
              kurtstam

              I've been working with Dave and Tom, and the current design is such that we're building a ConfigurationController, which monitors the file mentioned above call jbossesb.xml. When updates to this file occur the controller reloads the file. Validates it against it's xsd (the current version can be found here: http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb.xsd
              When the document passes validation it will run through a an xslt process which generates new versions of the jbossesb-listener.xml and jbossesb-gateway.xml, which in turn will get picked up by the current Listener and Gateway controller. The idea is that this code is pretty much a facade to the current configuration.

              --Kurt

              • 4. Re: Configuration - normalization
                tfennelly

                 

                "kurt.stam@jboss.com" wrote:
                I've been working with Dave and Tom, and the current design is such that we're building a ConfigurationController, which monitors the file mentioned above call jbossesb.xml. When updates to this file occur the controller reloads the file. Validates it against it's xsd (the current version can be found here: http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb.xsd
                When the document passes validation it will run through a an xslt process which generates new versions of the jbossesb-listener.xml and jbossesb-gateway.xml, which in turn will get picked up by the current Listener and Gateway controller. The idea is that this code is pretty much a facade to the current configuration.

                --Kurt


                After looking at the XSD and what we need to generate from it, I think it would be quite difficult to do this with XSLT. So, where I'm at is - using XMLBeans to build a config model from the XSD based config and from that, generate the 2 configs we need (ala the current format).

                I think that performing this generation with anything other than raw Java would be difficult because the XSD is so highly normalised. We need to be able to iterate over and jump around the XSD based config model instance in order to generate the current "ConfigTree" type configurations. Of course this is possible in XSLT, but I think it's far easier in something like Java.

                So basically, we have a "Generator" class which looks as follows:
                public class Generator {
                
                 /**
                 * Public constructor.
                 * @param config The input configuration file.
                 * @throws ConfigurationException Bad listener ESB configuration.
                 * @throws IOException Unable to read the ESB listener configuration.
                 */
                 public Generator(File config) throws ConfigurationException, IOException {...}
                
                 /**
                 * Get the list of server names for which the configuration instance (supplied at
                 * construction)specifies configurations.
                 * @return List of server names.
                 */
                 public List<String> getServers() {...}
                
                 /**
                 * Generate the configuration set for the named server in the supplied output directory.
                 * <p/>
                 * The names of the generated files will be <i><serverName></i>-esb-config.xml
                 * (ESB Aware Listeners configuration) and <i><serverName></i>-esb-config-gateway.xml
                 * (Gateway Listeners configuration).
                 *
                 * @param serverName The server name.
                 * @param outdir The output directory to where the configuration set is to be generated.
                 * The directory will be created if it doesn't already exist.
                 * @throws ConfigurationException Failed to generate configuration set.
                 */
                 public void generate(String serverName, File outdir) throws ConfigurationException {...}
                
                }
                


                • 5. Re: Configuration - normalization
                  tfennelly

                  Hi there.

                  Kurt Dave and I have been discusing a number of issues re the config normalisation. The only bit that we still haven't decided fully on is the question of how far do we take the normalisation of the config?

                  One argument is to take normalisation as far as it can possible go and thereby remove all duplication of any type. This of course removes that family of error caused by duplication. It was also mentioned that it should make tooling easier. An example of this is outlined at the start of this thread where the configuration has 4 main sections - hosts, servers, buses/channels and services.

                  The other argument is for an approach somewhere in th middle i.e. normalising away most of the duplication we have in the current config, but living with a little and not going the whole way i.e by dropping the need to have the hosts and servers blocks (see sample config below). The thinking here is that the 100% normalised config is just a bit too difficult for a human to comprehend, because you're continually referencing across sections of the config.

                  There's no clear cut right/wrong answer to this question. Basically, we need to strick a balance between the benefits of normalisation and the need for the config to be understandable by a human. Of course, once we have tool support in this area it won't be an issue. There might also be the question of slightly different definitions of "host", "server", "bus" and "listener".

                  <?xml version = "1.0" encoding = "UTF-8"?>
                  <jbossesb xmlns="http://www.jboss.org/soa/esb/listeners/config">
                  
                   <buses>
                   <jms-bus busid="server1-jms"
                   connection-factory="ConnectionFactory"
                   jndi-context-factory="org.jnp.interfaces.NamingContextFactory"
                   jndi-URL="localhost" />
                   </buses>
                   <services>
                   <service category="Bank" name="Reconciliation" description="Bank Reconciliation Service">
                  
                   <listeners>
                   <jms-listener name="Bank-JMS-Gateway"
                   is-gateway="true"
                   busidref="server1-jms"
                   destination-type="QUEUE"
                   destination-name="queue/A"
                   message-selector="service='Reconciliation'"
                   maxThreads="1"
                   />
                   <jms-listener name="Bank-Listener"
                   is-gateway="false"
                   busidref="server1-jms"
                   destination-type="TOPIC"
                   destination-name="queue/B"
                   message-selector="service='Reconciliation'"
                   maxThreads="2"
                   />
                   </listeners>
                  
                   <actions>
                   <action name="TestDefaultRouteAction" process="route" class="org.jboss.soa.esb.actions.CbrProxyAction">
                   <property name="propName" value="propValue" />
                   </action>
                   </actions>
                  
                   </service>
                   </services>
                  
                  </jbossesb>
                  


                  The current XSD can be found here.

                  • 6. Re: Configuration - normalization
                    kurtstam

                    So I think the point Tom is making is 'readability' of the configuration. And that the readabilty increases by not having to 'jump' through the file. We seem to agree that normalization by itself is a good thing :). So that said, I actually like to hide the details (pull out duplicate info) and give is human readable names.

                    For example when I reference the jndi-* parameter I can care less. I'd rather see the servername like "jboss-4.0.4" or something. I guess my brain likes to see the big picture first.. I don't think tooling affects this, although tooling will be easier when the thing is fully normalized.

                    --Kurt

                    • 7. Re: Configuration - normalization
                      kurtstam

                      So to stay with the example - assuming the gateway and the listener use the same Queue, I would take this info and add it to the bus. And whatever is in the bus *now* I think should go in a server. :)

                      • 8. Re: Configuration - normalization
                        tfennelly

                        Kurt's brain and mine are very different I think :-) I wouldn't see the big OR small picture when the config becomes over fragmented with everything referencing everything else. You end up replacing duplication of the actual config parameters with duplication of reference ids all over your config, while not being able to see the wood from the trees :-)

                        I'm looking forward to hearing what others think about this. In th mean time, I'm off home :-)

                        • 9. Re: Configuration - normalization
                          kurtstam

                          When I read 'server' I see a box' when I read 'bus/channel' I see a line. I don't like to be bothered by the details :) and I certainly don't want to have those details in my file more then once :).

                          • 10. Re: Configuration - normalization
                            kurtstam

                            Oh and a server box goes around small service boxes and the connections go between those services. Sorry I mis-typed there.

                            • 11. Re: Configuration - normalization
                              marklittle

                              Configuration duplication leads to errors. Can we take an approach that reduces that?

                              • 12. Re: Configuration - normalization
                                tfennelly

                                 

                                "mark.little@jboss.com" wrote:
                                Configuration duplication leads to errors. Can we take an approach that reduces that?


                                I think that's what we're trying to do :-) We're just trying to find a balance here.

                                My point would be 2-fold in that you will always have duplication of one sort or another...

                                if not normalised enough - excessive duplication of the actual config params

                                if normalised to the nth degree - excessive duplication of reference id's etc, without a system to manage this referential integrity (unlike a DB eg).

                                ... adding to that the fact that normalisation to the nth degree will definitely lead to fragmentation of the config on a grand scale => difficult to comprehend (perhaps not for all people ;-) ).

                                So sure, we want to reduce errors caused by duplication, but surely we don't want to introduce another family of errors :-)

                                • 13. Re: Configuration - normalization

                                  Luckily I can just keep using the base xml config files a la "old style" ;-)

                                  • 14. Re: Configuration - normalization
                                    marklittle

                                    Having helped to develop some fairly complex schemas in various WS-* specifications, my preference is for references versus configuration duplication. Tools can and will sort this out eventually.

                                    1 2 Previous Next