1 2 Previous Next 19 Replies Latest reply on Aug 10, 2010 6:32 AM by emuckenhuber

    Socket definitions

    brian.stansberry

      While at JBW, David and I spent some time working on how sockets should be represented in the domain schema, and I wanted to throw what we came up with out there for broader input.

       

      XSD behind what I'll describe here can be found in my topic branch for AS 7 schema development at http://github.com/bstansberry/jboss-as/tree/schema-dev/domain/src/main/resources/schema

       

      The key thing here is we want to:

       

      1) Declare configuration of services that open sockets at the domain level

      2) Declare what address sockets bind to at the server level, without having to repeat the service configuration and without relying on system property substitution

      3) Make it simple for a server to override the ports for a set of sockets to avoid port conflicts with another server that binds to the same interface.

       

      The first piece of the puzzle is at the domain level, a set of "named interfaces" are declared at the domain level in domain.xml. The only required configuration at this level is the interface name. Optionally, criteria for how to determine the IP address to use for the interface can be declared, but this isn't required (since machine-specific things like addresses are meant to be configured in host.xml.

       

          <!--  
               Named interfaces that can be referenced elsewhere. Different
               mechanisms for associating an IP address with the interface
               are shown.
          -->
          <interfaces>
              
              <interface name="default">
                  <!--  Here we directly state the IP address -->
                  <inet-address value="127.0.0.1"/>
              </interface>
              <interface name="external">
                  <!--  
                      Here we name the NIC; the IP address associated with NIC whose
                      byte representation has the lowest value is chosen
                  -->
                  <nic name="eth1"/>
              </interface>
              <interface name="internal">
                  <!-- 
                      Here we provide a set of criteria that are used to narrow the
                      set of IP addresses available on the host to one that is acceptable.
                   -->
                  <not><loopback/></not>
                  <point-to-point/>
                  <multicast/>
                  <up/>
                  <private-address/>
                  
                  <!-- Alternatively... 
                  <public-address/>
                  -->
                  <nic name="eth0"/>
                  <nic-match value="eth[0-9]+"/>
                  <!-- The 'any' element means the included criteria are not exclusive -->
                  <any>
                      <subnet-match value="10.0.0.0/8"/>
                      <subnet-match value="192.168.0.0/16"/>
                  </any>
              </interface>
              <!-- 
                   For these next two, no address selection criteria is provided, so 
                   an override at the host or server level is required. 
              -->
              <interface name="public3"/>
              <interface name="public4"/>
          </interfaces>

       

      As the example above shows, there are a couple mechanisms for defining the IP address; either by directly declaring the address, or by providing criteria that can be applied to the set of addresses available on the host to find the best match. The latter is particularlly important on the cloud where addresses are not known in advance.

       

      In host.xml, the interface definitions can be overridden either at the host-wide level or per-server:

       

      <host xmlns="urn:jboss:host:1.0">
      
          <interfaces>
              <interface name="public3">
                  <inet-address value="10.0.0.1"/>
              </interface>
          </interfaces>
      
          <servers>
              <server name="server-one" group="main-server-group" start="false">
                  
                  <interface-specs>
                      <!-- public4 address wasn't defined in domain or host so
                           we have to define it at server level -->
                      <interface name="public4">
                          <inet-address value="192.168.100.1"/>
                      </interface>
                  </interface-specs>
      

       

      Interface names are used in named socket declarations at the domain level in domain.xm. The socket declarations combine an interface name with the relevant port, plus multicast information if the socket is a multicast socket. The same socket name can appear in different "socket groups", in case some environments in the domain want to use different interfaces or ports:

       

          <socket-groups>
               <socket-group name="standard-sockets" default-interface="external">
                  <socket name="http" port="8080"/>
                  <socket name="https" port="8447"/>
                  <socket name="remoting" port="4447"/>
               </socket-group>
               <socket-group name="standard-clustered-sockets" extends="standard-sockets">
                  <socket name="cluster-udp" interface="internal" port="" multicast-port="55200"/>
                  <socket name="cluster-failure-detection" interface="internal" port="54200"/>
                  <socket name="ha-jndi" port="1100"/>
                  <socket name="ha-jndi-discovery" multicast-port="1102"/>
               </socket-group>
               <!-- Some servers want all sockets bunched together -->         <socket-group name="compact-sockets" default-interface="external">
                  <socket name="http" port="10001"/>
                  <socket name="https" port="10002"/>
                  <socket name="remoting" port="10003"/>
               </socket-group>
          </socket-groups>

       

      In the server-group, the socket-group to use can optionally be declared. This serves as an overridable default for all servers in the group.

       

          <server-group group-name="main-server-group" profile="something">
      
                 <socket-group name="standard-clustered-sockets"/>
      

       

      If not declared at the server-group level, the socket-group must be  declared for each server:

       

          <server name="server-x" group="main-server-group" start="false">
                  
                  <socket-group name="standard-clustered-sockets"/>
      
      

       

      The domain level elements that actually involve sockets shouldn't specify addresses/ports in their configuration. Instead they should reference a socket by name:

       

      <containers>
          <web-containers>
              <web-container name="jboss.web">
                  <http-connector id="httpConnector" socket="http" scheme="http" default-virtual-server="localhost"/>

       

      Avoiding port conflicts

       

      If multiple servers are run on the same host, they have 3 mechanisms to avoid port conflicts:

       

      1) A server changes the address associated with an interface name so it's distinct from the others.

      2) A server uses the socket-group element's "port-offset" attribute to  change the port values for the sockets in its group

      3) A server changes the socket-group it's using to a group that has  different interfaces or ports (this approach would be unusual)

       

          <servers>
              <server name="server-one" group="main-server-group">
      
                  <!-- server-one inherits the default socket-group declared in the server-group -->
      
              </server>
              <server name="server-two" group="main-server-group" start="false">
                  
                  <!-- server-two avoids port conflicts by incrementing the ports in 
                       the default socket-group declared in the server-group -->
                  <socket-group name="standard-clustered-sockets" port-offset="150"/>
      
              </server>
              <server name="server-three" group="main-server-group">
                  
                  <!-- server-three inherits the default socket-group declared in 
                       the server-group but avoids port conflicts with server-one
                       by using a different IP for the relevant interfaces -->
                  <interface-specs>
                      <interface name="external">
                          <inet-address value="4.5.6.7"/>
                      </interface>
                      <interface name="internal">
                          <inet-address value="10.0.0.3"/>
                      </interface>
                  </interface-specs>
      
              </server>
      
              <server name="server-four" group="main-server-group">
                  <!-- server-four avoids port conflicts by using a socket-group with distinct ports -->
                  <socket-group name="compressed-sockets"/>
              </server>
          </servers>
      
        • 1. Re: Socket definitions
          brian.stansberry

          The thing I dislike about the above is the separation of the socket specification from the configuration of whatever is using it. It's more intuitive to have that as part of, e.g. the container configuration, but using a consistent XML type. The socket group for a particular server group is then the union of all sockets declared in the various containers associated with the server group.

           

          What's lost with that though is:

           

          1) Ability for a named socket to appear in two different socket-groups, with different interface/port bindings. Different servers/server groups could pick a different socket-group to get different bindings appropriate for their environment. They could only change the IP address / interface mappings or use the port-offset to change ports. Not sure if this is a real loss though.

           

          2) The AS will include a socket factory service; many services will get their sockets (or at least the configuration values for them) from this service. The configuration choices discussed on this thread will end up driving the configuration object used by this service. End user deployments may also wish to inject this service and delegate socket management to it, thereby taking advantage of the full capabilities of the domain API to manage their sockets. If so, we'd still need some sort of socket-group element, as that's where custom end-user socket configurations would go.  Having a socket-group element for end-user stuff plus socket definitions dispersed throughout the rest of the config -- complex and confusing.

           

          Any opinions?

          • 2. Re: Socket definitions
            dmlloyd

            Brian Stansberry wrote:

             

            The thing I dislike about the above is the separation of the socket specification from the configuration of whatever is using it. It's more intuitive to have that as part of, e.g. the container configuration, but using a consistent XML type. The socket group for a particular server group is then the union of all sockets declared in the various containers associated with the server group.

             

            What's lost with that though is:

             

            1) Ability for a named socket to appear in two different socket-groups, with different interface/port bindings. Different servers/server groups could pick a different socket-group to get different bindings appropriate for their environment. They could only change the IP address / interface mappings or use the port-offset to change ports. Not sure if this is a real loss though.

             

            2) The AS will include a socket factory service; many services will get their sockets (or at least the configuration values for them) from this service. The configuration choices discussed on this thread will end up driving the configuration object used by this service. End user deployments may also wish to inject this service and delegate socket management to it, thereby taking advantage of the full capabilities of the domain API to manage their sockets. If so, we'd still need some sort of socket-group element, as that's where custom end-user socket configurations would go.  Having a socket-group element for end-user stuff plus socket definitions dispersed throughout the rest of the config -- complex and confusing.

             

            Any opinions?

            Let's just keep it separate.  It seems unintuitive now, but I think once people get the hang of it it might actually be better: everything network-related is in one place.  It maps nicely to the OSI model, if that's any kind of endorsement

            • 3. Re: Socket definitions
              dmlloyd

               

              <containers>
                  <web-containers>
                      <web-container name="jboss.web">
                          <http-connector id="httpConnector" socket="http" scheme="http" default-virtual-server="localhost"/>

               


               

              I really don't like "socket" as the name of this concept.  Maybe "binding"?  "network-endpoint"?  "binding-configuration"?

              • 4. Re: Socket definitions
                brian.stansberry

                I assume you mean throughout, not just the particular attribute name in that snippet. If so, I like network-endpoint best, as "binding" could mean anything and thus wouldn't be such a good name for the socket-group, socket elements.

                • 5. Re: Socket definitions
                  dmlloyd

                  Maybe "socket-binding" would be better still.  I'm just thinking that "endpoint" is already pretty overloaded between WS and Remoting.

                  • 6. Re: Socket definitions
                    brian.stansberry

                    Sold!

                    • 7. Re: Socket definitions
                      brian.stansberry

                      I pushed a rename of the socket-xxx elements to socket-binding-xxx a couple days ago in my schema-dev branch at http://github.com/bstansberry/jboss-as/tree/schema-dev/ . David/Alexey could one of you have a look; if OK I'll push to upstream.

                      • 8. Re: Socket definitions
                        dmlloyd

                        Brian Stansberry wrote:

                         

                        I pushed a rename of the socket-xxx elements to socket-binding-xxx a couple days ago in my schema-dev branch at http://github.com/bstansberry/jboss-as/tree/schema-dev/ . David/Alexey could one of you have a look; if OK I'll push to upstream.

                         

                         

                        Looks OK to me.

                        • 9. Re: Socket definitions
                          brian.stansberry

                          Thanks; I pushed that to upstream.

                          • 10. Re: Socket definitions
                            trustin

                            A little bit like a material for a separate thread, but perhaps we could allow specifying MAC address in the <nic/> tag?

                             

                                <nic hwaddr="00:12:34:56:78:9a"/>
                            
                            • 11. Re: Socket definitions
                              brian.stansberry

                              Makes sense. Thanks!

                              • 12. Re: Socket definitions
                                brian.stansberry

                                I'm going to change <private-address/> to <site-local-address/> and <link-local-address/>. This matches the properties on InetAddress. The <public-address/> tag is basically shorthand for <not><site-local-address/><link-local-address/></not>

                                • 13. Re: Socket definitions
                                  dmlloyd

                                  Trustin Lee wrote:

                                   

                                  A little bit like a material for a separate thread, but perhaps we could allow specifying MAC address in the <nic/> tag?

                                   

                                      <nic hwaddr="00:12:34:56:78:9a"/>
                                  

                                  Good idea - it should also support partial matches as well (since NICs are organized with manufacturer codes it might help choose one by prefix, when applied to multiple similar machines).

                                  • 14. Re: Socket definitions
                                    dmlloyd

                                    Brian Stansberry wrote:

                                     

                                    I'm going to change <private-address/> to <site-local-address/> and <link-local-address/>. This matches the properties on InetAddress. The <public-address/> tag is basically shorthand for <not><site-local-address/><link-local-address/></not>

                                     

                                    Ah I didn't even think of using the properties on InetAddress.  Good idea.

                                    1 2 Previous Next