Socket definitions
brian.stansberry Jul 7, 2010 11:44 PMWhile at JBW, David and I spent some time working on how sockets should be represented in the domain schema, and I wanted to throw what we came up with out there for broader input.
XSD behind what I'll describe here can be found in my topic branch for AS 7 schema development at http://github.com/bstansberry/jboss-as/tree/schema-dev/domain/src/main/resources/schema
The key thing here is we want to:
1) Declare configuration of services that open sockets at the domain level
2) Declare what address sockets bind to at the server level, without having to repeat the service configuration and without relying on system property substitution
3) Make it simple for a server to override the ports for a set of sockets to avoid port conflicts with another server that binds to the same interface.
The first piece of the puzzle is at the domain level, a set of "named interfaces" are declared at the domain level in domain.xml. The only required configuration at this level is the interface name. Optionally, criteria for how to determine the IP address to use for the interface can be declared, but this isn't required (since machine-specific things like addresses are meant to be configured in host.xml.
<!-- Named interfaces that can be referenced elsewhere. Different mechanisms for associating an IP address with the interface are shown. --> <interfaces> <interface name="default"> <!-- Here we directly state the IP address --> <inet-address value="127.0.0.1"/> </interface> <interface name="external"> <!-- Here we name the NIC; the IP address associated with NIC whose byte representation has the lowest value is chosen --> <nic name="eth1"/> </interface> <interface name="internal"> <!-- Here we provide a set of criteria that are used to narrow the set of IP addresses available on the host to one that is acceptable. --> <not><loopback/></not> <point-to-point/> <multicast/> <up/> <private-address/> <!-- Alternatively... <public-address/> --> <nic name="eth0"/> <nic-match value="eth[0-9]+"/> <!-- The 'any' element means the included criteria are not exclusive --> <any> <subnet-match value="10.0.0.0/8"/> <subnet-match value="192.168.0.0/16"/> </any> </interface> <!-- For these next two, no address selection criteria is provided, so an override at the host or server level is required. --> <interface name="public3"/> <interface name="public4"/> </interfaces>
As the example above shows, there are a couple mechanisms for defining the IP address; either by directly declaring the address, or by providing criteria that can be applied to the set of addresses available on the host to find the best match. The latter is particularlly important on the cloud where addresses are not known in advance.
In host.xml, the interface definitions can be overridden either at the host-wide level or per-server:
<host xmlns="urn:jboss:host:1.0"> <interfaces> <interface name="public3"> <inet-address value="10.0.0.1"/> </interface> </interfaces> <servers> <server name="server-one" group="main-server-group" start="false"> <interface-specs> <!-- public4 address wasn't defined in domain or host so we have to define it at server level --> <interface name="public4"> <inet-address value="192.168.100.1"/> </interface> </interface-specs>
Interface names are used in named socket declarations at the domain level in domain.xm. The socket declarations combine an interface name with the relevant port, plus multicast information if the socket is a multicast socket. The same socket name can appear in different "socket groups", in case some environments in the domain want to use different interfaces or ports:
<socket-groups> <socket-group name="standard-sockets" default-interface="external"> <socket name="http" port="8080"/> <socket name="https" port="8447"/> <socket name="remoting" port="4447"/> </socket-group> <socket-group name="standard-clustered-sockets" extends="standard-sockets"> <socket name="cluster-udp" interface="internal" port="" multicast-port="55200"/> <socket name="cluster-failure-detection" interface="internal" port="54200"/> <socket name="ha-jndi" port="1100"/> <socket name="ha-jndi-discovery" multicast-port="1102"/> </socket-group> <!-- Some servers want all sockets bunched together --> <socket-group name="compact-sockets" default-interface="external"> <socket name="http" port="10001"/> <socket name="https" port="10002"/> <socket name="remoting" port="10003"/> </socket-group> </socket-groups>
In the server-group, the socket-group to use can optionally be declared. This serves as an overridable default for all servers in the group.
<server-group group-name="main-server-group" profile="something"> <socket-group name="standard-clustered-sockets"/>
If not declared at the server-group level, the socket-group must be declared for each server:
<server name="server-x" group="main-server-group" start="false"> <socket-group name="standard-clustered-sockets"/>
The domain level elements that actually involve sockets shouldn't specify addresses/ports in their configuration. Instead they should reference a socket by name:
<containers> <web-containers> <web-container name="jboss.web"> <http-connector id="httpConnector" socket="http" scheme="http" default-virtual-server="localhost"/>
Avoiding port conflicts
If multiple servers are run on the same host, they have 3 mechanisms to avoid port conflicts:
1) A server changes the address associated with an interface name so it's distinct from the others.
2) A server uses the socket-group element's "port-offset" attribute to change the port values for the sockets in its group
3) A server changes the socket-group it's using to a group that has different interfaces or ports (this approach would be unusual)
<servers> <server name="server-one" group="main-server-group"> <!-- server-one inherits the default socket-group declared in the server-group --> </server> <server name="server-two" group="main-server-group" start="false"> <!-- server-two avoids port conflicts by incrementing the ports in the default socket-group declared in the server-group --> <socket-group name="standard-clustered-sockets" port-offset="150"/> </server> <server name="server-three" group="main-server-group"> <!-- server-three inherits the default socket-group declared in the server-group but avoids port conflicts with server-one by using a different IP for the relevant interfaces --> <interface-specs> <interface name="external"> <inet-address value="4.5.6.7"/> </interface> <interface name="internal"> <inet-address value="10.0.0.3"/> </interface> </interface-specs> </server> <server name="server-four" group="main-server-group"> <!-- server-four avoids port conflicts by using a socket-group with distinct ports --> <socket-group name="compressed-sockets"/> </server> </servers>