Per-host configuration

[dmlloyd]

In an IRC chat, Brian and I were discussing the idea of per-host configuration and how it could work.  A couple things that came up:

1. Per-host configuration could be produced by each host and consumed by the domain controller (DC), as opposed to the domain configuration which is produced by the DC and consumed by the hosts.
2. This implies that the host configuration is separate; so each host will have a domain.xml and a host.xml, and the DC will have the master domain.xml plus one host.xml for every host it knows about.
3. The DC should be able to perform admin operations against hosts; however in the event a known host is down or unavailable, the per host management operations for that host may be read-only; alternatively, a strategy may be worked out to merge configurations when the host comes back on line (this is more complex though).
4. Requirement: a user should be able to copy an installed image from one host to another and start it up without any unexpected behavior.
5. Requirement: we should have a way to create a "bootstrap" environment, which would consist of an installed image with a server configured which acts as the DC.  (This is working on the assumption that the DC is a server role of some sort.)
6. The host configuration contains a list of servers to start, each of which is assigned to a server group.  This is where any JVM configuration will go, as well as any OS-specific stuff (setting "nice" values, CPU or other resource affinity, etc).
7. The host configuration contains mapping of domain concepts (i.e. named network interfaces) to physical resources (i.e. an actual IP address).  Note that the IP address case needs some additional discussion (i.e. Bela's cloud case where the IP is detected somehow).
8. The service binding manager-equivalent service will be involved in ensuring a distinct network configuration for each server.

There's still more to figure out but I think this is the basic framework we're dealing with.

[brian.stansberry]

In the Domain Server Lifecycle Use Cases article, there are number of places where I discuss alternatives for how to provide information to a launching process -- either via command line params, or via a reference to a local domain.xml copy from which the information is parsed. If we go with what David outlines, essentially, those references to a "domain.xml" can be replaced with "host.xml".

This move divides our "database" of domain information into two pieces: service/deployment configuration and a mapping thereof onto named sets (i.e. ServerGroups). And then physical topology information.  (3 pieces really counting the content repository).  The two are distinct because the rules for who is authoritative vary between the two. The DC is always authoritative for service/deployment configuration. For a new unknown-to-the-DC host that is joining the domain, the DC initially can't be authoritative for the physical topology information; it has to come from the joiner.

[brian.stansberry]

David Lloyd wrote:

The DC should be able to perform admin operations against hosts; however in the event a known host is down or unavailable, the per host management operations for that host may be read-only; alternatively, a strategy may be worked out to merge configurations when the host comes back on line (this is more complex though).

The general process of synchronizing between the DC and a ServerManager/Server will probably involve tracking and persisting metadata about content. For example, the hash of a domain.xml file or any host.xml. So, one mechanism to allow configuration via the DC of an offline host is for the DC to retain the hash of the host.xml last presented by the host. Following modification by an admin, the DC also has a new hash for the current state. If the host starts again and presents the old hash, the DC knows the host is out of date and can push the new host.xml content to it.

If the restarted host presents a hash that is different from the last one it presented (i.e. someone modified the host's copy of host.xml directly) then the DC can log a WARN and accept the host's content.

[dmlloyd]

Here's a first cut at a host config XSD:

<?xml version="1.0" encoding="UTF-8"?>
<!--
  ~ JBoss, Home of Professional Open Source.
  ~ Copyright 2010, Red Hat Middleware LLC, and individual contributors
  ~ as indicated by the @author tags. See the copyright.txt file in the
  ~ distribution for a full listing of individual contributors.
  ~
  ~ This is free software; you can redistribute it and/or modify it
  ~ under the terms of the GNU Lesser General Public License as
  ~ published by the Free Software Foundation; either version 2.1 of
  ~ the License, or (at your option) any later version.
  ~
  ~ This software is distributed in the hope that it will be useful,
  ~ but WITHOUT ANY WARRANTY; without even the implied warranty of
  ~ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
  ~ Lesser General Public License for more details.
  ~
  ~ You should have received a copy of the GNU Lesser General Public
  ~ License along with this software; if not, write to the Free
  ~ Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
  ~ 02110-1301 USA, or see the FSF site: http://www.fsf.org.
  -->
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
            targetNamespace="urn:jboss:host:1.0"
            xmlns="urn:jboss:host:1.0"
            elementFormDefault="qualified"
            attributeFormDefault="unqualified"
            version="1.0">
    <xs:element name="host-config" type="host-config"/>
    <xs:complexType name="host-config">
        <xs:all>
            <xs:element name="domain-controller" type="location" minOccurs="1"/>
            <xs:element name="servers" type="servers" minOccurs="0"/>
            <xs:element name="interfaces" type="interfaces" minOccurs="0"/>
            <xs:element name="jvms" type="jvms" minOccurs="0"/>
            <xs:element name="extensions" type="extensions" minOccurs="0"/>
        </xs:all>
    </xs:complexType>
    <xs:complexType name="network-configuration">
        <xs:choice minOccurs="0" maxOccurs="unbounded">
            <xs:element name="server-socket" type="net-conf-server"/>
        </xs:choice>
        <xs:attribute name="group-size" type="xs:positiveInteger" default="0" use="optional"/>
        <xs:attribute name="maximum-offset" type="xs:positiveInteger" default="100" use="optional"/>
    </xs:complexType>
    <xs:complexType name="location">
        <xs:attribute name="location" type="xs:anyURI" use="required"/>
        <!-- the interface to bind to for outbound connections to the given location -->
        <xs:attribute name="bind-interface" type="xs:string" use="optional"/>
    </xs:complexType>
    <xs:complexType name="servers">
        <xs:choice minOccurs="0" maxOccurs="unbounded">
            <xs:element name="server" type="server"/>
        </xs:choice>
    </xs:complexType>
    <xs:complexType name="server">
        <xs:all>
            <!-- this element is present if the server is started at boot; otherwise, the server is configured but idle -->
            <xs:element name="running" minOccurs="0"/>
            <xs:element name="port-group" type="ref" minOccurs="0"/>
            <xs:element name="interfaces" type="interfaces" minOccurs="0"/>
            <xs:element name="jvm" type="jvm" minOccurs="0"/>
        </xs:all>
        <xs:attribute name="name" type="xs:string" use="required"/>
        <xs:attribute name="group" type="xs:string" use="required"/>
    </xs:complexType>
    <xs:complexType name="bounded-memory-size">
        <xs:attribute name="size" type="xs:string"/>
        <xs:attribute name="max-size" type="xs:string"/>
    </xs:complexType>
    <xs:complexType name="memory-size">
        <xs:attribute name="size" type="xs:string" use="required"/>
    </xs:complexType>
 
    <xs:complexType name="interfaces">
        <xs:choice minOccurs="0" maxOccurs="unbounded">
            <xs:element name="interface" type="interface"/>
        </xs:choice>
    </xs:complexType>
    <xs:complexType name="interface">
        <xs:attribute name="name" type="xs:string" use="required"/>
        <xs:attribute name="address" type="xs:string" use="required"/>
    </xs:complexType>
    <xs:complexType name="arguments">
        <xs:choice minOccurs="0" maxOccurs="unbounded">
            <xs:element name="argument" type="argument"/>
        </xs:choice>
    </xs:complexType>
    <xs:complexType name="jvm-spec">
        <xs:choice minOccurs="0" maxOccurs="unbounded">
            <!-- Boot class paths? -->
            <!-- Default VM arguments? -->
        </xs:choice>
        <!-- Default depends on platform. Used to pick the right options -->
        <xs:attribute name="type" type="xs:string" use="optional"/>
        <!-- The value of java.home -->
        <xs:attribute name="path" type="xs:string" use="required"/>
        <!-- The name of this JVM -->
        <xs:attribute name="name" type="xs:string" use="required"/>
    </xs:complexType>
    <xs:complexType name="jvm">
        <xs:all>
            <!-- Xms, Xmx -->
            <xs:element name="heap" type="bounded-memory-size" minOccurs="0"/>
            <!-- XX:PermSize, XX:MaxPermSize -->
            <xs:element name="permgen" type="bounded-memory-size" minOccurs="0"/>
            <!-- Xss -->
            <xs:element name="stack" type="memory-size" minOccurs="0"/>
            <!-- Large pages? -->
            <!-- GC tuning settings? -->
            <!-- Arch-specific: CPU affinity? -->
            <!-- Arch-specific: "nice" settings? -->
            <!-- Generic VM arguments -->
            <xs:element name="arguments" type="arguments" minOccurs="0"/>
            <xs:element name="system-properties" type="properties" minOccurs="0"/>
            <xs:element name="environment-properties" type="properties" minOccurs="0"/>
            <!-- Boot class paths? -->
        </xs:all>
        <!-- Default JVM uses java.home and auto-detects type -->
        <xs:attribute name="jvm" type="xs:string" use="optional"/>
    </xs:complexType>
    <xs:complexType name="properties">
        <xs:choice minOccurs="0" maxOccurs="unbounded">
            <xs:element name="property" type="property"/>
        </xs:choice>
    </xs:complexType>
    <xs:complexType name="property">
        <xs:attribute name="name" type="xs:string" use="required"/>
        <xs:attribute name="value" type="xs:string" use="required"/>
    </xs:complexType>
 
    <xs:complexType name="argument">
        <xs:attribute name="value" type="xs:string" use="required"/>
    </xs:complexType>
    <xs:complexType name="ref">
        <xs:attribute name="name" type="xs:string" use="required"/>
    </xs:complexType>
    <xs:complexType name="net-conf-server">
        <xs:all>
            <xs:element name="interface" type="ref" minOccurs="0"/>
            <xs:element name="extensions" type="extensions" minOccurs="0"/>
        </xs:all>
        <xs:attribute name="service-name" type="xs:string" use="required"/>
        <xs:attribute name="binding-name" type="xs:string" default="default" use="optional"/>
        <xs:attribute name="port" type="port-number" use="required"/>
        <xs:attribute name="fixed" type="xs:boolean" default="false" use="optional"/>
    </xs:complexType>
    <xs:simpleType name="port-number">
        <xs:restriction base="xs:int">
            <xs:minInclusive value="0"/>
            <xs:maxExclusive value="65536"/>
        </xs:restriction>
    </xs:simpleType>
    <xs:complexType name="jvms">
        <xs:choice minOccurs="0" maxOccurs="unbounded">
            <xs:element name="jvm" type="jvm-spec"/>
        </xs:choice>
    </xs:complexType>
    <xs:complexType name="extensions">
        <xs:choice minOccurs="0" maxOccurs="unbounded">
            <xs:any/>
        </xs:choice>
    </xs:complexType>
</xs:schema>

This is pretty rough though.  In particular I don't like having a dedicated "extensions" element for plugging in stuff from other schemas; the alternatives to using "xs:all" are to use "xs:choice" and make everything unbounded (with additional runtime validation), or using "xs:sequence" and losing the ability to make order be significant between different element types.  Thanks schema!

Also, I think that since this file and domain.xml have many types in common we should be merging this schema with the domain schema and using multiple root elements (one for host, one for domain).

[dmlloyd]

Alexey, any feedback as to how this might integrate with what you have for domain.xml?

[aloubyansky]

The host.xml idea does make sense to me. I've actually been talking to Emanuel earlier about a separate config per host. Things like JVM-, network-, perhaps threads-related options, etc should be able to be specified per host. The hardware of servers maybe different and there should be a way to adjust settings per host/server.

Let's think about what:

- belongs only to domain.xml;

- belongs only to host.xml;

- can be specified in domain.xml and overriden in host.

As to the host schema itself, I'll write it according to the style/format we'll use for other domain-related schemas. Also re-using common types and elements.

[dmlloyd]

We need to intersect better on this, since it sounds like there are more than one group working on the same problem in parallel...

[rmaucher]

Would it be possible to not call this "host", due to possible confusion with virtual hosts from the web server ?

[dmlloyd]

Remy Maucherat wrote:

Would it be possible to not call this "host", due to possible confusion with virtual hosts from the web server ?

The problem is that we're running out of terminology here.  We have "[application] server" which is a running instance of the AS and a "host" which is the system upon which the server runs.  If the term, when used with the web server, would continue to be "virtual host", maybe it won't be too big of a problem?

[aloubyansky]

I've added jboss-domain-host.xsd which is an edited version of the host schema above.

The differences are:

- changed the namespace to the domain one;
- changed the type names to follow the naming style in other domain schemas;
- serverType collisions and changes:
  - attribute 'group' from serverType in host.xsd lost is removed (there is server-group-ref element in the domain schema with maxOccurs=unbounded);
  - attribute 'name' from serverType in host.xsd added to the existing serverType in domain schema;
  - 'running', 'interfaces' and 'port-group' elements added to the existing serverType in domain schema;
- java-configType from the domain schema received permgen and stack elements;
- jvms element in host-config became java-config of type java-configType.

So now only the host.xml is supposed to have configuration for servers (not to confuse with server-groups), right? I am going to remove servers from under the domain element.

I'll write some example XMLs to have a better idea of how it looks like.

[dmlloyd]

Alexey Loubyansky wrote:

I've added jboss-domain-host.xsd which is an edited version of the host schema above.

The differences are:

- changed the namespace to the domain one;
- changed the type names to follow the naming style in other domain schemas;
- serverType collisions and changes:
  - attribute 'group' from serverType in host.xsd lost is removed (there is server-group-ref element in the domain schema with maxOccurs=unbounded);
  - attribute 'name' from serverType in host.xsd added to the existing serverType in domain schema;
  - 'running', 'interfaces' and 'port-group' elements added to the existing serverType in domain schema;
- java-configType from the domain schema received permgen and stack elements;
- jvms element in host-config became java-config of type java-configType.

So now only the host.xml is supposed to have configuration for servers (not to confuse with server-groups), right? I am going to remove servers from under the domain element.

I'll write some example XMLs to have a better idea of how it looks like.

What's the rationale for splitting it up into multiple XSD files?  I'm finding it a bit hard to read, honestly.

Will review the rest of the changes and comment shortly...

[aloubyansky]

Yes, I'm gonna merge them.

[aloubyansky]

Actually, I'd probably keep domain and host separately for now. So, I'd have common, domain and host schemas with domain and host including the common. Would that be fine?

[dmlloyd]

Alexey Loubyansky wrote:

Actually, I'd probably keep domain and host separately for now. So, I'd have common, domain and host schemas with domain and host including the common. Would that be fine?

That's probably fine for now.