This article is a brief description of some work I've been doing on an "offline CLI". It's meant to be a brief intro, seed for future sections of the Admin Guide and background material for design discussions around this work.
This document describes work currently in a topic branch at bstansberry/wildfly-core at cli-embed-server · GitHub.
An initial discussion of it on the wildfly-dev mailing list is at [wildfly-dev] Embedding a WF instance in the CLI.
What is this?
Ability to embed a WildFly process inside the CLI process, with the CLI interacting with the embedded server in a manner consistent with how it interacts with a remote WildFly process.
This work relies on the embedded WildFly code that was primarily developed by Thomas Diesler and Kabir Khan. Many thanks to both of them!
Why this?
Direct local administration of a WildFly Core-based installation via the CLI without requiring a socket-based connection. The general use case is initial setup type activities where the user doesn't want to have to launch a WF server or HC process and potentially have it be visible on the network. https://issues.jboss.org/browse/WFLY-3288 is one use case; another is a desire some folks have expressed in being able to do configuration without first having to edit any xml to avoid port conflicts on 9990 or 9999.
As the CLI is itself embeddable, this also opens up the possibility of embedding the CLI inside some other process (say a test class, a provisioning tool or a jar with a main) and then launching the embedded server and using the embedded CLI as a convenient management tool.
How to Play With It
Check out and build my topic branch at bstansberry/wildfly-core at cli-embed-server · GitHub. That's sufficient to let you play with a standalone WildFly Core server.
If you want the full set of extensions available with full WildFly, first build my branch of WildFly Core and then check out the master branch of WildFly. To integrate your local build of my branch, build WildFly full as follows:
mvn install -rf web-feature-pack -DskipTests -Dorg.wildfly.core=1.0.0.Beta1-SNAPSHOT
The -Dorg.wildfly.core=1.0.0.Alpha18-SNAPSHOT overrides the version of WildFly Core specified in the WildFly pom with another version. As of this writing you would specify 1.0.0.Beta1-SNAPSHOT, but check the pom in my branch as it will be updated to the next version whenever WildFly Core is released.
Simple Example
Here I start the CLI in a modular environment, launch an embedded server, do a couple simple CLI things, and stop the embedded server:
$ bin/jboss-cli.sh
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] embed-server --std-out=echo
12:10:15,300 INFO [org.jboss.modules] (main) JBoss Modules version 1.4.1.Final
12:10:15,983 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.4.Final
12:10:16,049 INFO [org.jboss.as] (MSC service thread 1-6) WFLYSRV0049: WildFly Full 9.0.0.Alpha2-SNAPSHOT (WildFly Core 1.0.0.Alpha18-SNAPSHOT) starting
12:10:16,891 INFO [org.jboss.as.controller.management-deprecated] (Controller Boot Thread) WFLYCTL0028: Attribute enabled is deprecated, and it might be removed in future version!
12:10:17,055 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 9.0.0.Alpha2-SNAPSHOT (WildFly Core 1.0.0.Alpha18-SNAPSHOT) started in 7113ms - Started 35 of 48 services (19 services are lazy, passive or on-demand)
[standalone@embedded /] ls -l
ATTRIBUTE VALUE TYPE
launch-type EMBEDDED STRING
management-major-version 3 INT
management-micro-version 0 INT
management-minor-version 0 INT
name taozi STRING
namespaces [] OBJECT
process-type Server STRING
product-name WildFly Full STRING
product-version 9.0.0.Alpha2-SNAPSHOT STRING
profile-name undefined STRING
release-codename Kenny STRING
release-version 1.0.0.Alpha18-SNAPSHOT STRING
running-mode ADMIN_ONLY STRING
schema-locations [] OBJECT
server-state running STRING
suspend-state RUNNING STRING
CHILD MIN-OCCURS MAX-OCCURS
core-service n/a n/a
deployment n/a n/a
deployment-overlay n/a n/a
extension n/a n/a
interface n/a n/a
path n/a n/a
socket-binding-group n/a n/a
subsystem n/a n/a
system-property n/a n/a
[standalone@embedded /] cd socket-binding-group=standard-sockets/socket-binding=management-http
[standalone@embedded socket-binding=management-http] :write-attribute(name=port,value=19990)
{"outcome" => "success"}
[standalone@embedded socket-binding=management-http] reload --admin-only=false
12:12:34,615 INFO [org.jboss.as] (MSC service thread 1-16) WFLYSRV0050: WildFly Full 9.0.0.Alpha2-SNAPSHOT (WildFly Core 1.0.0.Alpha18-SNAPSHOT) stopped in 16ms
12:12:34,621 INFO [org.jboss.as] (MSC service thread 1-16) WFLYSRV0049: WildFly Full 9.0.0.Alpha2-SNAPSHOT (WildFly Core 1.0.0.Alpha18-SNAPSHOT) starting
. . . .
12:12:36,176 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 9.0.0.Alpha2-SNAPSHOT (WildFly Core 1.0.0.Alpha18-SNAPSHOT) started in 1505ms - Started 202 of 379 services (210 services are lazy, passive or on-demand)
[standalone@embedded socket-binding=management-http] stop-embedded-server
12:12:43,352 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-5) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
12:12:43,352 INFO [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0019: Host default-host stopping
12:12:43,364 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-13) WFLYJCA0019: Stopped Driver service with driver-name = h2
12:12:43,368 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0008: Undertow HTTP listener default suspending
12:12:43,380 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0007: Undertow HTTP listener default stopped, was bound to /127.0.0.1:8080
12:12:43,384 INFO [org.wildfly.extension.undertow] (MSC service thread 1-13) WFLYUT0004: Undertow 1.2.0.Beta8 stopping
12:12:43,393 INFO [org.jboss.as] (MSC service thread 1-7) WFLYSRV0050: WildFly Full 9.0.0.Alpha2-SNAPSHOT (WildFly Core 1.0.0.Alpha18-SNAPSHOT) stopped in 13ms
[disconnected socket-binding=management-http] quit
Here I do a similar thing, but using the non-modular jboss-cli-client.jar included in the bin/client directory of the WildFly full distribution. I also don't include the "--std-out=echo" param, so I don't see the server logging in the console:
$ java -jar bin/client/jboss-cli-client.jar
You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands.
[disconnected /] embed-server --jboss-home=/home/brian/dev/wildfly/dist/target/wildfly-9.0.0.Alpha2-SNAPSHOT
[standalone@embedded /] cd socket-binding-group=standard-sockets/socket-binding=management-http
[standalone@embedded socket-binding=management-http] :read-attribute(name=port)
{
"outcome" => "success",
"result" => 19990
}
[standalone@embedded socket-binding=management-http] quit
I'll explain the "--jboss-home" command below. See "Modular vs Non-Modular Classloading and JBOSS_HOME".
Here I build an entire server config via a CLI script (script is attached to this document). The resulting config is equivalent to the standard standalone-full-ha.xml config that ships with WildFly full:
$ bin/jboss-cli.sh --file=/Users/bstansberry/tmp/embedded-server.txt The batch executed successfully The batch executed successfully process-state: reload-required taozi:wildfly-9.0.0.Alpha2-SNAPSHOT bstansberry$
It works!
$ bin/standalone.sh -c standalone-empty.xml ========================================================================= JBoss Bootstrap Environment JBOSS_HOME: /Users/bstansberry/dev/wildfly/wildfly/dist/target/wildfly-9.0.0.Alpha2-SNAPSHOT JAVA: /Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home/bin/java JAVA_OPTS: -server -XX:+UseCompressedOops -server -XX:+UseCompressedOops -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true ========================================================================= 12:24:45,565 INFO [org.jboss.modules] (main) JBoss Modules version 1.4.1.Final 12:24:45,775 INFO [org.jboss.msc] (main) JBoss MSC version 1.2.4.Final 12:24:45,843 INFO [org.jboss.as] (MSC service thread 1-6) WFLYSRV0049: WildFly Full 9.0.0.Alpha2-SNAPSHOT (WildFly Core 1.0.0.Alpha18-SNAPSHOT) starting . . . . 12:24:48,649 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management 12:24:48,649 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990 12:24:48,649 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 9.0.0.Alpha2-SNAPSHOT (WildFly Core 1.0.0.Alpha18-SNAPSHOT) started in 3365ms - Started 246 of 478 services (281 services are lazy, passive or on-demand)
Finer Points
Modular vs Non-Modular Classloading and JBOSS_HOME
As you can see above, the CLI can either be running in a modular classloading environment (bin/jboss-cli.sh case) or in a flat classpath (java --jar bin/client/jboss-cli-client.jar case.) Either way, the embedded server runs in a modular classloading environment. There are some behavior differences between the two cases though:
- If the CLI is running in a modular classloading environment:
- the embedded server will use the same boot module loader as the CLI. The implication here is the CLI and server are running from the same WildFly installation, with the same module path.
- the embedded server will need to know where the root of the WildFly installation is. This must be provided to the CLI via the JBOSS_HOME environment variable. The jboss-cli.sh script sets this. If some other mechanism is used for starting the CLI, the JBOSS_HOME environment variable must be set.
- If the CLI is not running in a modular classloading environment:
- the embedding logic will set up an appropriate modular classloading environment for the server. The module path for the modular classloader will have a single element:
- <root_of_wildfly_installation>/modules
- the embedded server will need to know where the root of the WildFly installation is. This must be provided to the CLI via one of the following mechanisms:
- the JBOSS_HOME environment variable
- the --jboss-home parameter to the "embed-server" command. If this is set, it takes precedence over any JBOSS_HOME environment variable
- the embedding logic will set up an appropriate modular classloading environment for the server. The module path for the modular classloader will have a single element:
The --jboss-home parameter to the "embed-server" command is not supported in a modular CLI environment, as it would imply that the root of the embedded server could be something other than the root of the install from which the CLI is running.
This is admittedly not very clean.
Specifying the Server Configuration
The embed-server command supports "-c" and "--server-config" parameters that can be used to specify the name of the configuration file to use. If not specified, the default is standalone.xml
The read-only-server-config notion supported by standalone.sh is not supported. It could be readily enough; it just wasn't a priority as the focus of this work is using this tool to change configuration.
Starting with an Empty Configuration
Embedding the server opens up the possibility of starting from a completely empty, even non-existent, configuration. Such a configuration makes no sense without embedding, as no config means no remote management interface, and thus no means to change the config to something useful. But an offline CLI doesn't have that problem.
So, the embed-server command allows you to specify that an initially empty configuration should be used:
[disconnected /] embed-server --server-config=my-config.xml --empty-config
That command will fail if file $JBOSS_HOME/standalone/configuration/my-config.xml already exists. This is to avoid accidental deletion of a config file. However, you can specify that you want any existing configuration removed:
[disconnected /] embed-server --server-config=my-config.xml --empty-config --remove-existing
Admin-only Mode
By default the embedded server will be started in admin-only mode. This is because the main expected use cases are for initial configuration.
This can be changed with a parameter to the embed-server command:
[disconnected /] embed-server --admin-only=false
Same as with a non-embedded server, a server can be moved in and out of admin-only using the CLI reload command:
[standalone@embedded /] reload --admin-only=false
Admin-only Mode and the Server's Management Interfaces
One of the goals of this work is to support use cases where the server sockets opened by the server need to be configured. The management interfaces themselves open sockets (e.g. port 9990, 9999), even when the server is in --admin-only mode. But, what if there is a part conflict on those ports, with the purpose of using offline CLI being to change settings to avoid the conflict?
To account for this, I have changed the behavior of the add management operation handler for the management interface resource. Now, if the processes' ProcessType is ProcessType.EMBEDDED_SERVER, and the RunningMode is RunningMode.ADMIN_ONLY, the MSC services for the remote management interfaces *will not* be started.
Boot Timeout
By default, the embed-server command will block indefinitely waiting for the embedded server to reach server-state "running"; i.e. to complete boot. The amount of time to wait can be controlled by using the --timeout parameter
[disconnected /] embed-server --timeout=30
The value is in seconds.
A value of less than 1 means the embed-server command will not block waiting for boot to complete. Rather, it will return as soon as boot proceeds to the point where a ModelController service is available, allowing the CLI to obtain a ModelControllerClient to use to interact with the server.
A server in admin-only mode would typically boot very quickly, so configuring this timeout would be more useful when --admin-only=false is used.
Controlling stdout
The CLI uses stdout heavily. The embedded server may also want to write to stdout, particularly for console logging. These two uses of stdout have the potential to interfere with each other, particularly in an interactive session where the CLI may output a command prompt and then the server logs something, resulting in the prompt being in the middle of server log messages, possibly in the middle of a line. The interactive CLI can still work if this happens, but it can be disorienting.
The embed-server command includes a parameter to allow the user to control what happens to output the embedded server writes to stdout:
- --std-out=echo -- the output from the server is allowed to go to the CLI's stdout, allowing the user to see logging, but at the risk of mixing the CLI prompt with server logging
- --std-out=discard -- the output the server attempts to send to stdout is discarded. Users should look at the server.log file to see server logging.
The default behavior is --std-out=discard
"stop-embedded-server" vs "shutdown"
When the embedded server is running, I disable the CLI's "shutdown" command and replace it with "stop-embedded-server". This is for two reasons:
- The shutdown command includes a --restart=true option and I'm not certain how to support that, and in any case haven't done so yet
- The shutdown command may in the future include other options that don't fit the embedded case (or vice versa), so it seemed wiser not to constrain either case by reusing a command
Logging
Controlling logging is an important issue in an embedded environment. The embedding process has its own logging requirements, which the embedded server also wants to log. In the CLI context, console logging from the embedded server is particularly important to control, as such logging can interfere with the CLI prompt.
My branch includes commits to establish a separate LogContext for the embedded server. This likely needs hardening though.
Future Work
- Harden the existing work, sort through the various issues that will certainly come up.
- Embed the CLI itself in some other process, and then use this. An integration test in WildFly Core's testsuite comes to mind. Build a deployment using shrinkwrap and then install it.
- An important missing piece is being able to pass a stream through somehow. Image java -jar helloworld.war where the WAR includes a main class and the jboss-cli-client.jar. The main class starts the embedded CLI and embedded server and then passes itself in as a deployment. (In a lot of ways, the CLI is an impediment to this use case vs the main class directly embedding the server and working with ModelControllerClient. But if configuration steps are wanted the CLI is a much nicer API for doing that work vs direct use of ModelControllerClient.)
- An important missing piece is being able to pass a stream through somehow. Image java -jar helloworld.war where the WAR includes a main class and the jboss-cli-client.jar. The main class starts the embedded CLI and embedded server and then passes itself in as a deployment. (In a lot of ways, the CLI is an impediment to this use case vs the main class directly embedding the server and working with ModelControllerClient. But if configuration steps are wanted the CLI is a much nicer API for doing that work vs direct use of ModelControllerClient.)
- Add the ability to embed a combination HostController+ProcessController process so equivalent functionality is available for working with managed domains.
Comments