App Deployment with Native Management API
jfisherdev Nov 20, 2017 2:12 AMI am currently looking to deploy applications using the native management API and encountering some difficulties.
This does seem to be a topic I haven't found a lot of documentation or examples for, relative to deployment via the CLI, web console, or other options. However, I do recognize that as with other management tasks these options are often simpler and preferable, which would explain why there is more information about them.
I will do my best to provide some background to explain why I am using the native management API and what I'm looking to do.
I am running a WildFly 9.0.2.final server in standalone mode. I start the server in two steps. First, the server is started "offline" in admin-only mode. Then, I run a standalone Java program [I refer to it as the "launcher" program, though it doesn't use the WildFly Launcher API at this time] that uses the native management API to configure management resources and then issues a reload command. I use the native management API here to satisfy some non-trivial dynamic deployment requirements and being able to leverage Java makes this much easier. It has worked quite well and has been easy to evolve as new configuration requirements arise. Anyone who is interested in how exactly this came about can refer to this discussion Re: Adding Logging Subsystem Resources At Boottime [thanks again to jamezp and ctomc for getting me on the right track].
Currently applications are deployed using the deployment scanner approach; however, I want to move away from that and to use the management API instead. Besides the main reasons of the deployment scanner approach not being recommended for production and only working for standalone mode, I want to use the management API for these reasons as well:
- The default parallel deployment behavior can be more resource intensive than desired depending on how many apps are being deployed. I have also found it can be hard to see what app(s) are deploying or taking too long to deploy in some cases where the jboss.as.management.timeout is exceeded.
- There are cases where an application has a deployment ordering dependency on another application that is optionally deployed [e.g. myapp.ear should be deployed after app-extras.war, but only when app-extras.war, which may not always be deployed, is deployed]. I don't think jboss-all.xml can be used for this scenario.
- I would like to be able to collect metrics for app deployment times.
My thinking is to expand my launcher program to handle app deployment. I will show an outline of what I have done and then explain what issues I am seeing.
First, I modeled the app deployment resource and schedule concepts like this:
class AppDeployment { //App name, deployment resource name String getAppName(); //URL to deployment resource, such as an EAR/WAR file URL getDeploymentURL(); }
class AppDeploymentSchedule { Set<Slot> getSlots(); static class Slot { ... Set<AppDeployment> getApps(); ... } }
At this time I am looking to deploy apps located on the same file system [in the deployments directory in fact--the scanner is deactivated] and in a sequential fashion [deployment schedule slots with a single app]. However, as you can see from the way I structured them, the models are intended to allow for a variety of deployment resource locations and deploying apps sequentially or in parallel.
As far as how the launcher app looks, here is an outline of what gets called from main:
class ServerLauncher { void launch() { //Wait for server to be available for management operations waitForServer(); configureServer(); //New addition, need to do this prior to reload so apps are deployed in the schedule undeployApps(); //This is where the program previously ended reloadServer(); //Need to wait for server to be available for management operations after reloading waitForServer(); deployApps(); } private void undeployApps() { final ServerDeploymentManager deploymentManager = ...; //Use ModelControllerClient to read deployment resource names final Set<String> existingDeployments = ...; DeploymentPlanBuilder undeploymentPlanBuilder = deploymentManager.newDeploymentPlan(); for(String existingDeployment : existingDeployments) { undeploymentPlanBuilder = undeploymentPlanBuilder.undeploy(existingDeployment).andRemoveUndeployed(); } deploymentManager.execute(undeploymentPlanBuilder.build()); } private void deployApps() { final ServerDeploymentManager deploymentManager = ...; final AppDeploymentSchedule schedule = ...; for(AppDeploymentSchedule.Slot slot : schedule.getSlots()) { DeploymentPlanBuilder slotPlanBuilder = deploymentManager.newDeploymentPlan(); for(AppDeployment app : slot.getApps()) { slotPlanBuilder = slotPlanBuilder.add(app.getDeploymentURL).andDeploy(); } deploymentManager.execute(slotPlanBuilder.build()); } } }
While there is more to the actual implementation, I think that covers the important details.
The biggest issue I'm running into is with OutOfMemoryErrors that usually look something like this:
Exception in thread "management-client-thread 2-2" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3236) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) at org.jboss.as.protocol.StreamUtils.copyStream(StreamUtils.java:52) at org.jboss.as.controller.client.impl.InputStreamEntry$InMemoryEntry.initialize(InputStreamEntry.java:76) at org.jboss.as.controller.client.impl.AbstractModelControllerClient$ReadAttachmentInputStreamRequestHandler$1.execute(AbstractModelControllerClient.java:220) at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:298) at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:518) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) at org.jboss.threads.JBossThread.run(JBossThread.java:320)
The memory footprint of the launcher application is pretty small to start with [usually around 50m with a 256m max heap size], but once you introduce app deployment it seems that memory requirements increase depending on the size of the deployment archives being deployed.
The issue seems to be rooted in the fact that all of the deployment content is read into memory at once. I even tried the DeploymentPlanBuilder.add(String, String, InputStream) passing in a BufferedInputStream that wraps the input stream from the URL and still had the issue. From what I can see, I think it gets down to the ModelControllerClient/Operation level and the default behavior for dealing with Operation InputStream attachments, as in AbstractModelControllerClient there is a TODO that states: "don't copy everything to memory... perhaps use InputStreamEntry.CachingStreamEntry"
I haven't worked with operation attachments, but I'm guessing it is possible to use input streams so operations aren't as resource intensive. I am willing to accept that using the native management API for app deployment may require an increase in memory to a degree; however, this seems beyond whatever that would be. Ultimately it looks like the DeploymentPlan gets translated into native management API operations, so I suppose it would be possible to write the deployment operations I need with appropriate content stream handling, but obviously it would be preferable to use the provided deployment API.
jamezp: As I was writing this, I came across a DeploymentManager class you wrote in the wildfly-plugin-core project. I didn't look at it extensively, but it looks like it has some similarities with the deployment API that comes with the wildfly-controller-client module, with a key difference being that it can handle standalone and domain servers with a single interface. I'm curious if this is intended for public consumption outside of the Maven plugin or if I would get anything out of using it?
If anyone could direct me on what the correct way to handle what I am looking to do, I would appreciate that very much.