5 Replies Latest reply on Nov 21, 2017 11:50 AM by jamezp

    App Deployment with Native Management API

    jfisherdev

      I am currently looking to deploy applications using the native management API and encountering some difficulties.

       

      This does seem to be a topic I haven't found a lot of documentation or examples for, relative to deployment via the CLI, web console, or other options. However, I do recognize that as with other management tasks these options are often simpler and preferable, which would explain why there is more information about them.

       

      I will do my best to provide some background to explain why I am using the native management API and what I'm looking to do.

       

      I am running a WildFly 9.0.2.final server in standalone mode. I start the server in two steps. First, the server is started "offline" in admin-only mode. Then, I run a standalone Java program [I refer to it as the "launcher" program, though it doesn't use the WildFly Launcher API at this time] that uses the native management API to configure management resources and then issues a reload command. I use the native management API here to satisfy some non-trivial dynamic deployment requirements and being able to leverage Java makes this much easier. It has worked quite well and has been easy to evolve as new configuration requirements arise. Anyone who is interested in how exactly this came about can refer to this discussion Re: Adding Logging Subsystem Resources At Boottime [thanks again to jamezp and ctomc for getting me on the right track].

       

      Currently applications are deployed using the deployment scanner approach; however, I want to move away from that and to use the management API instead. Besides the main reasons of the deployment scanner approach not being recommended for production and only working for standalone mode, I want to use the management API for these reasons as well:

      • The default parallel deployment behavior can be more resource intensive than desired depending on how many apps are being deployed. I have also found it can be hard to see what app(s) are deploying or taking too long to deploy in some cases where the jboss.as.management.timeout is exceeded.
      • There are cases where an application has a deployment ordering dependency on another application that is optionally deployed [e.g. myapp.ear should be deployed after app-extras.war, but only when app-extras.war, which may not always be deployed, is deployed]. I don't think jboss-all.xml can be used for this scenario.
      • I would like to be able to collect metrics for app deployment times.

       

      My thinking is to expand my launcher program to handle app deployment. I will show an outline of what I have done and then explain what issues I am seeing.

       

      First, I modeled the app deployment resource and schedule concepts like this:

       

      class AppDeployment {
      
           //App name, deployment resource name
           String getAppName();
      
           //URL to deployment resource, such as an EAR/WAR file
           URL getDeploymentURL();
      }
      

       

      class AppDeploymentSchedule {
      
           Set<Slot> getSlots();
      
           static class Slot {
                ...
                Set<AppDeployment> getApps();
                ...
           }
      }
      

       

      At this time I am looking to deploy apps located on the same file system [in the deployments directory in fact--the scanner is deactivated] and in a sequential fashion [deployment schedule slots with a single app]. However, as you can see from the way I structured them, the models are intended to allow for a variety of deployment resource locations and deploying apps sequentially or in parallel.

       

      As far as how the launcher app looks, here is an outline of what gets called from main:

       

      class ServerLauncher {
      
           void launch() {
                //Wait for server to be available for management operations
                waitForServer();
                configureServer();
                //New addition, need to do this prior to reload so apps are deployed in the schedule
                undeployApps();
                //This is where the program previously ended
                reloadServer();
                //Need to wait for server to be available for management operations after reloading
                waitForServer();
                deployApps();
           }
      
           private void undeployApps() {
                final ServerDeploymentManager deploymentManager = ...;
                //Use ModelControllerClient to read deployment resource names
                final Set<String> existingDeployments = ...;
                DeploymentPlanBuilder undeploymentPlanBuilder = deploymentManager.newDeploymentPlan();
                for(String existingDeployment : existingDeployments) {
                     undeploymentPlanBuilder = undeploymentPlanBuilder.undeploy(existingDeployment).andRemoveUndeployed();
                }
                deploymentManager.execute(undeploymentPlanBuilder.build());
           }
      
           private void deployApps() {
                final ServerDeploymentManager deploymentManager = ...;
                final AppDeploymentSchedule schedule = ...;
                for(AppDeploymentSchedule.Slot slot : schedule.getSlots()) {
                     DeploymentPlanBuilder slotPlanBuilder = deploymentManager.newDeploymentPlan();
                     for(AppDeployment app : slot.getApps()) {
                          slotPlanBuilder = slotPlanBuilder.add(app.getDeploymentURL).andDeploy();
                     }
                     deploymentManager.execute(slotPlanBuilder.build());
                }
           }
      }
      

       

      While there is more to the actual implementation, I think that covers the important details.

       

      The biggest issue I'm running into is with OutOfMemoryErrors that usually look something like this:

       

      Exception in thread "management-client-thread 2-2" java.lang.OutOfMemoryError: Java heap space
      at java.util.Arrays.copyOf(Arrays.java:3236)
      at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
      at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
      at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
      at org.jboss.as.protocol.StreamUtils.copyStream(StreamUtils.java:52)
      at org.jboss.as.controller.client.impl.InputStreamEntry$InMemoryEntry.initialize(InputStreamEntry.java:76)
      at org.jboss.as.controller.client.impl.AbstractModelControllerClient$ReadAttachmentInputStreamRequestHandler$1.execute(AbstractModelControllerClient.java:220)
      at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:298)
      at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:518)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:748)
      at org.jboss.threads.JBossThread.run(JBossThread.java:320)
      

       

      The memory footprint of the launcher application is pretty small to start with [usually around 50m with a 256m max heap size], but once you introduce app deployment it seems that memory requirements increase depending on the size of the deployment archives being deployed.

       

      The issue seems to be rooted in the fact that all of the deployment content is read into memory at once. I even tried the DeploymentPlanBuilder.add(String, String, InputStream) passing in a BufferedInputStream that wraps the input stream from the URL and still had the issue. From what I can see, I think it gets down to the ModelControllerClient/Operation level and the default behavior for dealing with Operation InputStream attachments, as in AbstractModelControllerClient there is a TODO that states: "don't copy everything to memory... perhaps use InputStreamEntry.CachingStreamEntry"

       

      I haven't worked with operation attachments, but I'm guessing it is possible to use input streams so operations aren't as resource intensive. I am willing to accept that using the native management API for app deployment may require an increase in memory to a degree; however, this seems beyond whatever that would be. Ultimately it looks like the DeploymentPlan gets translated into native management API operations, so I suppose it would be possible to write the deployment operations I need with appropriate content stream handling, but obviously it would be preferable to use the provided deployment API.

       

      jamezp: As I was writing this, I came across a DeploymentManager class you wrote in the wildfly-plugin-core project. I didn't look at it extensively, but it looks like it has some similarities with the deployment API that comes with the wildfly-controller-client module, with a key difference being that it can handle standalone and domain servers with a single interface. I'm curious if this is intended for public consumption outside of the Maven plugin or if I would get anything out of using it?

       

      If anyone could direct me on what the correct way to handle what I am looking to do, I would appreciate that very much.

        • 1. Re: App Deployment with Native Management API
          jamezp

          Hi Josh,

          The wildfly-plugin-core is meant to be a public API so you are welcome to use that. You can definitely write you're own deployment operations too, but you can also hide those details with the wildfly-plugin-core API. The readme has some basic examples, but I'd be happy to share more examples if you need.

           

          --

          James R. Perkins

          1 of 1 people found this helpful
          • 2. Re: App Deployment with Native Management API
            jfisherdev

            James,

             

            Thank you for responding.

             

            I'll take a look at those examples and see if that works.

             

            Two questions I do have offhand:

            1. What version of that project should be used with WildFly 9.0.2.Final? When I originally started using it in my Launcher project for the waitForStandalone() utility method, I pulled the 1.1.0.Beta1 version as that was the latest version available at the time, but now it looks like there is a 1.1.0.Final version available along with 1.2.x versions.
            2. Do you have any recommendations for keeping memory usage within a particular limit, even if it comes at the expense of not being quite as fast, for the deployment content upload component, which is the main issue I'm facing.
            • 3. Re: App Deployment with Native Management API
              jamezp

              If you can use Java 8 I'd use the 1.2.1.Final, though 1.1.0.Final would be fine too. The API hasn't really changed.

               

              I think using a File, Path or URL would be the least memory intensive. A very simple example would look something like:

              final StandaloneCommandBuilder builder = StandaloneCommandBuilder.of(Paths.get("/home/wildfly/servers/wildfly-11.0.0.Final"));
              final Process process = Launcher.of(builder).inherit().launch();
              try (ModelControllerClient client = createClient()) {
                  ServerHelper.waitForStandalone(process, client, 30);
                  final DeploymentManager deploymentManager = DeploymentManager.Factory.create(client);
                  final Deployment deployment = Deployment.of(Paths.get("/home/wildfly/deployments/test.war"));
                  try {
                      deploymentManager.deploy(deployment).assertSuccess();
                  } finally {
                      deploymentManager.undeploy(UndeployDescription.of(deployment)).assertSuccess();
                  }
                  ServerHelper.shutdownStandalone(client);
                  process.waitFor();
              } finally {
                  process.destroyForcibly();
              }
              

               

              --

              James R. Perkins

              1 of 1 people found this helpful
              • 4. Re: App Deployment with Native Management API
                jfisherdev

                I tried this out and immediately I could see that memory usage dropped dramatically [it now only uses about 25m, which is normal for the application] and performance improved noticeably. I'm impressed with what this API has to offer as well as its ease of use.

                 

                I am using 1.1.0.Final at the moment, but will likely try 1.2.1.Final. I can use Java 8. The only thing I wasn't sure about was if there were any incompatibilities with WildFly 9.0.2.Final and the newer wildfly-controller-client version it looks like 1.2.1.Final is using.

                 

                The thing that I think confused me and may be worth noting is that when using the deployment API that comes with wildfly-controller-client to build a deployment plan, the various DeploymentPlanBuilder.add() methods don't translate to different methods of handling deployment content like they do with the plugin deployment API. For example, using DeploymentPlanBuilder.add(URL) doesn't translate to using the URL approach for specifying deployment content [e.g. with the cli, deploy --url=...] like using Deployment.of(URL) does. Instead it always translates to using the input stream approach, which I get the impression is generally more resource intensive, especially depending on how much content is being sent with the operation [I still have more to learn about operation streams and attachments].

                 

                One other thing I found with DeploymentPlanBuilder that I would like to mention is that none of the add() methods that take a string parameter, such as add(String, File), work unless you specify the deployment archive name. If you use something else, deployment content is not uploaded properly. I found that none of the subdeployments and other content within an archive were present. The only one of these methods that actually allows you to use a different name is add(String, String, InputStream). Granted, this isn't something I need to do and I have found the plugin deployment API to be a better option for what I'm doing, but I thought it may be worth mentioning in case these are things that should be reviewed.

                 

                In any case, I want to thank you again for your prompt and helpful responses in helping me solve this problem.

                • 5. Re: App Deployment with Native Management API
                  jamezp

                  I am using 1.1.0.Final at the moment, but will likely try 1.2.1.Final. I can use Java 8. The only thing I wasn't sure about was if there were any incompatibilities with WildFly 9.0.2.Final and the newer wildfly-controller-client version it looks like 1.2.1.Final is using.

                  It should work down to WildFly 8. If you have any issues though feel free to file an issue at https://issues.jboss.org/browse/WFMP. The goal is to keep backwards compatibility.

                   

                  The DeploymentPlanBuilder was create quite a while ago. Possibly before url was even supported as part of the operation. I'm guessing it copies the URL input stream into memory. The main reason I created the API was using the operations seemed to perform better, as you've noticed, than using the DeploymentPlanBuilder. I didn't really look into why it performs better, but maybe I should at some point .

                   

                  --

                  James R. Perkins