1 Reply Latest reply on Sep 19, 2018 2:16 PM by jfisherdev

    Random CancellationException When Executing Reload With ModelControllerClient

    jfisherdev
    jfisherdev Sep 13, 2018 1:55 PM

      I have a standalone Java application that handles the launch process for WildFly servers and uses the native management API. I am currently using this with WildFly 9.0.2.Final and only with standalone servers.

       

      Basically it consists of these four steps:

      1. Connect to server started in admin-only mode [eventually I want this application to handle starting the server]
      2. Perform configuration operations
      3. Reload the server so it's in normal running mode
      4. Perform application deployment operations

       

      I have in some rare cases--less than 5 at this time, all very recent, and that would be out of thousands of runs over a period of around 18 months--seen the launch process fail due to a java.util.concurrent.CancellationException that looks like this:

       

      Caused by: java.io.IOException: java.util.concurrent.CancellationException: Operation was cancelled
at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:149)
at org.jboss.as.controller.client.impl.AbstractModelControllerClient.execute(AbstractModelControllerClient.java:75)
at org.jboss.as.controller.client.helpers.DelegatingModelControllerClient.execute(DelegatingModelControllerClient.java:63)
at Launcher.reload(Launcher.java:###)
..
Caused by: java.util.concurrent.CancellationException: Operation was cancelled
at org.jboss.threads.AsyncFutureTask.operationCancelled(AsyncFutureTask.java:70)
at org.jboss.threads.AsyncFutureTask.get(AsyncFutureTask.java:267)
at org.jboss.as.controller.client.impl.AbstractDelegatingAsyncFuture.get(AbstractDelegatingAsyncFuture.java:57)
at org.jboss.as.controller.client.impl.AbstractModelControllerClient.executeForResult(AbstractModelControllerClient.java:147)

       

      The strange thing is the reload operation is actually successful so the server is properly configured and running in normal mode. However, because this particular exception is not handled the launch process aborts, which normally would make sense, so it never gets to the app deployment step which results in a server with no apps deployed [I undeploy any previously deployed apps during the configuration phase, due to needing a specialized deployment order, if there is a better approach I would be interested], which is how I first noticed this problem.

       

      A simplified view of the client program essentially looks like this:

       

      class Launcher {

     private ModelControllerClient client = ...;

     void launch() throws LaunchException, ... {
          waitForServer();
          configure();
          //This sometimes fails
          reload();
          waitForServer();
          deployApps();
     }

     void reload() throws LaunchException, ... {
          final ModelNode reloadOperation = Operations.createOperation("reload");

          //This call sometimes fails due to a java.util.concurrent.CancellationException
          final ModelNode reloadResult = client.execute(reloadOperation);

          //Handle reload result, throw LaunchException if result is unsuccessful
          ...
     }

}

       

      What I am wondering is if I should be doing something different with the reload operation, such as using a different ModelControllerClient method, or is this an exception I should be catching and handling? Also, is there anything in newer releases that address this or recommended changes when using the API to do this?

       

      In searching for information about this, most of what I have come across has been about JBoss CLI, which I suppose would be facing the same underlying issue, like this one [AS7-1833] CLI usability: operations that shutdown the server-side management interface result in "Communication error" … That one suggests using a different reload operation ["reload" versus ":reload"], which makes me wonder if I should be doing something different with the operation, though I am not sure what the native management API equivalent would be. At the same time I also see [WFLY-4732] Potential race in test suite where ops can execute before server has reloaded - JBoss Issue Tracker  which is using the native management API in almost exactly the same way that I am, but catches and ignores CancellationExceptions on reload, which maybe suggests implementing different exception handling.

       

      Any information on this would be appreciated.