Asynchronous results from executing a deployment plan
brian.stansberry Sep 7, 2010 10:15 AMOne of the main things I've been struggling with when implementing deployments is how to provide information to the caller on the results of executing a deployment plan. In the deployment API discussed at https://community.jboss.org/thread/155937?tstart=0 the StandaloneDeploymentPlan.execute(DeploymentPlan) method returns a DeploymentPlanResult object. That's straightforward enough; what's interesting is making getting the details of those results asynchronous.
There are two facets of the asynchronous problem:
1) As discussed on https://community.jboss.org/thread/154922?tstart=0 there's a desire to immediately return control to the user and let them come back later to check for results. I'm looking at doing that by encapsulating the result details in a Future and executing the plan on another thread. Simple enough.
2) Even if we made the caller always block waiting for the results, it's still complex, because the deployment process itself is asynchronous. More specifically, installing the services generated from the deployment is multi-threaded -- MSC breaks down all service start/stop work into tasks that are executed by threads from an Executor. So, the thread that's executing the deployment plan and trying to assemble the results can't just invoked BatchBuilder.install() and assume everything is done when that call return. After install() returns that thread needs to find a way to detect what all the services associated with the deployment are and monitor their status as other threads actually register and start them.
The 2) issue is the focus of the rest of this post.
The approach I'm looking at using is making use of the o.j.as.deployment.DeploymentService class to facilitate this. Currently that class is being used as a sort of empty placeholder on which all other services associated with a deployment depend. Telling the MSC to stop/remove a DeploymentService instance is thus a simple way to trigger removal of all the associated services. This is how undeploy and rollback of a failed deployment are working.
What I'm looking at doing is giving DeploymentService a richer set of behaviors. Basically giving it the ability to track what the services are that were associated with a deployment and an API that lets callers find out about those services. Users interested in finding out details of the results of executing a deployment plan could indirectly call into that API.
The DeploymentService learns about the services associated with a deployment by getting callbacks from a ServiceListener that is registered with the sub-batch that's actually doing the deployment:
public void activate(final ServiceActivatorContext context) { ....... final BatchBuilder batchBuilder = context.getBatchBuilder(); // Create deployment service final ServiceName deploymentServiceName = DeploymentService.SERVICE_NAME.append(deploymentName); DeploymentService deploymentService = new DeploymentService(); batchBuilder.addService(deploymentServiceName, deploymentService); // Create a sub-batch for this deployment final BatchBuilder deploymentSubBatch = batchBuilder.subBatchBuilder(); // Setup a batch level dependency on deployment service deploymentSubBatch.addDependency(deploymentServiceName); // Let deploymentService listen to services in the subbatch deploymentSubBatch.addListener(deploymentService.getDependentStartupListener()); // Add a deployment failure listener to the batch deploymentSubBatch.addListener(new DeploymentFailureListener(deploymentServiceName)); ..... go on and create deployment unit and pass it to deployer chain
Important (i.e. new) bit is in bold.
The DeploymentService (and the listener class used above) look like this:
public class DeploymentService implements Service<DeploymentService> { public static final ServiceName SERVICE_NAME = ServiceName.JBOSS.append("deployment"); private static Logger logger = Logger.getLogger("org.jboss.as.deployment"); private final Map<ServiceName, ServiceController<?>> dependents = new HashMap<ServiceName, ServiceController<?>>(); /** Dependent services that have not yet reached a terminal state in their initial startup (UP, FAILED, DOWN, REMOVED) */ private final Set<ServiceName> incompleteDependents = new HashSet<ServiceName>(); private final Lock lock = new ReentrantLock(); private final Condition startupCondition = lock.newCondition(); private final Condition stoppedCondition = lock.newCondition(); /** Whether start() has been invoked since initialization or the last stop() call */ private boolean started = false; /** Whether stop() has been invoked since initialization or the last start() call */ private boolean stopped = false; /** * Start the deployment. This will re-mount the deployment root if service is restarted. * * @param context The start context * @throws StartException if any problems occur */ public void start(StartContext context) throws StartException { lock.lock(); try { started = true; stopped = false; startupCondition.notifyAll(); } finally { lock.unlock(); } } /** * Stop the deployment. This will close the virtual file mount. * * @param context The stop context */ public void stop(StopContext context) { lock.lock(); try { stopped = true; started = false; stoppedCondition.notifyAll(); } finally { lock.unlock(); } } /** {@inheritDoc} **/ public DeploymentService getValue() throws IllegalStateException { return this; } /** * Blocks until all services associated with this deployment have * completed startup (not necessarily successfully). * * @throws InterruptedException */ public void awaitDependentStartup() throws InterruptedException { lock.lock(); try { while (!stopped && (!started || incompleteDependents.size() > 0)) { if (onlyNeverMode()) { break; } startupCondition.await(); } } finally { lock.unlock(); } } /** * Blocks until all services associated with this deployment have * completed startup (not necessarily successfully) or the specified * timeout occurs. * * @throws InterruptedException */ public void awaitDependentStartup(long timeout, TimeUnit timeUnit) throws InterruptedException { lock.lock(); try { while (!stopped && (!started || incompleteDependents.size() > 0)) { if (onlyNeverMode()) { break; } startupCondition.await(timeout, timeUnit); } } finally { lock.unlock(); } } /** * Blocks until this service is stopped. * * @throws InterruptedException */ public void awaitStop() throws InterruptedException { lock.lock(); try { while (!stopped) { stoppedCondition.await(); } } finally { lock.unlock(); } } /** * Blocks until this service is stopped or the specified * timeout occurs. * * @throws InterruptedException */ public void awaitStop(long timeout, TimeUnit timeUnit) throws InterruptedException { lock.lock(); try { while (!stopped) { stoppedCondition.await(timeout, timeUnit); } } finally { lock.unlock(); } } /** * Gets any exceptions that occurred during start of the services that * are associated with this deployment. * * @return the exceptions keyed by the name of the service. Will not be <code>null</code> */ public Map<ServiceName, StartException> getDependentStartupExceptions() { lock.lock(); try { Map<ServiceName, StartException> result = new HashMap<ServiceName, StartException>(); for (Map.Entry<ServiceName, ServiceController<?>> entry : dependents.entrySet()) { StartException se = entry.getValue().getStartException(); if (se != null) result.put(entry.getKey(), se); } return result; } finally { lock.unlock(); } } /** * Gets the {@link ServiceController.State state} of the services that * are associated with this deployment. * * @return the services and their current state. Will not be <code>null</code> */ public Map<ServiceName, ServiceController.State> getDependentStates() { lock.lock(); try { Map<ServiceName, ServiceController.State> result = new HashMap<ServiceName, ServiceController.State>(dependents.size()); for (Map.Entry<ServiceName, ServiceController<?>> entry : dependents.entrySet()) { result.put(entry.getKey(), entry.getValue().getState()); } return result; } finally { lock.unlock(); } } /** * Gets a {@link ServiceListener} that can track startup events for * services associated with the deployment this service represents. This * listener should * be associated with a {@link BatchBuilder#subBatchBuilder() sub-batch} * of this services batch that encapsulates the creation of services that * are associated with the deployment. * * @return the service listener */ public ServiceListener<Object> getDependentStartupListener() { return new DependentServiceListener(); } /** Checks whether all incomplete dependents are Mode.NEVER. Must be called with the lock held */ private boolean onlyNeverMode() { int ever = incompleteDependents.size(); for (ServiceName name : incompleteDependents) { ServiceController<?> controller = dependents.get(name); if (controller == null || controller.getMode() == Mode.NEVER) ever--; } return ever == 0; } private class DependentServiceListener extends AbstractServiceListener<Object> { /** * This will be called for all dependent services before the * BatchBuilder.install() call returns. So at that point we know what * the dependent services are; other threads will invoke the other * callbacks are services are started. */ @Override public void listenerAdded(ServiceController<? extends Object> controller) { lock.lock(); try { dependents.put(controller.getName(), controller); incompleteDependents.add(controller.getName()); } finally { lock.unlock(); } } @Override public void serviceFailed(ServiceController<? extends Object> controller, StartException reason) { lock.lock(); try { incompleteDependents.remove(controller.getName()); startupCondition.notifyAll(); } finally { lock.unlock(); } } @Override public void serviceRemoved(ServiceController<? extends Object> controller) { lock.lock(); try { incompleteDependents.remove(controller.getName()); startupCondition.notifyAll(); } finally { lock.unlock(); } } @Override public void serviceStopped(ServiceController<? extends Object> controller) { lock.lock(); try { incompleteDependents.remove(controller.getName()); startupCondition.notifyAll(); } finally { lock.unlock(); } } } }
Besides the listener, the other interesting bit in the above are the awaitXXX methods. Those are what allow a caller that actually wants to find out what happened with a deployment to block until the asynchronous MSC tasks complete.
The awaitStop() methods are straightforward enough.
The awaitDependentStartup() implementation is more subtle. It depends on the fact that the listener's listenerAdded() method should be invoked passing in any services associated with the deployment before the BatchBuilder.install() method returns. My understanding of how MSC works tells me this is the case -- all listeners associated with a batch are passed to ServiceBuilderImpl and the listenerAdded method is invoked as part of ServiceBuilderImpl.doCreate(). This is all done as part of executing BatchBuilder.install(). This seems like a logical and necessary part of the BatchBuilder.install() contract; it would be good if it were documented as such.
There is a subtle race here though. In BatchBuilder.install() the DeploymentService itself could have all dependencies satisfied and tasks executed by another thread to start it before the thread executing install() processes the dependent services and calls listenerAdded(). If a caller invoked awaitDependentStartup() during this window, it would return even though the dependent services are not yet started. I'm dealing with that by having the thread that executes BatchBuilder.install() not expose DeploymentService.awaitDependentStartup() to any calling threads until the install() method returns.