Parallel deployments
kabirkhan Jun 16, 2009 12:44 PMI've started taking a look at parallel deployments. My initial idea was to wrap the full install process in a separate thread, by modifying the following method in AbstractController
protected void install(ControllerContext context, boolean trace) throws Throwable { boolean asynch = false; if (context.getMode() == ControllerMode.ASYNCHRONOUS) { if (executor != null) { ControllerContextTask task = new InstallControllerContextTask(context, trace); executor.execute(task); asynch = true; } else { log.warn("Uninstalling a context with ControllerMode.ASYNCHRONOUS but no executor in controller: " + context); } } if (!asynch) { internalInstall(context, trace); } } private void internalInstall(ControllerContext context, boolean trace) throws Throwable { //Original contents of install(ControllerContext, boolean) } abstract class ControllerContextTask implements Runnable { ControllerContext context; boolean trace; public ControllerContextTask(ControllerContext context, boolean trace) { this.context = context; this.trace = trace; } public ControllerContext getControllerContext() { return context; } } class InstallControllerContextTask extends ControllerContextTask { public InstallControllerContextTask(ControllerContext context, boolean trace) { super(context, trace); } public void run() { try { System.out.println(Thread.currentThread().getName() + " starting " + context.getName() + "=================="); internalInstall(context, trace); System.out.println(Thread.currentThread().getName() + " called internalInstall" + "=================="); } catch (Throwable e) { log.error("Problem installing context asynchronously " + context); } } }
However, this "does not work". In my tests I am deploying two ControllerContexts with ControllerMode.ASYNCHRONOUS, using custom ControllerContext and ControllerContextActions implementations, where I am sleeping for a long time upon entry to the different states, and found that only one thread gets access to this at one time. My initial thoughts were that there must be something wrong with the read/write locks, but they are fine.
I also added some output to this method which I use below
protected boolean resolveContexts(ControllerState fromState, ControllerState toState, boolean trace) { ... if (resolved.isEmpty() == false) { for (ControllerContext context : resolved) { ... else if (installing.add(context) == false) { System.out.println(Thread.currentThread().getName() + " already installed " + context.getName() + "!!!!!!!!!!!!!!!!!"); if (trace) log.trace("Already installing " + name + " for " + toState.getStateString()); } else { System.out.println(Thread.currentThread().getName() + " adding to installing " + context.getName() + "!!!!!!!!!!!!!!!!!"); toProcess.add(context); } } try { if (toProcess.isEmpty() == false) { for (Iterator<ControllerContext> iter = toProcess.iterator(); iter.hasNext(); ) { try { ... } finally { installing.remove(context); System.out.println(Thread.currentThread().getName() + " remove from installing " + context.getName() + "!!!!!!!!!!!!!!!!!"); } } } } finally { ... } }
What is actually happening is that both threads start up:
1 DEBUG [AsynchronousTestCase] ==== setUp org.jboss.test.dependency.controller.test.AsynchronousTestCase ==== 37 DEBUG [AsynchronousTestCase] ==== Starting testTwoAsynchronousContexts ==== pool-1-thread-1 starting Bean1================== pool-1-thread-2 starting Bean2==================
but one of these wins in adding the beans to AbstractController.installing in resolveContexts()
pool-1-thread-2 adding to installing Bean2!!!!!!!!!!!!!!!!! pool-1-thread-2 adding to installing Bean1!!!!!!!!!!!!!!!!!
Meaning the other thread has nothing to install, and completes
pool-1-thread-1 already installed Bean1!!!!!!!!!!!!!!!!! pool-1-thread-1 already installed Bean2!!!!!!!!!!!!!!!!! pool-1-thread-1 called internalInstall==================
The only remaining thread then takes over and installs both beans sequentially:
pool-1-thread-2 remove from installing Bean2!!!!!!!!!!!!!!!!! pool-1-thread-2 remove from installing Bean1!!!!!!!!!!!!!!!!! pool-1-thread-2 adding to installing Bean2!!!!!!!!!!!!!!!!! pool-1-thread-2 adding to installing Bean1!!!!!!!!!!!!!!!!! pool-1-thread-2 Bean2 sleeping 4000 Finished sleeping pool-1-thread-2 Bean2 sleeping 100 pool-1-thread-2 Bean2 sleeping 300 Bean2 enters DESCRIBE 1 pool-1-thread-2 remove from installing Bean2!!!!!!!!!!!!!!!!! pool-1-thread-2 remove from installing Bean2!!!!!!!!!!!!!!!!! pool-1-thread-2 Bean1 sleeping 300 Bean1 enters DESCRIBE 2 pool-1-thread-2 remove from installing Bean1!!!!!!!!!!!!!!!!! ....
Next, rather than trying to parallelize the whole install sequence, I thought it might make more sense to target when the context is due to enter a state. My understanding is that this is for beans that have heavy initialization (e.g. JGroups), so the important part will be invoking the ControllerContextAction which in turn may invoke create(), start(), install() etc., i tried something like:
protected void install(ControllerContext context, ControllerState fromState, ControllerState toState) throws Throwable { ... //Check ControllerMode and run in executor if ASYNCHRONOUS context.install(fromState, toState); }
However, this fails since if for example the DESCRIBE phase takes 20 seconds, the executor will return right away and the controller will enter the next state, meaning that if there are no sleeps in the INSTANTIATE, CONFIGURE, CREATE, START and INSTALLED states, they could complete before the DESCRIBE phase. There is a fair bit of housekeeping stuff going on following the call to context.install() which gets confused when I do it this way.
Finally, I have had some success with a revised version of the first version, where I modify the InstallControllerContextTask to record the contexts being installed against the executor thread being used, and checking in resolveContexts (where it is doing the 'installing' check) to see if the context being installed should be managed by the current thread. I will need to investigate a bit more though, since I think there might be some problems if an asynchronous context is being installed and lacking dependencies that come from another asynchronous context or an automatic context (done in main thread). I'll post some more about that once I see what happens.