1 2 3 Previous Next 34 Replies Latest reply on Aug 24, 2009 8:08 AM by alesj

    Parallel deployments

    kabirkhan

      I've started taking a look at parallel deployments. My initial idea was to wrap the full install process in a separate thread, by modifying the following method in AbstractController

       protected void install(ControllerContext context, boolean trace) throws Throwable
       {
       boolean asynch = false;
       if (context.getMode() == ControllerMode.ASYNCHRONOUS)
       {
       if (executor != null)
       {
       ControllerContextTask task = new InstallControllerContextTask(context, trace);
       executor.execute(task);
       asynch = true;
       }
       else
       {
       log.warn("Uninstalling a context with ControllerMode.ASYNCHRONOUS but no executor in controller: " + context);
       }
       }
      
       if (!asynch)
       {
       internalInstall(context, trace);
       }
       }
      
       private void internalInstall(ControllerContext context, boolean trace) throws Throwable
       {
       //Original contents of install(ControllerContext, boolean)
       }
      
       abstract class ControllerContextTask implements Runnable
       {
       ControllerContext context;
       boolean trace;
      
       public ControllerContextTask(ControllerContext context, boolean trace)
       {
       this.context = context;
       this.trace = trace;
       }
      
       public ControllerContext getControllerContext()
       {
       return context;
       }
       }
      
       class InstallControllerContextTask extends ControllerContextTask
       {
       public InstallControllerContextTask(ControllerContext context, boolean trace)
       {
       super(context, trace);
       }
      
       public void run()
       {
       try
       {
       System.out.println(Thread.currentThread().getName() + " starting " + context.getName() + "==================");
       internalInstall(context, trace);
       System.out.println(Thread.currentThread().getName() + " called internalInstall" + "==================");
       }
       catch (Throwable e)
       {
       log.error("Problem installing context asynchronously " + context);
       }
       }
       }
      



      However, this "does not work". In my tests I am deploying two ControllerContexts with ControllerMode.ASYNCHRONOUS, using custom ControllerContext and ControllerContextActions implementations, where I am sleeping for a long time upon entry to the different states, and found that only one thread gets access to this at one time. My initial thoughts were that there must be something wrong with the read/write locks, but they are fine.

      I also added some output to this method which I use below
       protected boolean resolveContexts(ControllerState fromState, ControllerState toState, boolean trace)
       {
       ...
       if (resolved.isEmpty() == false)
       {
       for (ControllerContext context : resolved)
       {
       ...
       else if (installing.add(context) == false)
       {
       System.out.println(Thread.currentThread().getName() + " already installed " + context.getName() + "!!!!!!!!!!!!!!!!!");
       if (trace)
       log.trace("Already installing " + name + " for " + toState.getStateString());
       }
       else
       {
       System.out.println(Thread.currentThread().getName() + " adding to installing " + context.getName() + "!!!!!!!!!!!!!!!!!");
       toProcess.add(context);
       }
       }
       try
       {
       if (toProcess.isEmpty() == false)
       {
       for (Iterator<ControllerContext> iter = toProcess.iterator(); iter.hasNext(); )
       {
       try
       {
       ...
       }
       finally
       {
       installing.remove(context);
       System.out.println(Thread.currentThread().getName() + " remove from installing " + context.getName() + "!!!!!!!!!!!!!!!!!");
       }
       }
       }
       }
       finally
       {
       ...
       }
       }
      


      What is actually happening is that both threads start up:
      1 DEBUG [AsynchronousTestCase] ==== setUp org.jboss.test.dependency.controller.test.AsynchronousTestCase ====
      37 DEBUG [AsynchronousTestCase] ==== Starting testTwoAsynchronousContexts ====
      pool-1-thread-1 starting Bean1==================
      pool-1-thread-2 starting Bean2==================
      

      but one of these wins in adding the beans to AbstractController.installing in resolveContexts()
      pool-1-thread-2 adding to installing Bean2!!!!!!!!!!!!!!!!!
      pool-1-thread-2 adding to installing Bean1!!!!!!!!!!!!!!!!!
      

      Meaning the other thread has nothing to install, and completes
      pool-1-thread-1 already installed Bean1!!!!!!!!!!!!!!!!!
      pool-1-thread-1 already installed Bean2!!!!!!!!!!!!!!!!!
      pool-1-thread-1 called internalInstall==================
      

      The only remaining thread then takes over and installs both beans sequentially:
      pool-1-thread-2 remove from installing Bean2!!!!!!!!!!!!!!!!!
      pool-1-thread-2 remove from installing Bean1!!!!!!!!!!!!!!!!!
      pool-1-thread-2 adding to installing Bean2!!!!!!!!!!!!!!!!!
      pool-1-thread-2 adding to installing Bean1!!!!!!!!!!!!!!!!!
      pool-1-thread-2 Bean2 sleeping 4000
      Finished sleeping
      pool-1-thread-2 Bean2 sleeping 100
      pool-1-thread-2 Bean2 sleeping 300
      Bean2 enters DESCRIBE 1
      pool-1-thread-2 remove from installing Bean2!!!!!!!!!!!!!!!!!
      pool-1-thread-2 remove from installing Bean2!!!!!!!!!!!!!!!!!
      pool-1-thread-2 Bean1 sleeping 300
      Bean1 enters DESCRIBE 2
      pool-1-thread-2 remove from installing Bean1!!!!!!!!!!!!!!!!!
      ....
      


      Next, rather than trying to parallelize the whole install sequence, I thought it might make more sense to target when the context is due to enter a state. My understanding is that this is for beans that have heavy initialization (e.g. JGroups), so the important part will be invoking the ControllerContextAction which in turn may invoke create(), start(), install() etc., i tried something like:

       protected void install(ControllerContext context, ControllerState fromState, ControllerState toState) throws Throwable
       {
       ...
       //Check ControllerMode and run in executor if ASYNCHRONOUS
       context.install(fromState, toState);
       }
      
      

      However, this fails since if for example the DESCRIBE phase takes 20 seconds, the executor will return right away and the controller will enter the next state, meaning that if there are no sleeps in the INSTANTIATE, CONFIGURE, CREATE, START and INSTALLED states, they could complete before the DESCRIBE phase. There is a fair bit of housekeeping stuff going on following the call to context.install() which gets confused when I do it this way.

      Finally, I have had some success with a revised version of the first version, where I modify the InstallControllerContextTask to record the contexts being installed against the executor thread being used, and checking in resolveContexts (where it is doing the 'installing' check) to see if the context being installed should be managed by the current thread. I will need to investigate a bit more though, since I think there might be some problems if an asynchronous context is being installed and lacking dependencies that come from another asynchronous context or an automatic context (done in main thread). I'll post some more about that once I see what happens.

        • 1. Re: Parallel deployments
          kabirkhan
          • 2. Re: Parallel deployments
            kabirkhan

            With the current setup, using dependencies, and deploying:
            A)
            -Bean1 w/ ControllerMode.ASYNCHRONOUS and a dependency on Bean2
            -Bean2 w/ControllerMode.AUTOMATIC

            Bean1 is installed in a thread until the dependency is needed, at which point the thread is returned to the pool. Once Bean2 is installed in the main thread, Bean1 goes through the remaining lifecycle in the main thread.

            B)
            -Bean1 w/ ControllerMode.AUTOMATIC and a dependency on Bean2
            -Bean2 w/ControllerMode.ASYNCHRONOUS

            Bean1 is installed in the main thread until the dependency is needed. Then I deploy Bean2, which happens in a thread, once Bean2 is installed resolveContexts(boolean) is then called from the thread, finds Bean1, but that is not associated with the thread and currently halts installation.

            I need to rethink how I'm attacking this. Maybe I'm being a bit too course-grained with where I am running ASYNCHRONOUS in a different thread, I'll see if I can make it happen later.

            • 3. Re: Parallel deployments
              dmlloyd

              I think semantically what you're after is a different approach - rather than trying to install everything in parallel, what is needed is a sort of lifecycle task abstraction.

              The logic would be something to the effect of:

              task is:
               execute lifecycle method;
               for each task that depends on me {
               remove me as a dependency;
               if no remaining dependencies {
               submit task to executor;
               }
               }
              


              In other words, one executor task per controller task. This would allow full concurrency of everything that can be expressed in terms of dependencies, including having different lifecycle methods being executed concurrently etc.

              Now I have no idea how this fits in to the existing lifecycle code. If at all. :-)

              • 4. Re: Parallel deployments
                smarlow

                On the other approaches page, instead of marking a bean as asynchronous, how about if a thread name was specified for each bean? For the first attempt at parallel deployments, we might start with two execution threads. Beans that have no thread name, execute in a default thread. The other beans will execute in the other named thread.

                The beans will need to be configured to avoid deadlocks. I prefer this logic to be in the beans as this will help us deal with object lock contention (dealing with lock contention is a different problem but very important to achieve for performance).

                • 5. Re: Parallel deployments
                  kabirkhan

                  I have something working now, and will try to break it with more tests. Before describing what I am doing, here is an outline of what the controller does in pseudocode.

                  * install() and change() both obtain the writeLock, update the ctx's required state and end up in resolveContexts(boolean)

                  void resolveContexts(boolean trace){
                   boolean resolved = true;
                   while (resolved){
                   for (state : each controller state registered in controller){
                   if (resolveContexts(state, state+1, trace){
                   resolved = true;
                   break;
                   }
                   }
                   }
                  }
                  
                  boolean resolveContexts(ControllerState fromState, ControllerState toState, boolean trace){
                   contexts = all contexts in the fromState that have not yet reached the required state, with dependencies satisfied to move to toState
                   for (ctx : contexts){
                   attempt to record the ctx as "installing"
                   if already installing (i.e. by another thread){
                   continue;
                   try{
                   incrementState(ctx, trace);
                   }finally{
                   unrecord the ctx as installing
                   }
                   }
                  }
                  
                  boolean incrementState(ControllerContext ctx, boolean trace){
                   confirm state ctx is in and which we want to move it to
                   try{
                   unlockWriteLock();
                   install(ctx, fromState, toState); //This invokes the context actions, i.e. the create(), start() methods
                   move the context to the appropriate state in the controller
                   invoke callbacks etc.
                   }finally{
                   lockWriteLock();
                   if (install caused error){
                   record ctx as failed in ctx and controller
                   return false;
                   }
                   }
                   return true;
                  }
                  

                  So basically, the call to resolveContexts(boolean) tries to increment the state of all contexts that can be moved on. If any contexts can be moved on, and succeed resolveContexts(boolean) starts again and goes on until no more contexts can be moved further.

                  I have added a wrapper around incrementState() which checks if the ctx is ASYNCHRONOUS. If not, it continues as before. If ASYNCHRONOUS, I record it as belonging to the executor, and then invoke a Runnable task using the executor.

                  The "installing" check in resolveContexts, has been upated to skip contexts belonging to the executor to avoid the asynchronous context to be picked up by the wrong thread.

                  The runnable task looks something like:
                   class InstallControllerContextTask implements Runnable
                   {
                   ControllerContext context;
                   boolean trace;
                  
                   public InstallControllerContextTask(ControllerContext context, boolean trace)
                   {
                   this.context = context;
                   this.trace = trace;
                   }
                  
                   public void run()
                   {
                   lockWrite();
                   //associate context with my thread
                   try
                   {
                   //Move the given context as far through the states towards context.requiredState() as possible
                   boolean stateChanged = installMyContext();
                  
                   //The given context had its state changed, now install anybody who might have been dependent on it
                   if (stateChanged)
                   {
                   resolveContexts(trace);
                   }
                   }
                   finally
                   {
                   //remove association between my context and thread
                   unlockWrite();
                   }
                   }
                   }
                  



                  I am currently testing that:
                  * N asynchronous contexts with no dependencies are getting installed in their own threads and that they can all be in the same lifecycle at the same time
                  * If an asynchronous context is installed with a dependency on a normal context, the asynchronous context executes in its own thread and stops when the unsatisfied dependency is needed. The dependency is then deployed, and once satisfied the controller picks up the asynchronous context again, and installs that further. Due to the wrapper around incrementState() the asynchronous context is installed in an executor thread again.
                  * If a normal context is installed with a dependency on an asynchronous context, the normal context is installed in the calling thread and stops when the unsatisfied dependency is needed. The dependency is then deployed, in an executor thread, and once installed the call to resolveContexts() at the end of the Runnable task installs the normal context. The install of the normal context after the wait for its dependencies happens in the executor thread.

                  I will add more tests on Monday.


                  • 6. Re: Parallel deployments
                    kabirkhan

                    I have a few questions regarding the executor. In the case where a context is asynchronous, and there is no executor in the controller, we decided to log a warning and to deploy the context as normal (i.e. not in a thread).

                    What to do if the executor "is full" and throws a RejectedExecutionException? Log warning and deploy as normal?

                    Also, the executor is currently only set on the main controller. How should asynchronous deployments be handled in a scoped/child controller? Search the parents until an executor is found?

                    • 7. Re: Parallel deployments
                      kabirkhan

                       

                      "kabir.khan@jboss.com" wrote:
                      I have a few questions regarding the executor. In the case where a context is asynchronous, and there is no executor in the controller, we decided to log a warning and to deploy the context as normal (i.e. not in a thread).

                      What to do if the executor "is full" and throws a RejectedExecutionException? Log warning and deploy as normal?

                      Also, the executor is currently only set on the main controller. How should asynchronous deployments be handled in a scoped/child controller? Search the parents until an executor is found?

                      I've done these as outlined

                      • 8. Re: Parallel deployments
                        alesj

                        I guess, apart from ByteMan, you can also put in the initial culprit that started off this whole discussion/impl - Bela's JGroups.
                        (if it's not too much of a config effort)

                        So that we see this "live", how it behaves with Asynch and w/o.

                        • 9. Re: Parallel deployments
                          kabirkhan

                          I have commited what I have so far, including tests using byteman.

                          I have only implemented the installs asynchronously, uninstalls are still handled the old way. I don't think uninstall is as important, but please chip in if you think I am wrong.

                          Before closing JBKERNEL-25 I will try this out in AS.

                          • 10. Re: Parallel deployments
                            smarlow

                            Nice! I'm looking forward to hearing how things go with AS! :)

                            Could you share your AS changes (if you don't check them in)? I would also like to experiment with using parallel deployment in AS.

                            -Scott

                            • 11. Re: Parallel deployments
                              kabirkhan

                              Sure,

                              I haven't built AS for a while, and am currently struggling with getting it set up in Eclipse. I've never had a problem in the past, so I am not sure what is going on.

                              Once that is done, and I can actually browse the source, I need to figure out how to set an executor in the controller.

                              If I understood Brian Stansberry correctly, I also need to modify hapartition-jboss-beans.xml

                               <bean name="HAPartition"
                               class="org.jboss.ha.framework.server.ClusterPartition"
                               mode="Asynchronous">
                              

                              and jboss-cache-manager-jboss-beans.xml
                              <bean name="CacheManager" class="org.jboss.ha.cachemanager.CacheManager" mode="Asynchronous">
                              


                              Once I can build AS in Eclipse, and run this I'll post my findings.

                              • 12. Re: Parallel deployments
                                kabirkhan

                                Currently there is no way to set the executor via the spi, you need to cast to AbstractController.

                                I could add the setExecutor() method to the Controller interface, but maybe it belongs better in an AsynchController interface (similarly to what was done for GraphController)?

                                • 13. Re: Parallel deployments
                                  kabirkhan

                                  I think there will be a corner-case issue when AS boots up. At the end, it checks to see what has not been fully deployed. If there are asynchronous deployments still being processed when the main AS thread finishes the boot, they will probably be picked up as not deployed.

                                  The AbstractController internally marks which contexts are currently being processed asynchronously, so we should probably expose a check in the spi

                                  boolean processingAsynchronousContexts()
                                  


                                  and at the end of the bootstrap do something like:

                                  //Main bootstrap code...
                                  
                                  if (controller instanceof AsynchController){
                                   while (controller.processingAsynchronousContexts()){
                                   Thread.sleep(100);
                                   }
                                  }
                                  
                                  //Report uninstalled contexts
                                  



                                  • 14. Re: Parallel deployments
                                    alesj

                                     

                                    "kabir.khan@jboss.com" wrote:

                                    I could add the setExecutor() method to the Controller interface, but maybe it belongs better in an AsynchController interface (similarly to what was done for GraphController)?

                                    No and no. :-)

                                    I don't we should expose Executor on Controller, as this is impl detail.
                                    The Controller should only expose "state machine" API.

                                    Afais AsynchController doesn't have any proper value,
                                    to only exist to expose setter/getter doesn't make sense.

                                    I guess what you could do is to create proper KernelConfig/Initializer,
                                    which knows how to supply proper Executor to new Controller instance.
                                    Or, I actually don't mind if you do a cast to AbstractContorller, as long as you do the check before.

                                    I'll think about this some more ... if there is any nicer way to handle this.

                                    1 2 3 Previous Next