1 2 Previous Next 16 Replies Latest reply on Mar 27, 2008 10:34 AM by Ales Justin

    Concurrent deployments

    Bela Ban Master

      The MC's dependency management currently starts all services sequentially, according to their dependencies.

      In JBoss 4.3, we have 4 JGroups channels, which take 5 secs *each* to start. However, they are completely independent from each other, so they could be started concurrently. This would bring the time needed down from 20 secs to 5 secs.

      I imagine we could do this for all independent services, a.k.a subtrees in the dependency tree. This would speed up start of JBoss significantly !

      Can we do this ?

      Cheers

        • 1. Re: Concurrent deployments
          Adrian Brock Master

           

          "bela@jboss.com" wrote:
          The MC's dependency management currently starts all services sequentially, according to their dependencies.


          Actually some services, e.g. MDBs are started asynchronously,
          but this is handled by the relevant container rather than the Micro-kernel/container.


          In JBoss 4.3, we have 4 JGroups channels, which take 5 secs *each* to start. However, they are completely independent from each other, so they could be started concurrently. This would bring the time needed down from 20 secs to 5 secs.

          I imagine we could do this for all independent services, a.k.a subtrees in the dependency tree. This would speed up start of JBoss significantly !

          Can we do this ?



          This question has been asked many times before.
          The answer is in principle yes, but in practice no.

          The main the practicality problems are:

          * Not all services define their dependencies properly. It's only the implicit
          sequential order that makes them work correctly.

          * The Sun classloading doesn't work very well concurrently. We avoid a number
          of potential problems by doing a sequential load of most of the key classes
          during the bootstrap.
          We have workarounds, but you can still see these issues on some
          platforms.

          * The "big ball of mud" unified classloader depends for consistency/predictability
          on having the classloaders created in order. It is this order that is used to
          search for global classes. If these were registered concurrently across threads
          the search order would be unpredicatble across different reboots.
          In fact, 4.x has no mechanism for classloading dependency at all.

          * The old 4.x MicroKernel has some thread safety issues,
          e.g. invocations of create/start are done in synchronized blocks which could
          cause deadlock problems, there are others related to ConcurrentModificationExceptions.
          This is not an issue with the 5.x Microcontainer.

          * We'd need to find a way for the BarrierService and "Server Started" notification
          to know when all the startup threads have finished, otherwise
          they would be broken.

          * Finally, less of a practical issue, but more a generic note. Using mulitple threads
          for startup will only work if there are multiple cpus/cores. In non SMP environments
          the bootstrap could be slower with multiple competing threads, although
          it may be able to take advantage of times during disk I/O waits
          to do some processing on other threads?

          • 2. Re: Concurrent deployments
            Brian Stansberry Master

             

            although it may be able to take advantage of times during disk I/O waits to do some processing on other threads?


            Not disk I/O, but threads blocking waiting for messages that will never arrive. The reason JGroups channels take a long time to start is they send a discovery packet to the network and block waiting a few seconds (configurable) to get a few replies. The first node in the cluster will of course receive no replies, so for each channel it waits a few seconds.

            BTW, I think Bela's intent here was to discuss this only in context of an improvement in AS 5.x.

            • 3. Re: Concurrent deployments
              Adrian Brock Master

              Couldn't you just implement this anyway in your services,
              using the 2PC create/start lifecycle?

              private volatile Thread startupThread;
              
              public void create() throws Exception
              {
               startupThread = new Thread(new Runnable()
               {
               public void run()
               {
               // start channel here
               }
               });
               startupThread.start();
              }
              
              public void start() throws Exception
              {
               startThread.join(); // wait for startup to complete before injecting ourselves onto others
               startThread = null;
              }
              


              Of course, we could do better (e.g. using thread pools and "optimizing"
              related work across threads based on dependencies) if this was a feature of the MC.

              • 4. Re: Concurrent deployments
                Adrian Brock Master

                A simple mechansim that I can think of would to have a new ControllerMode
                http://viewvc.jboss.org/cgi-bin/viewvc.cgi/jbossas/trunk/jboss-dependency/src/main/org/jboss/dependency/spi/ControllerMode.java?revision=31727&view=markup&pathrev=31727
                which you could add to your deployment.

                <deployment mode="asynchronous">
                 <!-- beans here -->
                </deployment>
                


                which would work like the "automatic" mode, except it submits the work
                to a background thread pool.

                In this scenario, it would largely be "self-optimizing" since
                it would automatically advance related dependencies that are eligible once
                these beans reach the relevant stages.

                But this wouldn't work so well for the old JMX MBeans which are hardwired to
                operate in "manual" mode driven by the "SARDeployer"/ServiceController
                for backwards compatibilty reasons.

                The Deployments (which are also managed for dependency by the MC)
                also run in "manual" mode which might be a good thing
                since it would still ensure that classloaders are constructed in a predictable order
                on one thread.

                • 5. Re: Concurrent deployments
                  Brian Stansberry Master

                  The ControllerMode approach sounds better. I expect starting a thread in create() won't help much in many cases, as the main thread will quickly proceed to start and then block.

                  Unfortunately, JBM opens 2 of the 3 underlying JChannels in AS 5, and it's deploying as an mbean.

                  HAPartition is opening the 3rd channel and definitely can benefit from this.

                  The other services (JBC) are opening MuxChannels, which really just multiplex on the underlying JChannel and thus don't block. But, we want to move to using the JGroups shared transport, in which case each service will have its own JChannel; I need to think through how to use the ControllerMode to improve the startup speed.

                  "adrian@jboss.org" wrote:

                  The Deployments (which are also managed for dependency by the MC)
                  also run in "manual" mode which might be a good thing
                  since it would still ensure that classloaders are constructed in a predictable order
                  on one thread.


                  Did you mean the "Deployers" here?

                  • 6. Re: Concurrent deployments
                    Adrian Brock Master

                     

                    "bstansberry@jboss.com" wrote:

                    "adrian@jboss.org" wrote:

                    The Deployments (which are also managed for dependency by the MC)
                    also run in "manual" mode which might be a good thing
                    since it would still ensure that classloaders are constructed in a predictable order
                    on one thread.


                    Did you mean the "Deployers" here?


                    No, I mean't the deployments. Each deployment is managed by the MC
                    as it goes through its states. It is "manual" because it will only advance
                    to the next state when the "MainDeployer' says it can.

                    During startup The MainDeployer tells all deployments to advance to Parse,
                    then all to advance to Describe, repeated for each stage.
                    But the MC may veto the request if a deployment is missing a dependency.
                    e.g. a deployment could have a classloading dependency on a different deployment
                    that is not deployed (osgi style classloading rules).

                    But in general this all happens currently on the bootstrap or hot deployment scanner
                    thread.

                    • 7. Re: Concurrent deployments
                      Bela Ban Master

                       

                      "adrian@jboss.org" wrote:


                      * Not all services define their dependencies properly. It's only the implicit
                      sequential order that makes them work correctly


                      The let's make the default sequential unless we explicitly tag a deployment as asynchronous (as you mention further down). Although I'd prefer it to be the other way round.


                      * The Sun classloading doesn't work very well concurrently. We avoid a number of potential problems by doing a sequential load of most of the key classes during the bootstrap.


                      What prevents us from loading those key classes up front, and then deploying in parallel ?



                      * The "big ball of mud" unified classloader depends for consistency/predictability on having the classloaders created in order. It is this order that is used to search for global classes. If these were registered concurrently across threads the search order would be unpredicatble across different reboots.


                      The big ball of mud is controlled by us, right ? So we can change it. The above sounds like a kludge anyway... (Caveat, I'm not a classloader expert... :-))

                      [quote
                      * The old 4.x MicroKernel has some thread safety issues,
                      e.g. invocations of create/start are done in synchronized blocks which could cause deadlock problems, there are others related to ConcurrentModificationExceptions.
                      This is not an issue with the 5.x Microcontainer.


                      Then the feature is for AS 5 only, and not available in 4.x.


                      * We'd need to find a way for the BarrierService and "Server Started" notification to know when all the startup threads have finished, otherwise
                      they would be broken.


                      Yes. One big barrier would probably be enough to sync on for all threads when they're completed, at least in a first impl. This would tell us that startup of JBossAS is 'done'.


                      * Finally, less of a practical issue, but more a generic note. Using mulitple threads for startup will only work if there are multiple cpus/cores. In non SMP environments the bootstrap could be slower with multiple competing threads, although
                      it may be able to take advantage of times during disk I/O waits
                      to do some processing on other threads?


                      • 8. Re: Concurrent deployments
                        Bela Ban Master

                         

                        "adrian@jboss.org" wrote:
                        Couldn't you just implement this anyway in your services,
                        using the 2PC create/start lifecycle?

                        private volatile Thread startupThread;
                        
                        public void create() throws Exception
                        {
                         startupThread = new Thread(new Runnable()
                         {
                         public void run()
                         {
                         // start channel here
                         }
                         });
                         startupThread.start();
                        }
                        
                        public void start() throws Exception
                        {
                         startThread.join(); // wait for startup to complete before injecting ourselves onto others
                         startThread = null;
                        }
                        


                        Of course, we could do better (e.g. using thread pools and "optimizing"
                        related work across threads based on dependencies) if this was a feature of the MC.


                        No, since create only creates the channel, but start() connects it (time consuming operation), we'd block the main thread for the same time with the join() as if we deployed on the main thread directly.

                        We could create a separate threads and not join() it, but that might have unintended consequences, such as channels not being ready when JBossCache (which has a dep on a channel) starts...

                        • 9. Re: Concurrent deployments
                          David Lloyd Master

                           

                          "adrian@jboss.org" wrote:
                          * Not all services define their dependencies properly. It's only the implicit sequential order that makes them work correctly.


                          If they don't define their dependencies properly, what guarantee is there that they will even work at all?

                          "adrian@jboss.org" wrote:
                          * Finally, less of a practical issue, but more a generic note. Using mulitple threads for startup will only work if there are multiple cpus/cores. In non SMP environments the bootstrap could be slower with multiple competing threads, although it may be able to take advantage of times during disk I/O waits
                          to do some processing on other threads?


                          Two things - first, I don't even know where I'd be able to buy a server-class system (heck even a desktop-class system come to think of it) that didn't have at least two cores (if not two or more multi-core CPUs). But that issue aside, any service that has non-CPU bound tasks (which includes not just disk I/O but network I/O) could possibly see a performance benefit from concurrent startup, even on a single-core system. And in any case we're talking about service deployment - I doubt the additional overhead of context switching between multiple threads could possibly make a measurable impact on performance. Especially in comparison to examples like Bela's 5-second timeout.

                          Also one could reasonably expect that rather than firing off a thread for every task, there would be a thread pool Executor of some sort. The thread pool could be sized in proportion to the number of cores available, or in some configurable fashion. Then the simple rule would be, any deployment task with no remaining dependencies gets put into the queue for execution.

                          I don't really understand the hairy details behind some of the classloader problems, but surely they could be worked around by imposing some reliance on the dependency system for them as well?

                          I agree with Bela - a concurrent startup would be hugely beneficial for startup time, not to mention just being cool. In fact when first learning about the MC and its dependency system, I had simply assumed that dependencies were resolved concurrently and never gave it a moment's thought. It wasn't until I spoke with Bela at JBW that I learned otherwise.

                          • 10. Re: Concurrent deployments
                            Adrian Brock Master

                             

                            "david.lloyd@jboss.com" wrote:

                            I don't really understand the hairy details behind some of the classloader problems, but surely they could be worked around by imposing some reliance on the dependency system for them as well?


                            If all the deployments specified their classloading dependencies
                            (osgi style) the problem would go away.

                            This also means automagically adding javaee (and the relevant container)
                            classloading dependencies into ejb/war deployments etc.
                            since we can't expect users to do this for themselves.

                            It's doable, but I doubt it is going to happen soon?

                            Another side affect of this, is that those deployments that
                            explictly list their package exports wouldn't have to go through the
                            scan of the filesystem/jars to determine what packages they export.

                            • 11. Re: Concurrent deployments
                              Adrian Brock Master

                               

                              "david.lloyd@jboss.com" wrote:
                              "adrian@jboss.org" wrote:
                              * Not all services define their dependencies properly. It's only the implicit sequential order that makes them work correctly.


                              If they don't define their dependencies properly, what guarantee is there that they will even work at all?


                              They use the "deterministic" implicit rules, which means you can test
                              it works for single bootstrap, but if you try to redeploy something that
                              only has implicit dependencies, you'll find it doesn't understand what is going on :-).


                              • 12. Re: Concurrent deployments
                                Adrian Brock Master

                                 

                                "adrian@jboss.org" wrote:

                                If all the deployments specified their classloading dependencies
                                (osgi style) the problem would go away.


                                The proposed mavenisation of the JBossAS build might help in determining
                                what these dependencies really are.

                                • 13. Re: Concurrent deployments
                                  Ales Justin Master

                                   

                                  "adrian@jboss.org" wrote:
                                  A simple mechansim that I can think of would to have a new ControllerMode
                                  http://viewvc.jboss.org/cgi-bin/viewvc.cgi/jbossas/trunk/jboss-dependency/src/main/org/jboss/dependency/spi/ControllerMode.java?revision=31727&view=markup&pathrev=31727
                                  which you could add to your deployment.

                                  <deployment mode="asynchronous">
                                   <!-- beans here -->
                                  </deployment>
                                  


                                  which would work like the "automatic" mode, except it submits the work
                                  to a background thread pool.

                                  In this scenario, it would largely be "self-optimizing" since
                                  it would automatically advance related dependencies that are eligible once
                                  these beans reach the relevant stages.

                                  How would this relate with KernelDeploymentDeployer, meaning that we need to 'convince' it not to split up the deployment into separate non-related beans.
                                  Or how do you plan to group all beans from a deployment marked with 'asynch' and install them in the same background thread?

                                  • 14. Re: Concurrent deployments
                                    Adrian Brock Master

                                     

                                    "alesj" wrote:

                                    How would this relate with KernelDeploymentDeployer, meaning that we need to 'convince' it not to split up the deployment into separate non-related beans.
                                    Or how do you plan to group all beans from a deployment marked with 'asynch' and install them in the same background thread?


                                    Why would you need to?

                                    You simply have an Executor backed by a thread pool + queue on the controller.
                                    You submit the task to the executor then it will do something like:

                                    <deployment mode="Asychronous">
                                     <bean name="A"/>
                                     <bean name="B"><depends>A</bean>
                                    </deployment>
                                    


                                    B is not satisified until A is ready so you can't submit it to the thread pool.

                                    However submitting A to the thread pool will reworkout that B is now satisified
                                    and can do it on the same thread.

                                    The real issue is when you have some other thread come in during the intermediate
                                    step. i.e. A is now done, but we haven't figured out that B will be done on the same
                                    thread.
                                    Some other thread might say, I can do B so you have two threads working on B.

                                    I believe this isn't really possible because of the lock in the controller, but we
                                    don't really many tests in this area so using ione of my laws of programming:

                                    Not Tested == Doesn't work

                                    P.S. You don't need to actually make concurrent deployment work for 2.0.0.GA
                                    You just need to add the enum value and through an error saying
                                    "not implemented yet" when somebody tries to use it.
                                    The important thing is that we don't change the api after 2.0.0.GA
                                    we can implement missing features without changing the api if the api already exists.

                                    1 2 Previous Next