1 2 3 Previous Next 36 Replies Latest reply on Feb 11, 2010 3:13 PM by brian.stansberry

    Deployment of on-demand web applications

    brian.stansberry

      I've been looking into how to deploy web application's on-demand; i.e. when the first user request that maps to the application comes in.

       

      One of the problems we need to solve to get the boot time of the AS down to a low level is the time it takes to deploy the various web applications we ship by default, particularly the admin console, which is quite heavy. We want to get the deployment time of these war's as low as possible, but some are almost surely always going to take a couple seconds or more to deploy, which is a large fraction of the overall boot time we'd like to see.

       

      An ugly solution is to not include these wars in our default profile; users would have to add them if they wanted them. Not very user-friendly.

       

      An alternate solution is to configure the web server to recognize that certain URLs target an as-yet-undeployed application, and have it call out to the ProfileService to trigger deployment of the web application. Turns out this is fairly straightforward to do. It requires:

       

      1. Fairly simple modification to the JBoss Web logic that maps requests to applications.
      2. Integration layer between JBoss Web and ProfileService
      3. Registration and activation of on-demand profiles

       

      I'll explain each in more depth

       

      JBoss Web Request Mapping Changes

       

      I've attached a jboss-web.patch that shows the proposed changes.

       

      JBoss Web / Tomcat has isolated the logic for mapping a request to the container it targets into a single class: org.apache.tomcat.util.http.mapper.Mapper. The attached patch modifies that class as follows:

       

      • Adds a new addOnDemandContext(String hostName, String path) method to allow registration of as-yet-undeployed web applications.
      • Adds a notification listener interface OnDemandContextMappingListener to Mapper's package.
      • Modifies the mapping algorithm such that when a request for an on-demand application comes in, any registered listener is notified. (The listener would presumably deploy the on-demand application, although the logic handles the case where this does not happen.) If the requested container is already deployed this logic adds a single boolean check and a single null check to the critical path, so it's quite lightweight. (The null check could even be removed by adding a return statement.)
      • Modifies the logic for when regular (i.e. non-on-demand) applications are registered so it handles the case of the regular context replacing the on-demand context.

       

      The patch doesn't include any unti tests, but the Mapper class is quite amenable to solid unit testing.

      Integration layer between JBoss Web and ProfileService

       

      I've attached a tomcat-module.patch that shows the proposed changes.

       

      Basically, this involves implementing the OnDemandContextMappingListener interface and adding hooks to access the JBoss Web Mapper objects so on-demand contexts can be registered and the listener can be registered.

       

      The patch has the TomcatService class do the wiring into JBoss Web, which was the easy approach for this prototype since TomcatService has easy access to the Mappers as it installs the web server. But this could be moved elsewhere if appropriate.

       

      Registration and activation of on-demand profiles

       

      This logic is also included in the attached tomcat-module.patch.

       

      Basically I cut and pasted this from the code that we've been using for a year to handle deployment of the content in clustering's deploy-hasingleton directory. On startup, an MC bean registers a Profile with the ProfileService but doesn't activate it. Later, when the bean gets a signal it activates the profile. In the deploy-hasingleton case, the signal is a change in the cluster topology causing the server to become the "master". Here the signal is the notification from JBoss Web that the context has been requested.

       

      I did a cut-and-paste job here, but this logic should be factored out into re-usable code in one of the ProfileService libraries.

       

      Usage

       

      First, I create a common/deploy folder.

      Next, I move each war in server/default/deploy to common/deploy, replacing it with an XXX-activator-jboss-beans.xml file. The wars are admin-console.war, jmx-console.war, jbossws-console.war, invoker.war and ROOT.war.

       

      A typical XXX-activator-jboss-beans.xml looks like this:

       

      <?xml version="1.0" encoding="UTF-8"?>
      <deployment xmlns="urn:jboss:bean-deployer:2.0">
      
          <bean name="AdminConsoleActivator" class="org.jboss.web.tomcat.service.ondemand.OnDemandContextProfileManager">
            
            <property name="profileService"><inject bean="ProfileService"/></property>
            <property name="profileFactory"><inject bean="ProfileFactory"/></property>
            <property name="onDemandContextIntegrator"><inject bean="WebServer"/></property>
      
            <property name="contextName">admin-console</property>
      
            <!-- Build a profile from the contents of this single URI -->
            <property name="singleURI">${jboss.home.url}common${/}deploy${/}admin-console.war</property>
      
         </bean>
          
      </deployment>
      

       

      The MC bean exposes some other properties in case you want to do something more sophisticated:

       

      <?xml version="1.0" encoding="UTF-8"?>
      <deployment xmlns="urn:jboss:bean-deployer:2.0">
      
          <bean name="JmxConsoleActivator" class="org.jboss.web.tomcat.service.ondemand.OnDemandContextProfileManager">
            
            <property name="profileService"><inject bean="ProfileService"/></property>
            <property name="profileFactory"><inject bean="ProfileFactory"/></property>
            <property name="onDemandContextIntegrator"><inject bean="WebServer"/></property>
      
            <property name="profileDomain">default</property>
            <property name="profileServer">default</property>
            <property name="profileName">jmx-console</property>
      
            <property name="serviceName">jboss.web</property>
            <property name="hostName">localhost</property>
            <property name="contextName">jmx-console</property>
      
            <!-- Build a profile from the contents of these URIs -->
            <property name="URIList">
               <list elementClass="java.net.URI">
                  <value>${jboss.home.url}common${/}deploy${/}jmx-console.war</value>
               </list>
            </property>
         </bean>
          
      </deployment>
      

       

      The example above doesn't actually do anything sophisticated; it shows the properties with their default values.

       

      Results

       

      Using a build of the current AS trunk, with the 'default' configuration modified as described above, on my workstation 'default' starts in 20.5 seconds. Unmodified, i.e. with the wars deploying as part of startup, it takes 38.6 seconds, so an 18.1 second reduction in startup time. Most of that is from deferring start of the admin-console, but a couple weeks back before the admin-console was restored to trunk I was seeing > 25 seconds boot time on this machine, so it seems moving out the other wars is also saving some time.

       

      Accessing all the on-demand wars with a browser, I saw no noticeable delays on the first request to any of them, except for the admin-console. There there is a 10-15 second delay before the server responds with it's "Redirecting to login page" screen.

       

      Issues

       

      I see a few issues with the above:

       

      1. Need for XXX-activator-jboss-beans.xml. It would be slick to somehow configure the war such that some deployer extracts relevant metadata, detects an "on-demand configuration, wires things up with JBoss Web, and then stops the deployment before anything expensive starts, with deployment continuing when the app is requested. Nice, yes, but much harder. For now I think we should stick to the KISS principle. The "profile activation" approach used here has been in use in the AS for well over a year.
      2. Integration with mod_cluster. This is a problem, as mod_cluster will know nothing about the "on-demand" context until the war is actually deployed. So, the load balancer will not know to route requests to the server. This needs fixing, perhaps with a hook into mod_cluster to allow the MC bean to tell it about the on-demand context. I don't think this is a critical problem until AS 6.0.0.CR1 though. We already configure mod_cluster to ignore all the standard apps we deploy; users have to configure ModClusterService to expose them via the load balancer. IMHO until CR1 it's OK to force users who want to expose a standard app to move it back into the deploy/ dir.
      3. Virtual host aliases and multiple contexts associated with the same application. The attached prototype doesn't deal with this, but that shouldn't be anything technically difficult to implement.
        • 1. Re: Deployment of on-demand web applications
          brian.stansberry
          Didn't say this above: I have no intention of committing any of those without thorough discussion, plus tests.
          • 2. Re: Deployment of on-demand web applications
            rmaucher

            I am not surprised with the method. As stated in my comments for the relevant bug reports, I dislike it.

             

            - These mechanisms are strongly detrimental in production (massive resource use while user processing is going on, unpredictability of deployment is hidden from the administrator).

            - Introduction of volatile ints. I read a performance evaluation where they are more comparable to syncing rather than a simple int. For this usage, I will not use any (production should never lazy load in the first place). The "replaced" flag looks suspicious to me.

            - This seems to be done for all webapps. This should always be an opt in mechnism, only enabled for select webapps (with the only one being the admin webapp).

            - What do you do with requests piling up while you deploy the actual webapp ? [personally, I don't think they should be waiting]

             

            So you are trying to help the developer here. Please never consider adding these mechanisms to a production oriented server.

             

            Overall, I think the root of the problem is that there's a belief the default profile must ship with such large webapps *and* be run on the slow overpriced macs everyone has bought ...

            • 3. Re: Deployment of on-demand web applications
              ssilvert

              As a compromise, perhaps it's time to revisit the profiles we are shipping.  We've talked about having seperate production and development profiles before.  Given the discussion above, this is starting to make more sense than ever.  Right now we have:

               

              minimal - total bare bones

              default - what most people use, but you have to do stuff like secure the consoel before production

              standard - as far as I know, this is only for JEE certification?

              web - just web stuff with JTA, JCA and JPA

              all - everything we can throw in, but mostly used for clustering

               

              So what if we did something like this?

               

              minimal - same as before

              web - JEE web profile

              standard - I guess we still need this?

              development - no admin console and less logging - super-fast boot time - (JSF2 PROJECT_STAGE set to "Development")

              production - includes admin console - secure - (JSF2 PROJECT_STAGE set to "Production")

              all renamed to cluster (this makes it more obvious what it is really for) - same as production, but with clustered services available


              With this, there would be no need for lazy loading.

               

              I bring up the JSF2 PROJECT_STAGE setting because it demonstrates that other projects are going to the Development-Mode/Production-Mode model.  We've talked about this before.  It's time to get in front of the trend.

               

              Stan

              • 4. Re: Deployment of on-demand web applications
                rmaucher
                Setting the admin webapp (and the invoker and jmx-config) as on demand should be fine, but the other webapps should not be on demand in the defaut config (most importantly including any webapp deployed by the user). And that's it, not need to change much. So someone who actually runs that in production would be relatively ok assuming he does not use the admin webapp, it would behave as the old AS releases. If the user actually reads the docs, he can set any webapp as on demand.
                • 5. Re: Deployment of on-demand web applications
                  rmaucher
                  I have committed the web code, since it does not do anything if not using on demand. This should allow working on the rest more easily when the next web build is integrated this week.
                  • 6. Re: Deployment of on-demand web applications
                    emuckenhuber

                    {quote}
                    I did a cut-and-paste job here, but this logic should be factored out
                    into re-usable code in one of the ProfileService libraries.
                    {quote}

                     

                    Is this supposted to be the way the admin-console will be integrated into AS in future then?

                    The plan is to move deployments out of deploy/ - so everything should be described in it's own profile. But ProfileService is not really aware of services activating profiles.
                    There will be something like a on-demand feature, which basically only registers a profile without activating it. However if someone actually decides to include the 'admin-console' profile in his configuraiton, it will get activated. So this on-demand context activator would also have to support that this profile is getting activated when starting AS (the ordering can be controlled if neccessary). I guess that should be no problem?

                    • 7. Re: Deployment of on-demand web applications
                      brian.stansberry

                      remy.maucherat@jboss.com wrote:

                       

                      Setting the admin webapp (and the invoker and jmx-config) as on demand should be fine, but the other webapps should not be on demand in the defaut config (most importantly including any webapp deployed by the user). And that's it, not need to change much.

                       

                      For sure, end-user apps shouldn't end up deployed as on-demand (unless they go out of their way to trigger that.)

                       

                      And yup, that means including ROOT.war in the on-demand list is a mistake.

                      • 8. Re: Deployment of on-demand web applications
                        bill.burke

                        bstansberry@jboss.com wrote:

                         

                        An ugly solution is to not include these wars in our default profile; users would have to add them if they wanted them. Not very user-friendly.

                         

                        I think this whole thing is a waste of time and that you should focus on other boot-time performance issues.  Why?  I think a "development profile" should be created that does not include any of these "management" wars anyways.  Embedded unit testing should be the main focus of this profile because if we develop that, this will be a big win for our user base.  In development, how often are these .WAR files used anyways?  Never, if we do our embedded stuff correctly.

                         

                        I think Remy and company should focus on ripping out all the catalina and tomcat bullshit and focus on optimizing deployment as well as nicer integration with the MC/VDF and JBoss AS as a whole.  JBossWeb needs some serious refactoring in this regards, because as I look at it now, it is just a series of hacks and ugly ass code.

                        • 9. Re: Deployment of on-demand web applications
                          brian.stansberry

                          Re: the general approach to slimming -- on-demand vs. just keeping stuff out of profiles...

                           

                          That's really the core discussion. I like the on-demand approach, and felt like actually coding after 1.5 months of mostly discussing, so I went for it. But if another general solution is better, that's what we should do.

                           

                          Stan, the thing I don't like about the multiple config approach is it somewhat mixes apples and oranges -- whether I want something included in the initial boot of the server vs whether I want it at all are two slightly different things. For example,

                           

                          1) I'm developing with the "cluster" profile. Now I have to wait 18 seconds on every boot for the admin-console. Ugh. (Believe me, I'm very conscious of the added boot time from the current "all profile. )

                           

                          2) I'm developing with the "development" profile. But now I want to look at the jmx-console or admin-console. I have to switch profiles or create a custom profile to get that. Ugh.

                           

                          Rémy, I 100% agree that on-demand on a production server is a terrible idea. So if the basic approach to making on-demand work is sound, a next step is to figure out a simple mechanism to allow the on-demand applications to be deployed immediately. The "activator" bean I describe above could easily be enhanced to add some logic to check an overall server state, and actually activate the profile at server start. For example,

                           

                          public enum ServerBootMode {
                             FAST, SOME_MIDDLING_TERM, FULL;
                          }
                          

                           

                          The "activator" bean then has a config property added

                           

                          <property name="onDemandBootMode">FAST</property>
                          

                           

                          In start() it checks the server's boot mode against its "onDemandBootMode" and if the server is greater, it activates the profile immediately, not as on-demand.

                           

                          A startup switch could control the server-wide boot mode. For certain profiles, we could add a bean that sets the boot mode to an appropriate default if the user hadn't done it via a startup switch. Production sets it to FULL, "development" to FAST etc.

                          • 10. Re: Deployment of on-demand web applications
                            bill.burke

                            bstansberry@jboss.com wrote:

                            2) I'm developing with the "development" profile. But now I want to look at the jmx-console or admin-console. I have to switch profiles or create a custom profile to get that. Ugh.


                            IMO, this will almost never happen.  I think you should focus on other optimizations and come back to this.

                            • 11. Re: Deployment of on-demand web applications
                              rmaucher

                              VDF integration is done for all metadata  (tlds, web.xml, web-fragment.xml). Configuration (server.xml and context.xml) reimplementation is waiting on prfile service. Cleanup has been done to remove a hard JMX dependency with the mapper. Integration can then be moved over gradually to the embedded API.

                               

                              The "ugly ass" standalone code will remain as a casual testing capability (Jean-Frédéric loves it). Sorry you dislike it, but it is mature and it works quite well, so I don't really have plans to delete it for the time being.

                              • 12. Re: Deployment of on-demand web applications
                                ssilvert

                                stansberry@jboss.com wrote:

                                 

                                Re: the general approach to slimming -- on-demand vs. just keeping stuff out of profiles...

                                 

                                That's really the core discussion. I like the on-demand approach, and felt like actually coding after 1.5 months of mostly discussing, so I went for it. But if another general solution is better, that's what we should do.

                                 

                                Stan, the thing I don't like about the multiple config approach is it somewhat mixes apples and oranges -- whether I want something included in the initial boot of the server vs whether I want it at all are two slightly different things.

                                That's true, they are slightly different.  But let's don't make the perfect the enemy of the good.  I have to agree with Bill on this one.  Doing on-demand adds complexity to the system and uses up valuable development time.

                                For example,

                                 

                                1) I'm developing with the "cluster" profile. Now I have to wait 18 seconds on every boot for the admin-console. Ugh. (Believe me, I'm very conscious of the added boot time from the current "all profile. )

                                 

                                2) I'm developing with the "development" profile. But now I want to look at the jmx-console or admin-console. I have to switch profiles or create a custom profile to get that. Ugh.

                                To solve #1, we would just need "clustered development" and "clustered produciton" profiles.

                                 

                                For #2, I don't think developers typically use the admin console.  (Maybe I'm wrong?)  JMX console should be enough for development and testing.

                                 

                                We'll never please everyone with our pre-defined profiles.  But we can make profiles that please most users most of the time.


                                My other point is that we need to do the development and production profiles anyway.  Other projects are going to that model and we'll be left behind if we don't address the development vs. production issue head on.  So why not kill two birds with one stone and move on to other things?
                                • 13. Re: Deployment of on-demand web applications
                                  rmaucher
                                  I think the point is to be able to add the admin console as default and not look too bad on the "great" benchmarks that are popping up on blogs, where the person simply tests the time it takes to execute "./run.sh" without even looking at what it is doing. So I think this is a useful feature, but we should only use it on a set of known deployments (which is everything except ROOT.war in our current default config, I agree with Brian on that one).
                                  • 14. Re: Deployment of on-demand web applications
                                    bill.burke

                                    remy.maucherat@jboss.com wrote:

                                     

                                    VDF integration is done for all metadata  (tlds, web.xml, web-fragment.xml). Configuration (server.xml and context.xml) reimplementation is waiting on prfile service. Cleanup has been done to remove a hard JMX dependency with the mapper. Integration can then be moved over gradually to the embedded API.

                                     

                                    The "ugly ass" standalone code will remain as a casual testing capability (Jean-Frédéric loves it). Sorry you dislike it, but it is mature and it works quite well, so I don't really have plans to delete it for the time being.

                                    I want deeper integration.  For example, it would make my life so much easier if I could instantiate instances of servlets and filters within my JAX-RS deployer for JBossWeb to consume and deploy.  But I guess thats another different discussion...

                                     

                                    Yeah, the "ugly ass" code I meant was all the JMX stuff.

                                    1 2 3 Previous Next