Deployment of on-demand web applications
I've been looking into how to deploy web application's on-demand; i.e. when the first user request that maps to the application comes in.
One of the problems we need to solve to get the boot time of the AS down to a low level is the time it takes to deploy the various web applications we ship by default, particularly the admin console, which is quite heavy. We want to get the deployment time of these war's as low as possible, but some are almost surely always going to take a couple seconds or more to deploy, which is a large fraction of the overall boot time we'd like to see.
An ugly solution is to not include these wars in our default profile; users would have to add them if they wanted them. Not very user-friendly.
An alternate solution is to configure the web server to recognize that certain URLs target an as-yet-undeployed application, and have it call out to the ProfileService to trigger deployment of the web application. Turns out this is fairly straightforward to do. It requires:
- Fairly simple modification to the JBoss Web logic that maps requests to applications.
- Integration layer between JBoss Web and ProfileService
- Registration and activation of on-demand profiles
I'll explain each in more depth
JBoss Web Request Mapping Changes
I've attached a jboss-web.patch that shows the proposed changes.
JBoss Web / Tomcat has isolated the logic for mapping a request to the container it targets into a single class: org.apache.tomcat.util.http.mapper.Mapper. The attached patch modifies that class as follows:
- Adds a new addOnDemandContext(String hostName, String path) method to allow registration of as-yet-undeployed web applications.
- Adds a notification listener interface OnDemandContextMappingListener to Mapper's package.
- Modifies the mapping algorithm such that when a request for an on-demand application comes in, any registered listener is notified. (The listener would presumably deploy the on-demand application, although the logic handles the case where this does not happen.) If the requested container is already deployed this logic adds a single boolean check and a single null check to the critical path, so it's quite lightweight. (The null check could even be removed by adding a return statement.)
- Modifies the logic for when regular (i.e. non-on-demand) applications are registered so it handles the case of the regular context replacing the on-demand context.
The patch doesn't include any unti tests, but the Mapper class is quite amenable to solid unit testing.
Integration layer between JBoss Web and ProfileService
I've attached a tomcat-module.patch that shows the proposed changes.
Basically, this involves implementing the OnDemandContextMappingListener interface and adding hooks to access the JBoss Web Mapper objects so on-demand contexts can be registered and the listener can be registered.
The patch has the TomcatService class do the wiring into JBoss Web, which was the easy approach for this prototype since TomcatService has easy access to the Mappers as it installs the web server. But this could be moved elsewhere if appropriate.
Registration and activation of on-demand profiles
This logic is also included in the attached tomcat-module.patch.
Basically I cut and pasted this from the code that we've been using for a year to handle deployment of the content in clustering's deploy-hasingleton directory. On startup, an MC bean registers a Profile with the ProfileService but doesn't activate it. Later, when the bean gets a signal it activates the profile. In the deploy-hasingleton case, the signal is a change in the cluster topology causing the server to become the "master". Here the signal is the notification from JBoss Web that the context has been requested.
I did a cut-and-paste job here, but this logic should be factored out into re-usable code in one of the ProfileService libraries.
Usage
First, I create a common/deploy folder.
Next, I move each war in server/default/deploy to common/deploy, replacing it with an XXX-activator-jboss-beans.xml file. The wars are admin-console.war, jmx-console.war, jbossws-console.war, invoker.war and ROOT.war.
A typical XXX-activator-jboss-beans.xml looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<deployment xmlns="urn:jboss:bean-deployer:2.0">
<bean name="AdminConsoleActivator" class="org.jboss.web.tomcat.service.ondemand.OnDemandContextProfileManager">
<property name="profileService"><inject bean="ProfileService"/></property>
<property name="profileFactory"><inject bean="ProfileFactory"/></property>
<property name="onDemandContextIntegrator"><inject bean="WebServer"/></property>
<property name="contextName">admin-console</property>
<!-- Build a profile from the contents of this single URI -->
<property name="singleURI">${jboss.home.url}common${/}deploy${/}admin-console.war</property>
</bean>
</deployment>
The MC bean exposes some other properties in case you want to do something more sophisticated:
<?xml version="1.0" encoding="UTF-8"?>
<deployment xmlns="urn:jboss:bean-deployer:2.0">
<bean name="JmxConsoleActivator" class="org.jboss.web.tomcat.service.ondemand.OnDemandContextProfileManager">
<property name="profileService"><inject bean="ProfileService"/></property>
<property name="profileFactory"><inject bean="ProfileFactory"/></property>
<property name="onDemandContextIntegrator"><inject bean="WebServer"/></property>
<property name="profileDomain">default</property>
<property name="profileServer">default</property>
<property name="profileName">jmx-console</property>
<property name="serviceName">jboss.web</property>
<property name="hostName">localhost</property>
<property name="contextName">jmx-console</property>
<!-- Build a profile from the contents of these URIs -->
<property name="URIList">
<list elementClass="java.net.URI">
<value>${jboss.home.url}common${/}deploy${/}jmx-console.war</value>
</list>
</property>
</bean>
</deployment>
The example above doesn't actually do anything sophisticated; it shows the properties with their default values.
Results
Using a build of the current AS trunk, with the 'default' configuration modified as described above, on my workstation 'default' starts in 20.5 seconds. Unmodified, i.e. with the wars deploying as part of startup, it takes 38.6 seconds, so an 18.1 second reduction in startup time. Most of that is from deferring start of the admin-console, but a couple weeks back before the admin-console was restored to trunk I was seeing > 25 seconds boot time on this machine, so it seems moving out the other wars is also saving some time.
Accessing all the on-demand wars with a browser, I saw no noticeable delays on the first request to any of them, except for the admin-console. There there is a 10-15 second delay before the server responds with it's "Redirecting to login page" screen.
Issues
I see a few issues with the above:
- Need for XXX-activator-jboss-beans.xml. It would be slick to somehow configure the war such that some deployer extracts relevant metadata, detects an "on-demand configuration, wires things up with JBoss Web, and then stops the deployment before anything expensive starts, with deployment continuing when the app is requested. Nice, yes, but much harder. For now I think we should stick to the KISS principle. The "profile activation" approach used here has been in use in the AS for well over a year.
- Integration with mod_cluster. This is a problem, as mod_cluster will know nothing about the "on-demand" context until the war is actually deployed. So, the load balancer will not know to route requests to the server. This needs fixing, perhaps with a hook into mod_cluster to allow the MC bean to tell it about the on-demand context. I don't think this is a critical problem until AS 6.0.0.CR1 though. We already configure mod_cluster to ignore all the standard apps we deploy; users have to configure ModClusterService to expose them via the load balancer. IMHO until CR1 it's OK to force users who want to expose a standard app to move it back into the deploy/ dir.
- Virtual host aliases and multiple contexts associated with the same application. The attached prototype doesn't deal with this, but that shouldn't be anything technically difficult to implement.
-
jboss-web.patch.zip 3.2 KB
-
tomcat-module.patch.zip 7.1 KB
