-
15. Re: Parallel deployments
"kabir.khan@jboss.com" wrote:
and at the end of the bootstrap do something like://Main bootstrap code... if (controller instanceof AsynchController){ while (controller.processingAsynchronousContexts()){ Thread.sleep(100); } } //Report uninstalled contexts
Yuck! :-)
Simply don't include those asynch beans in the check if they are not done yet.
Or print an info about them, that they are still in progress.
If they fail to be fully installed, we will get that info later on,
with the next install / deployment.
I guess this will (also) need to be addressed in MainDeployer::checkComplete.
I'll check this once I update Deployers to use your new Kernel asynch code. -
16. Re: Parallel deployments
"smarlow@redhat.com" wrote:
Could you share your AS changes (if you don't check them in)? I would also like to experiment with using parallel deployment in AS.
-Scott
I currently have a bean for local testing in the server/ module:package org.jboss.executor; import java.util.concurrent.Executor; import org.jboss.dependency.plugins.AbstractController; import org.jboss.dependency.spi.ControllerContext; /** * Bean used to install an Executor in the controller * * @author <a href="kabir.khan@jboss.com">Kabir Khan</a> * @version $Revision: 1.1 $ */ public class ControllerExecutorInstaller { ControllerContext context; Executor executor; public ControllerContext getContext() { return context; } public void setContext(ControllerContext context) { this.context = context; } public Executor getExecutor() { return executor; } public void setExecutor(Executor executor) { this.executor = executor; } public void start() { ((AbstractController)context.getController()).setExecutor(executor); } public void stop() { ((AbstractController)context.getController()).setExecutor(null); } }
configured via conf/bootstrap/asynch.xml (added to bootstrap.xml)<?xml version="1.0" encoding="UTF-8"?> <!-- The Executor used by the controller to do asynchronous deployments --> <deployment xmlns="urn:jboss:bean-deployer:2.0"> <classloader><inject bean="asynch-classloader:0.0.0"/></classloader> <classloader name="asynch-classloader" xmlns="urn:jboss:classloader:1.0" export-all="NON_EMPTY" import-all="true"> <root>${jboss.common.lib.url}jboss.jar</root> </classloader> <bean name="ControllerExecutorInstaller" class="org.jboss.executor.ControllerExecutorInstaller"> <property name="context"><inject fromContext="context"/></property> <property name="executor"><inject name="ControllerExecutor"/></property> </bean> <bean name="ControllerExecutor" class="java.util.concurrent.Executor"> <constructor factoryClass="java.util.concurrent.Executors" factoryMethod="newFixedThreadPool"> <parameter>10</parameter> </constructor> </bean> </deployment>
However, it is not working properly in AS yet, I get the following errors for the asynchronous beans13:13:29,126 ERROR [AbstractKernelController] Error installing to Create: name=CacheManager state=Create mode=Asynchronous requiredState=Installed java.lang.Exception: Error calling callback JMXCreateDestroyAdvice for target context CacheManager at org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.install(AbstractLifecycleCallbackItem.java:91) at org.jboss.dependency.plugins.AbstractController.handleLifecycleCallbacks(AbstractController.java:1687) at org.jboss.dependency.plugins.AbstractController.handleInstallLifecycleCallbacks(AbstractController.java:1652) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:961) at org.jboss.dependency.plugins.AbstractController.executeOrIncrementStateDirectly(AbstractController.java:1187) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1105) at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1002) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:792) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:557) at org.jboss.system.ServiceController.doInstall(ServiceController.java:653) at org.jboss.system.ServiceController.register(ServiceController.java:356) at org.jboss.system.microcontainer.jmx.ServiceControllerRegistrationLifecycleCallback.install(ServiceControllerRegistrationLifecycleCallback.java:104) at sun.reflect.GeneratedMethodAccessor243.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:59) at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:150) at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66) at org.jboss.beans.info.plugins.AbstractBeanInfo.invoke(AbstractBeanInfo.java:300) at org.jboss.kernel.plugins.dependency.AbstractKernelControllerContext.invoke(AbstractKernelControllerContext.java:291) at org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.install(AbstractLifecycleCallbackItem.java:87) at org.jboss.dependency.plugins.AbstractController.handleLifecycleCallbacks(AbstractController.java:1687) at org.jboss.dependency.plugins.AbstractController.handleInstallLifecycleCallbacks(AbstractController.java:1652) at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:961) at org.jboss.dependency.plugins.AbstractController$InstallControllerContextTask.installMyContext(AbstractController.java:2240) at org.jboss.dependency.plugins.AbstractController$InstallControllerContextTask.run(AbstractController.java:2185) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676) at java.lang.Thread.run(Thread.java:613) Caused by: java.lang.IllegalStateException: jboss.cache:service=CacheManager is already installed. at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:733) at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:557) at org.jboss.system.ServiceController.doInstall(ServiceController.java:653) at org.jboss.system.ServiceController.register(ServiceController.java:356) at org.jboss.system.ServiceController.register(ServiceController.java:325) at org.jboss.system.ServiceController.register(ServiceController.java:312) at org.jboss.system.ServiceController.create(ServiceController.java:387) at org.jboss.system.ServiceController.create(ServiceController.java:370) at org.jboss.system.microcontainer.jmx.ServiceControllerCreateDestroyLifecycleCallback.install(ServiceControllerCreateDestroyLifecycleCallback.java:44) at sun.reflect.GeneratedMethodAccessor251.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:59) at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:150) at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66) at org.jboss.beans.info.plugins.AbstractBeanInfo.invoke(AbstractBeanInfo.java:300) at org.jboss.kernel.plugins.dependency.AbstractKernelControllerContext.invoke(AbstractKernelControllerContext.java:291) at org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.install(AbstractLifecycleCallbackItem.java:87) ... 28 more
This error does not happen if I remove 'mode="Asynchronous"' from the beans. I'll add some more tests to core kernel to see if lifecycle callbacks are handled properly for asynchronous beans. -
17. Re: Parallel deployments
Scott,
I have fixed the problem I mentioned, so if you want to try it out check out kernel trunk and update as lib with jboss-dependency.jar and jboss-kernel.jar. I'll post my findings here once I have some. -
18. Re: Parallel deployments
There was some talk on #jboss-dev about using an executor in the controller during bootstrap and then to avoid having too many thread pools configured to replace that with the system thread pool once that is installed.
Assuming that the system thread pool in AS trunk is the one deployed in thread-pool-jboss-beans.xml:<!-- Basic system thread pool deployment --> <deployment xmlns="urn:jboss:bean-deployer:2.0"> <threads xmlns="urn:jboss:threads:1.0"> <thread-group name="SystemThreadFactoryGroup" group-name="System Threads"/> <thread-factory name="SystemThreadFactory" group="SystemThreadFactoryGroup"/> <!-- ~ This thread pool is for SHORT-RUNNING tasks that block very little or not at all. Long-running ~ tasks submitted to this pool may cause starvation and extended blocking. --> <thread-pool-executor name="ThreadPool" thread-factory="SystemThreadFactory" queue-length="1000"> <core-pool-size count="5" per-cpu="2"/> <max-pool-size count="10" per-cpu="3"/> <keepalive-time time="30" unit="seconds"/> <reject-policy name="block"/> </thread-pool-executor> </threads> </deployment>
I don't think this pool will work for the controller if my understanding of<reject-policy name="block"/>
is correct (that the caller blocks until a thread is available). The behaviour of the controller is to try to deploy asynchronous contexts via the executor, but if the executor is full to deploy the context in the calling thread. So I need a reject policy that behaves like ThreadPoolExecutor.AbortPolicy or ThreadPoolExecutor.CallerRunsPolicy for the pool handling asynchronous deployments.
So we have the choice of reusing the system thread pool but potentially blocking on asynchronous contexts (which I think kind of defeats the purpose), or having a small thread pool dedicated to deploying asynchronous contexts. -
19. Re: Parallel deployments
A design change I'd like to make to jboss-threads for AS 6 is the ability to separate the thread pool from the executor config, so that two executors can have different rejection policies for the same underlying thread pool. Whether or not it will be possible remains to be seen though.
-
20. Re: Parallel deployments
From a performance perspective, there actually isn't a lot of difference between blocking when the queue is full vs. executing directly - it's the difference between N possible executing threads and N-1 possible executing threads, if you think about it.
-
21. Re: Parallel deployments
If we are worried about not having enough threads to start the server, couldn't we require that the pool be configured with enough threads to boot the server? This could be via a comment in the xml file.
A related concern is how we will deal with several deployment threads trying to synchronize on the same object locks. I assume this is were additional tuning will come into play (reducing object lock contention through code modifications). One possible solution, is that we could manually designate which thread (e.g. "cluster partition init") is to be used on a per component basis. Before diving into solving this potential issue, we need to evaluate how expensive object lock contention will be (to avoid premature optimizing). But, it doesn't hurt to consider that we might need to do something in this area. If we were to asynchronously deploy in named threads, that might require a separate thread pool.
Another concern is redeployment in a production app server deployment. We need to be clear in the thread pool xml file (via comments) as to what JBoss elements use the system thread pools (there is a long running task pool and short running task pool). If a customer sizes the pools only on the application needs, then redeployment on a loaded system could run short of threads needed to redeploy. Even if the pool is increased to accommodate the number of threads needed for deployment, applications can easily consume those as well in a loaded system. I'm wondering if we need to introduce throttling between the users of the system thread pool, as a way to carve up the thread pool between different types of tasks. Logically, this would create sub-pools of threads, where perhaps there could be borrowing of the threads between sub-pools. I'm not sure if this is over-engineering, what could possibly be handled via configuration and operational considerations (only hot-deploy during idle time). -
22. Re: Parallel deployments
"david.lloyd@jboss.com" wrote:
From a performance perspective, there actually isn't a lot of difference between blocking when the queue is full vs. executing directly - it's the difference between N possible executing threads and N-1 possible executing threads, if you think about it.
Hmm, maybe you're right. I'll implement some underneath swapping of the thread pool.
I have marked the HAPartition and CacheManager beans as asynchronous, and am seeing some improvements.
Original
---------
36.631
33.447
33.21
32.914
32.840
33.638
33.78
33.157
Asynchronous
---------------
30.432
31.166
31.13
31.68
30.788
31.467
30.455
30.459
I need to talk a bit more with Brian about how to test this with actual deployments using the caches, since the caches are not started until actually used. -
23. Re: Parallel deployments
"kabir.khan@jboss.com" wrote:
Hmm, maybe you're right. I'll implement some underneath swapping of the thread pool.
This is done https://jira.jboss.org/jira/browse/JBAS-7083.
To get the actual asynch behaviour though you will need to upgrade to an mc snapshot -
24. Re: Parallel deployments
I have marked the jboss.messaging:service=PostOffice mbean as asynchronous as well, since that is taking a very long time to start up. The cool thing is that now the core server takes 23 seconds to start up (down from 33s), but we are getting the unsatisfied dependencies I was expecting in a previous post. I'll try to fix that as Ales suggested, but does it look weird that the server keeps installing stuff after it has "started"? If user applications have dependencies on jms, they cannot be started until the background stuff has installed.
18:54:07,391 INFO [STDOUT] --------------------------------------------------------- GMS: address is 127.0.0.1:50224 (cluster=MessagingPostOffice-CTRL) --------------------------------------------------------- ... 18:54:07,568 INFO [ProfileServiceBootstrap] Loading profile: ProfileKey@ece555[domain=default, server=default, name=all] 18:54:07,572 ERROR [ProfileServiceBootstrap] Failed to load profile: Summary of incomplete deployments (SEE PREVIOUS ERRORS FOR DETAILS): DEPLOYMENTS MISSING DEPENDENCIES: Deployment "jboss.messaging.connectionfactory:service=ClusterPullConnectionFactory" is missing the following dependencies: Dependency "jboss.messaging:service=PostOffice" (should be in state "Start", but is actually in state "Create") Deployment "jboss.messaging.connectionfactory:service=ClusteredConnectionFactory" is missing the following dependencies: Dependency "jboss.messaging:service=PostOffice" (should be in state "Start", but is actually in state "Create") Deployment "jboss.messaging.connectionfactory:service=ConnectionFactory" is missing the following dependencies: Dependency "jboss.messaging:service=PostOffice" (should be in state "Start", but is actually in state "Create") Deployment "jboss.messaging.destination:name=DLQ,service=Queue" is missing the following dependencies: Dependency "jboss.messaging:service=PostOffice" (should be in state "Start", but is actually in state "Create") Deployment "jboss.messaging.destination:name=ExpiryQueue,service=Queue" is missing the following dependencies: Dependency "jboss.messaging:service=PostOffice" (should be in state "Start", but is actually in state "Create") Deployment "jboss.messaging:service=PostOffice" is missing the following dependencies: 18:54:07,580 INFO [Http11Protocol] Starting Coyote HTTP/1.1 on http-127.0.0.1-8080 18:54:07,594 INFO [AjpProtocol] Starting Coyote AJP/1.3 on ajp-127.0.0.1-8009 18:54:07,599 INFO [ServerImpl] JBoss (Microcontainer) [6.0.0-SNAPSHOT] Started in 22s:959ms 18:54:09,394 INFO [GroupMember] org.jboss.messaging.core.impl.postoffice.GroupMember$ControlMembershipListener@f3a92 got new view [127.0.0.1:50224|0] [127.0.0.1:50224], old view is null 18:54:09,395 INFO [GroupMember] I am (127.0.0.1:50224) 18:54:09,395 INFO [GroupMember] New Members : 1 ([127.0.0.1:50224]) 18:54:09,395 INFO [GroupMember] All Members : 1 ([127.0.0.1:50224]) 18:54:09,406 INFO [STDOUT] --------------------------------------------------------- GMS: address is 127.0.0.1:7900 (cluster=MessagingPostOffice-DATA) --------------------------------------------------------- 18:54:14,438 INFO [QueueService] Queue[/queue/DLQ] started, fullSize=200000, pageSize=2000, downCacheSize=2000 18:54:14,495 INFO [ConnectionFactory] Connector bisocket://127.0.0.1:4457 has leasing enabled, lease period 10000 milliseconds 18:54:14,495 INFO [ConnectionFactory] org.jboss.jms.server.connectionfactory.ConnectionFactory@5d24e1 started 18:54:14,497 INFO [ConnectionFactory] Connector bisocket://127.0.0.1:4457 has leasing enabled, lease period 10000 milliseconds 18:54:14,497 INFO [ConnectionFactory] org.jboss.jms.server.connectionfactory.ConnectionFactory@a285d0 started 18:54:14,704 WARN [JBossASSecurityMetadataStore] WARNING! POTENTIAL SECURITY RISK. It has been detected that the MessageSucker component which sucks messages from one node to another has not had its password changed from the installation default. Please see the JBoss Messaging user guide for instructions on how to do this. 18:54:15,538 INFO [ConnectionFactory] Connector bisocket://127.0.0.1:4457 has leasing enabled, lease period 10000 milliseconds 18:54:15,538 INFO [ConnectionFactory] org.jboss.jms.server.connectionfactory.ConnectionFactory@d4ca03 started 18:54:15,540 INFO [QueueService] Queue[/queue/ExpiryQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
The log message at 18:54:07 is output after installing the first of two channels by the post office service, so it is taking a long time. I'm not familiar with JBoss Messaging, but maybe those two channels can be installed in parallel as Brian did for the HAPartition bean, if JBoss Messaging is not doing so already. As I understand it, we're due a new release of JBoss Messaging in AS trunk sometime? -
26. Re: Parallel deployments
If installing 2 beans
-Bean depends on Dependency
-Dependency as Asynchronous
And then checking if the deployment is complete while Dependency is installed in a background thread, we now get the following error message:Summary of incomplete deployments (SEE PREVIOUS ERRORS FOR DETAILS): DEPLOYMENTS MISSING DEPENDENCIES: Deployment "Bean" is missing the following dependencies: Dependency "Dependency" (is currently being installed in a background thread)
-
27. Re: Parallel deployments
"kabir.khan@jboss.com" wrote:
If installing 2 beans
-Bean depends on Dependency
-Dependency as Asynchronous
And then checking if the deployment is complete while Dependency is installed in a background thread, we now get the following error message:Summary of incomplete deployments (SEE PREVIOUS ERRORS FOR DETAILS): DEPLOYMENTS MISSING DEPENDENCIES: Deployment "Bean" is missing the following dependencies: Dependency "Dependency" (is currently being installed in a background thread)
Done for https://jira.jboss.org/jira/browse/JBDEPLOY-204 -
28. Re: Parallel deployments
When looking at http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4243667 I realised that I have not tested/handled the following scenario:
*Install ctx A asynchronously
*Uninstall ctx A while is is being installed asynchronously.
I will add support for this -
29. Re: Parallel deployments
Kabir,
I understand that you are still working on parallel deployment but wanted to ask a few questions. Are you still seeing a ten second improvement on your machine?
When do you think the parallel deployment support will show up in AS trunk?
Thanks,
Scott
