Concurrent deployments

[belaban]

The MC's dependency management currently starts all services sequentially, according to their dependencies.

In JBoss 4.3, we have 4 JGroups channels, which take 5 secs *each* to start. However, they are completely independent from each other, so they could be started concurrently. This would bring the time needed down from 20 secs to 5 secs.

I imagine we could do this for all independent services, a.k.a subtrees in the dependency tree. This would speed up start of JBoss significantly !

Can we do this ?

Cheers

[adrian.brock]

The MC's dependency management currently starts all services sequentially, according to their dependencies.

Actually some services, e.g. MDBs are started asynchronously,
but this is handled by the relevant container rather than the Micro-kernel/container.

In JBoss 4.3, we have 4 JGroups channels, which take 5 secs *each* to start. However, they are completely independent from each other, so they could be started concurrently. This would bring the time needed down from 20 secs to 5 secs.

I imagine we could do this for all independent services, a.k.a subtrees in the dependency tree. This would speed up start of JBoss significantly !

Can we do this ?

This question has been asked many times before.
The answer is in principle yes, but in practice no.

The main the practicality problems are:

* Not all services define their dependencies properly. It's only the implicit
sequential order that makes them work correctly.

* The Sun classloading doesn't work very well concurrently. We avoid a number
of potential problems by doing a sequential load of most of the key classes
during the bootstrap.
We have workarounds, but you can still see these issues on some
platforms.

* The "big ball of mud" unified classloader depends for consistency/predictability
on having the classloaders created in order. It is this order that is used to
search for global classes. If these were registered concurrently across threads
the search order would be unpredicatble across different reboots.
In fact, 4.x has no mechanism for classloading dependency at all.

* The old 4.x MicroKernel has some thread safety issues,
e.g. invocations of create/start are done in synchronized blocks which could
cause deadlock problems, there are others related to ConcurrentModificationExceptions.
This is not an issue with the 5.x Microcontainer.

* We'd need to find a way for the BarrierService and "Server Started" notification
to know when all the startup threads have finished, otherwise
they would be broken.

* Finally, less of a practical issue, but more a generic note. Using mulitple threads
for startup will only work if there are multiple cpus/cores. In non SMP environments
the bootstrap could be slower with multiple competing threads, although
it may be able to take advantage of times during disk I/O waits
to do some processing on other threads?

[brian.stansberry]

although it may be able to take advantage of times during disk I/O waits to do some processing on other threads?

Not disk I/O, but threads blocking waiting for messages that will never arrive. The reason JGroups channels take a long time to start is they send a discovery packet to the network and block waiting a few seconds (configurable) to get a few replies. The first node in the cluster will of course receive no replies, so for each channel it waits a few seconds.

BTW, I think Bela's intent here was to discuss this only in context of an improvement in AS 5.x.

[adrian.brock]

Couldn't you just implement this anyway in your services,
using the 2PC create/start lifecycle?

private volatile Thread startupThread;

public void create() throws Exception
{
 startupThread = new Thread(new Runnable()
 {
 public void run()
 {
 // start channel here
 }
 });
 startupThread.start();
}

public void start() throws Exception
{
 startThread.join(); // wait for startup to complete before injecting ourselves onto others
 startThread = null;
}

Of course, we could do better (e.g. using thread pools and "optimizing"
related work across threads based on dependencies) if this was a feature of the MC.

[adrian.brock]

A simple mechansim that I can think of would to have a new ControllerMode
http://viewvc.jboss.org/cgi-bin/viewvc.cgi/jbossas/trunk/jboss-dependency/src/main/org/jboss/dependency/spi/ControllerMode.java?revision=31727&view=markup&pathrev=31727
which you could add to your deployment.

<deployment mode="asynchronous">
 <!-- beans here -->
</deployment>

which would work like the "automatic" mode, except it submits the work
to a background thread pool.

In this scenario, it would largely be "self-optimizing" since
it would automatically advance related dependencies that are eligible once
these beans reach the relevant stages.

But this wouldn't work so well for the old JMX MBeans which are hardwired to
operate in "manual" mode driven by the "SARDeployer"/ServiceController
for backwards compatibilty reasons.

The Deployments (which are also managed for dependency by the MC)
also run in "manual" mode which might be a good thing
since it would still ensure that classloaders are constructed in a predictable order
on one thread.

[brian.stansberry]

The ControllerMode approach sounds better. I expect starting a thread in create() won't help much in many cases, as the main thread will quickly proceed to start and then block.

Unfortunately, JBM opens 2 of the 3 underlying JChannels in AS 5, and it's deploying as an mbean.

HAPartition is opening the 3rd channel and definitely can benefit from this.

The other services (JBC) are opening MuxChannels, which really just multiplex on the underlying JChannel and thus don't block. But, we want to move to using the JGroups shared transport, in which case each service will have its own JChannel; I need to think through how to use the ControllerMode to improve the startup speed.

The Deployments (which are also managed for dependency by the MC)
also run in "manual" mode which might be a good thing
since it would still ensure that classloaders are constructed in a predictable order
on one thread.

Did you mean the "Deployers" here?

[adrian.brock]

The Deployments (which are also managed for dependency by the MC)
also run in "manual" mode which might be a good thing
since it would still ensure that classloaders are constructed in a predictable order
on one thread.

Did you mean the "Deployers" here?

No, I mean't the deployments. Each deployment is managed by the MC
as it goes through its states. It is "manual" because it will only advance
to the next state when the "MainDeployer' says it can.

During startup The MainDeployer tells all deployments to advance to Parse,
then all to advance to Describe, repeated for each stage.
But the MC may veto the request if a deployment is missing a dependency.
e.g. a deployment could have a classloading dependency on a different deployment
that is not deployed (osgi style classloading rules).

But in general this all happens currently on the bootstrap or hot deployment scanner
thread.

[belaban]

* Not all services define their dependencies properly. It's only the implicit
sequential order that makes them work correctly

The let's make the default sequential unless we explicitly tag a deployment as asynchronous (as you mention further down). Although I'd prefer it to be the other way round.

* The Sun classloading doesn't work very well concurrently. We avoid a number of potential problems by doing a sequential load of most of the key classes during the bootstrap.

What prevents us from loading those key classes up front, and then deploying in parallel ?

* The "big ball of mud" unified classloader depends for consistency/predictability on having the classloaders created in order. It is this order that is used to search for global classes. If these were registered concurrently across threads the search order would be unpredicatble across different reboots.

The big ball of mud is controlled by us, right ? So we can change it. The above sounds like a kludge anyway... (Caveat, I'm not a classloader expert... :-))

[quote
* The old 4.x MicroKernel has some thread safety issues,
e.g. invocations of create/start are done in synchronized blocks which could cause deadlock problems, there are others related to ConcurrentModificationExceptions.
This is not an issue with the 5.x Microcontainer.


Then the feature is for AS 5 only, and not available in 4.x.

* We'd need to find a way for the BarrierService and "Server Started" notification to know when all the startup threads have finished, otherwise
they would be broken.

Yes. One big barrier would probably be enough to sync on for all threads when they're completed, at least in a first impl. This would tell us that startup of JBossAS is 'done'.

* Finally, less of a practical issue, but more a generic note. Using mulitple threads for startup will only work if there are multiple cpus/cores. In non SMP environments the bootstrap could be slower with multiple competing threads, although
it may be able to take advantage of times during disk I/O waits
to do some processing on other threads?

[belaban]

Couldn't you just implement this anyway in your services,
using the 2PC create/start lifecycle?

private volatile Thread startupThread;

public void create() throws Exception
{
 startupThread = new Thread(new Runnable()
 {
 public void run()
 {
 // start channel here
 }
 });
 startupThread.start();
}

public void start() throws Exception
{
 startThread.join(); // wait for startup to complete before injecting ourselves onto others
 startThread = null;
}

Of course, we could do better (e.g. using thread pools and "optimizing"
related work across threads based on dependencies) if this was a feature of the MC.

No, since create only creates the channel, but start() connects it (time consuming operation), we'd block the main thread for the same time with the join() as if we deployed on the main thread directly.

We could create a separate threads and not join() it, but that might have unintended consequences, such as channels not being ready when JBossCache (which has a dep on a channel) starts...

[dmlloyd]

* Not all services define their dependencies properly. It's only the implicit sequential order that makes them work correctly.

If they don't define their dependencies properly, what guarantee is there that they will even work at all?

* Finally, less of a practical issue, but more a generic note. Using mulitple threads for startup will only work if there are multiple cpus/cores. In non SMP environments the bootstrap could be slower with multiple competing threads, although it may be able to take advantage of times during disk I/O waits
to do some processing on other threads?

Two things - first, I don't even know where I'd be able to buy a server-class system (heck even a desktop-class system come to think of it) that didn't have at least two cores (if not two or more multi-core CPUs). But that issue aside, any service that has non-CPU bound tasks (which includes not just disk I/O but network I/O) could possibly see a performance benefit from concurrent startup, even on a single-core system. And in any case we're talking about service deployment - I doubt the additional overhead of context switching between multiple threads could possibly make a measurable impact on performance. Especially in comparison to examples like Bela's 5-second timeout.

Also one could reasonably expect that rather than firing off a thread for every task, there would be a thread pool Executor of some sort. The thread pool could be sized in proportion to the number of cores available, or in some configurable fashion. Then the simple rule would be, any deployment task with no remaining dependencies gets put into the queue for execution.

I don't really understand the hairy details behind some of the classloader problems, but surely they could be worked around by imposing some reliance on the dependency system for them as well?

I agree with Bela - a concurrent startup would be hugely beneficial for startup time, not to mention just being cool. In fact when first learning about the MC and its dependency system, I had simply assumed that dependencies were resolved concurrently and never gave it a moment's thought. It wasn't until I spoke with Bela at JBW that I learned otherwise.

[adrian.brock]

I don't really understand the hairy details behind some of the classloader problems, but surely they could be worked around by imposing some reliance on the dependency system for them as well?

If all the deployments specified their classloading dependencies
(osgi style) the problem would go away.

This also means automagically adding javaee (and the relevant container)
classloading dependencies into ejb/war deployments etc.
since we can't expect users to do this for themselves.

It's doable, but I doubt it is going to happen soon?

Another side affect of this, is that those deployments that
explictly list their package exports wouldn't have to go through the
scan of the filesystem/jars to determine what packages they export.

[adrian.brock]

* Not all services define their dependencies properly. It's only the implicit sequential order that makes them work correctly.

If they don't define their dependencies properly, what guarantee is there that they will even work at all?

They use the "deterministic" implicit rules, which means you can test
it works for single bootstrap, but if you try to redeploy something that
only has implicit dependencies, you'll find it doesn't understand what is going on :-).

[adrian.brock]

If all the deployments specified their classloading dependencies
(osgi style) the problem would go away.

The proposed mavenisation of the JBossAS build might help in determining
what these dependencies really are.

[alesj]

A simple mechansim that I can think of would to have a new ControllerMode
http://viewvc.jboss.org/cgi-bin/viewvc.cgi/jbossas/trunk/jboss-dependency/src/main/org/jboss/dependency/spi/ControllerMode.java?revision=31727&view=markup&pathrev=31727
which you could add to your deployment.

<deployment mode="asynchronous">
 <!-- beans here -->
</deployment>

which would work like the "automatic" mode, except it submits the work
to a background thread pool.

In this scenario, it would largely be "self-optimizing" since
it would automatically advance related dependencies that are eligible once
these beans reach the relevant stages.

How would this relate with KernelDeploymentDeployer, meaning that we need to 'convince' it not to split up the deployment into separate non-related beans.
Or how do you plan to group all beans from a deployment marked with 'asynch' and install them in the same background thread?

[adrian.brock]

"alesj" wrote:

How would this relate with KernelDeploymentDeployer, meaning that we need to 'convince' it not to split up the deployment into separate non-related beans.
Or how do you plan to group all beans from a deployment marked with 'asynch' and install them in the same background thread?

Why would you need to?

You simply have an Executor backed by a thread pool + queue on the controller.
You submit the task to the executor then it will do something like:

<deployment mode="Asychronous">
 <bean name="A"/>
 <bean name="B"><depends>A</bean>
</deployment>

B is not satisified until A is ready so you can't submit it to the thread pool.

However submitting A to the thread pool will reworkout that B is now satisified
and can do it on the same thread.

The real issue is when you have some other thread come in during the intermediate
step. i.e. A is now done, but we haven't figured out that B will be done on the same
thread.
Some other thread might say, I can do B so you have two threads working on B.

I believe this isn't really possible because of the lock in the controller, but we
don't really many tests in this area so using ione of my laws of programming:

Not Tested == Doesn't work

P.S. You don't need to actually make concurrent deployment work for 2.0.0.GA
You just need to add the enum value and through an error saying
"not implemented yet" when somebody tries to use it.
The important thing is that we don't change the api after 2.0.0.GA
we can implement missing features without changing the api if the api already exists.