Installing a context leads to resolve of other unrelated con

[jaikiran]

I was looking at the deployment framework/MC kernel flow and noticed this code in the AbstractController:

public void install(ControllerContext context) throws Throwable
 {
 // removed irrelevant code from forum post
...
 install(context, trace);
 }

protected void install(ControllerContext context, boolean trace) throws Throwable
 {

 // set the required state
 ControllerMode mode = context.getMode();
 context.setRequiredState(mode.getRequiredState());

 if (trace)
 log.trace("Installing " + context.toShortString());

 context.setController(this);
 DependencyInfo dependencies = context.getDependencyInfo();
 if (trace)
 {
 String dependsOn = "[]";
 if (dependencies != null)
 {
 try
 {
 Set<DependencyItem> set = dependencies.getIDependOn(null);
 if (set != null)
 dependsOn = set.toString();
 }
 catch (Throwable t)
 {
 log.warn("Exception getting dependencies: " + t);
 dependsOn = null;
 }
 }
 if (dependsOn != null)
 log.trace("Dependencies for " + name + ": " + dependsOn);
 }

 boolean ok = incrementState(context, trace);
 if (ok)
 {
 try
 {
 registerControllerContext(context);
 }
 catch (Throwable t)
 {
 // This is probably unreachable? But let's be paranoid
 ok = false;
 throw t;
 }
 }
 if (ok)
 {
 // Jaikiran: Why do we do this?
 resolveContexts(trace);
 }

...

1) We start with install a specific context
2) Increment the state of that specific context
3) Register the controller context for that specific context
4) Resolve "other" (unrelated) contexts. Why do we do this? As an example, when a specific context = A is being installed why do we start resolving (which internally involves incrementing/moving the context to different state) contexts B, C etc... which are not related to A? Or am i wrong in understanding this piece of code?

[alesj]

Whenever something gets installed we try to move whatever can be moved.
Or how else are you gonna know what and when something can be moved?
e.g. you might have some implicit dependencies
* some bean sets some static flag, the other one inspects that as its dependency
* we put something into JNDI, something else tries to pull that out

Not to mention it would be too much + duplicated work to check for only those which might be affected by this new install / move.
Whereas this is simple, but pretty much bullet proof concept.

[jaikiran]

"alesj" wrote:

Or how else are you gonna know what and when something can be moved?

Isn't that the job of the deployers (ex: BeanMetadataDeployer) to install (which internally will move it to various states) the specific context(s)?

"alesj" wrote:

e.g. you might have some implicit dependencies
* some bean sets some static flag, the other one inspects that as its dependency
* we put something into JNDI, something else tries to pull that out

Shouldn't an implicit dependency be turned (by some piece of code) into an explicit dependency? For #1, the other bean is actually depending on the "some bean" for the static flag, so shouldn't this dependency be created explicitly? Same holds for #2

"alesj" wrote:

Not to mention it would be too much + duplicated work to check for only those which might be affected by this new install / move.
Whereas this is simple, but pretty much bullet proof concept.

Looking at what resolveContexts does:
1) Get "all" possible From and To states
2) Try to resolve "all" contexts for each of these From state to To state
3) #2 internally involves resolving dependencies of each of these contexts
4) Install any resolved contexts

Isn't this expensive (correct me if i am wrong) and is being done for every context being installed.

[alesj]

"jaikiran" wrote:

Isn't that the job of the deployers (ex: BeanMetadataDeployer) to install (which internally will move it to various states) the specific context(s)?

Deployers are just one way of installing a component into MC.
You can do it in many different ways - take any direct API approach.

"jaikiran" wrote:

Shouldn't an implicit dependency be turned (by some piece of code) into an explicit dependency? For #1, the other bean is actually depending on the "some bean" for the static flag, so shouldn't this dependency be created explicitly? Same holds for #2

Not all dependencies could be made explicit.

e.g. let's again take the JNDI example
One bean all it cares is if certain JNDI name is present.
The other one or a few are potential candidates to provide this JNDI name.
But the catch is that providing this name depends on N variables; timestamp, weather, ...

So, there is no way to know in a particular timeframe which bean will resolve the first bean --> implicit.

"jaikiran" wrote:

Isn't this expensive (correct me if i am wrong) and is being done for every context being installed.

Re-read the JNDI example. ;-)
Or how else are you gonna make sure you don't miss a potential dependency resolving bean,
if you don't check them all?

[smarlow]

Re-read the JNDI example. ;-)
Or how else are you gonna make sure you don't miss a potential dependency resolving bean,
if you don't check them all?

One idea would be to process the dependencies statically via an installation/configuration step that generates initialization code. This could be done via the MC compiler (MCC or maybe YAMCC).

I know that we don't have this now but I like the general idea of pre-compiling/caching what ever stuff makes sense to improve performance.

[jason.greene]

Hmm, would it not be more efficient to build a dependency tree/graph in a first pass, and then process explicit deps in order. Then the remaining "implicit" deps can be processed brute-force. This of course assumes that the majority of deps are explicit, which is something we want in the AS. In fact, all of our services in AS should be explicit.

[dmlloyd]

Yeah, I don't buy this "implicit dependencies" business. The beauty of the MC deployer architecture is that when you deploy stuff, you can make dependencies which are implicit (as far as the user sees) translate to explicit dependencies before the container even sees the deployment.

As for "depending" on JNDI, I think that should always translate into a real explicit dependency, which is filled by a special bean deployment that represents the JNDI binding. These special deployments can be implicitly created by the EJB and JCA deployers, much like how I bind logging handlers to logger categories.

I think that the value of having a highly efficient (and simple) dependency resolution algorithm will far, far outweigh its support for edge cases that can be easily implemented in other ways.

[alesj]

Even when you have explicit dependencies, it's far from trivial to determine what are one's dependencies,
as the info can come from almost anywhere,
component name, type just being two of the few, probably the most trivial one's to match against.

And this is actually the real beauty of having such dependency abstraction,
the component mixture simply falls out-of-the-box.

But this again means you need to check all of them.

[dmlloyd]

Implicit dependency by type is not a valid use case. There can only be one bean of a given name, but any number of beans that are of a certain type. Anyway, is there a single instance of this type-based dependency in JBossAS?

[dmlloyd]

Okay, it might be a valid use case for autowiring or something. But you don't need to scan all dependencies every deployment to support it; you just need a map of class to a list of active deployments which is updated every time something is deployed/undeployed.

[alesj]

you just need a map of class to a list of active deployments which is updated every time something is deployed/undeployed.

This is what we're already doing.
But that's not what I wanted to highlight.

The real issue is should we sacrifice dependency API simplicity
in order to be able to handle (= gather dependency info for) each custom case.
Meaning it will become a lot more complex to define a dependency,
if you then have hard time providing "gathering" info.

And I don't think we would gain that much,
as we only check our new deployment against not yet fully installed components.

11:51:53,375 INFO [AllGraphCreator] Vertices: 834, Edges: 1561

This is the info of how many component we currently have in AS default.
Each component has in average less than 2 dependencies.
And most of then are quickly moved to Installed state,
hence checking against the rest should be pretty fast.

[jaikiran]

And I don't think we would gain that much,
as we only check our new deployment against not yet fully installed components.

11:51:53,375 INFO [AllGraphCreator] Vertices: 834, Edges: 1561

This is the info of how many component we currently have in AS default.
Each component has in average less than 2 dependencies.

That's good info. Is it from the MC tools that are being developed? Where can i find it?

[alesj]

"jaikiran" wrote:
Is it from the MC tools that are being developed? Where can i find it?

Yes.
It's here:
- http://anonsvn.jboss.org/repos/jbossas/projects/mc-tools/grapher/trunk/

Simply do "mvn package" to get grapher.war, which you deploy in AS.
There are a bunch of useful (url) commands, see map package:
- http://anonsvn.jboss.org/repos/jbossas/projects/mc-tools/grapher/trunk/src/main/java/org/jboss/mctools/grapher/map/

I need to properly document this via wiki, forum, etc.
But I'm still in a phase of polishing graphics ... :-)

[bill.burke]

Did some profiling on the default/ profile for AS.  If I'm reading the profiler correctly there are 57000 invocations on

   resolveContexts(ControllerState fromState, ControllerState toState, boolean trace)

The horrible thing is that this is a result of 400 invocations on BeanMetaData.deploy().  I think its pretty obvious that cutting down this number is the next big performance improvement to the MC/VDF.  How can we cut it down?  I suggest these steps:

1. I don't know enough about how the MC/VDF works...so, maybe there is a performance bug somewhere.  I want to step through code to make sure that we're not doing unnecessary resolveContexts() by accident.

2.  We're going to have to start indexing dependencies.  What do I mean by that?  For example, when a bean gets deployed, it should be able to look up whatever depends on it and only try to resolve those dependencies.  You can do this incrementally for explicit dependency items.  You'd also keep track of which contexts had implicit vs. explicit dependency items and only do a resolveContext on implicit ones after the explicit ones were finished.

[alesj]

You'd also keep track of which contexts had implicit vs. explicit dependency items and only do a resolveContext on implicit ones after the explicit ones were finished.

How do you track implicit ones?

As I imagine this, implicit dependency is some impl detail,

that when you "push" it, if somehow resolved, it will move fwd.

But it's an impl detail that you really don't know about.

e.g. could be some static flag via demand/supply