ServiceController Enhancement

[jdoble]

My company has developed a network management appliance using JBoss as an underlying framework. Up to now, we haven't had any need to modify the JBoss code itself, but we recently ran into a issue that can easily be resolved with a minor modification to org.jboss.system.ServiceController.java.

Given that our product is an appliance, we have had to write additional Operations, Administration and Maintenance (OAM) services (e.g. system monitoring & automatic recovery, backup & restore, etc.) which typically require individual services, or groups of services to be stopped and/or started. We have been invoking the stop and start JMX operations in ServiceController.java to request the individual stop/start operations, so that ServiceController's cached service states are kept up to date, and so that any services that are listening to ServiceControllers notifications are informed when other services are started or stopped.

The problem we currently have is that the start method not only starts the requested service, but then proceeds to start any services that depend an the requested service, which will then start any services that depend on those services and so on. This is a problem for us because there are times when we need to start a service, do some work, then start the dependent services.

The solution we are proposing is this:

Modify the start method in ServiceController.java to have the following signature:

public synchronized void start(ObjectName serviceName, Boolean startDependents)

and modify the code which starts dependent services to look like the following:

if (startDependents.booleanValue())
{
 // Those that depend on me are waiting for my start, recursively start them
 log.debug("Starting dependent components for: " + serviceName
 + " dependent components: " + ctx.dependsOnMe);
 ArrayList tmp = new ArrayList(ctx.dependsOnMe);
 for (int n = 0; n < tmp.size(); n++)
 {
 // marcf fixme circular dependencies?
 ServiceContext ctx2 = (ServiceContext) tmp.get(n);
 start(ctx2.objectName);
 }
 tmp.clear();
}

In order to ensure that existing clients of ServiceController do not need to be modified, add the following method:

public synchronized void start(ObjectName serviceName) throws Exception
{
 start(serviceName, Boolean.TRUE);
}

Both of the start methods would need to be tagged as JMX operations, so that they clients can invoke them via the JMX server.

I am happy to make this change (if approved/allowed/etc.). I am new to the JBoss contributor process, so I will likely have a few questions about how best to proceed. I am hoping that this forum was the right place to start. If not, please advise.

[richardfriedman]

I would like hear more about this appliance.

[adrian.brock]

The correct way to do is to allow each "service" to have its target state
defined independently. Of course a dependent can never be in a higher
state than a dependee regardless of the requested target state.

http://www.jboss.org/index.html?module=bb&op=viewtopic&t=64673

[jdoble]

Hmm. I'm not sure what you mean by "target state" and "defined independently". At first I thought that maybe you were suggesting that we define another state variable for our services, which would be orthogonal to the existing notion of service lifecycle. That way, we could bypass ServiceController to do our starting and stopping (i.e. even if ServiceController views our service as started, we could have orthogonal "active" and "inactive" states. Of course it would not be completely orthogonal, because if the state combination (stopped, active) wouldn't make any sense. The problem I have with this approach is that it complicates the service state machines for our services (which my background in telecom tells me is almost always a bad idea, because it makes it much more difficult to prove that your state machines are correct, and would be a royal pain because we have dozens of services), and probably requires us to create another service, comparable to ServiceController, to manage the active/inactive states of services, and to broadcast notifications to other interested services. I think it would be a shame to do this, because the existing service lifecycle is so close to what we need. In fact, it was one of the things that drew me to JBoss in the first place.

My second thought is that I may not be understanding your reply at all, in which case, some clarification would be appreciated.

We have always been surprised by the semantics of the start method in ServiceController. The stop method works as we would expect (i.e. when someone requests a service to be stopped, all of its dependents are first stopped). For the start method, when someone requests a service to be started, we would have expected any services the requested service depends on to first be started, then the requested service, and no dependents on the requested service). Instead, if any services the requested service depends on are not started, the method just returns. If not, the service is started, then the transitive closure of its dependents are started. This is great if your plan is to start all services anyway, but no so great if you need finer control, or need to do work between starting services.

That being said, JBoss has been very useful to us as an overall framework, so we have attempted to work around this behavior (i.e. we ensure that all of the services a given service depends on are started before we request the given service to be started), but as is, the automatic starting of dependent services cannot be worked around, without introducing a completely orthogonal concept of service state as described earlier.

Maybe (hopefully) I am missing or misunderstanding something. Please clarify the approach which you consider to be the "correct" one.

[adrian.brock]

I don't think the problem is that I misunderstood your request.
Please try to keep your post less verbose (or at least less dense in text :-).

As it is, I'm not going to read your full posts, just search for the optimal phrase that
says what you actually want.

I also never consider solutions without understanding the problem.
You just proposed a particular solution that does not consider the full problem space.

This is a problem for us because there are times when we need to start a service, do some work, then start the dependent services.

Which under the new controller that allows manual
control would be (in pseudo code)

public void myRoutine(ObjectName service)
{
 // Get the context we want to control
 ControllerContext context = controller.getContext(service);

 // Make sure dependents don't progress beyond configured
 for (ControllerContext dependent: context.getDependents())
 dependent.change(dependent, ControllerState.CONFIGURED);

 // Start the context
 controller.change(context, ControllerState.INSTALLED);

 // Do some work here
 doWork();

 // Start the dependents
 for (ControllerContext dependent: context.getDependents())
 dependent.change(dependent, ControllerState.INSTALLED);
}

I also don't see why even under the current model, you can't have
the doWork() either inside start() or on an MBean notification listening
for the start state change, but there you go....

[jdoble]

1) Apologies. When I can't understand the other guy's reply, I try to assume there is a possibility the other guy never understood me.

2) I don't know what a ControllerContext is, or how CONFIGURED and INSTALLED relate to started and stopped, but I'll do some digging.

3) I can't have doWork() inside start because the work is not associated with the service that is being started. It depends on the service being started, but is not that service's business. Doing the work in an MBean notification listening for the state change will work only if the listeners get to do their thing before ServiceController goes ahead and starts the dependents (which I will research), and only if we want to go ahead and start the dependents regardless of the outcome of the work (may not always be the case).

[adrian.brock]

"jdoble" wrote:

2) I don't know what a ControllerContext is, or how CONFIGURED and INSTALLED relate to started and stopped, but I'll do some digging.

It is a part of the new microcontainer being developed for JBoss5,
see the link I posted above.
You will be able to see a limited version of it in jboss-4.0.3

The important part of my "solution" is that the controller is responsible for implementing
certain basic primitives.

"Scripting" more complex usecases that require additional/more complex
behaviours should be done outside the controller using those primitives.

If you introduce a method into the controller for every possible usecase that somebody
can imagine you will very quickly get a piece of software that is both untestable and
unsupportable because the combinations of configuration that need to be tested
grow like n^2.

[jdoble]

We've been using 4.0.0, so that explains much. I think the use of more primitive management operations will address our requirements. The key for us is to be able to cause state transitions in individual services, along with associated notifications, without undesirable side-effects.