Deploying services and error handling

[aloubyansky]

Right now standalone fails to start because there are nonexistent deployments in its config. (BTW, it's logged only in its boot.log, not on the console. So the console logs include just JBoss Bootstrap Environment and some envvar value. And then silently returns to the shell prompt.)

The boot.log contains

{quote:title=boot.log}14:36:51,090 ERROR [stderr] org.jboss.as.server.ServerStartException: Failed to start server
14:36:51,090 ERROR [stderr]     at org.jboss.as.server.AbstractServer.start(AbstractServer.java:106)
14:36:51,090 ERROR [stderr]     at org.jboss.as.server.StandaloneServer.start(StandaloneServer.java:71)
14:36:51,091 ERROR [stderr]     at org.jboss.as.server.Main.boot(Main.java:92)
14:36:51,091 ERROR [stderr]     at org.jboss.as.server.Main.main(Main.java:71)
14:36:51,091 ERROR [stderr]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
14:36:51,091 ERROR [stderr]     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
14:36:51,091 ERROR [stderr]     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
14:36:51,091 ERROR [stderr]     at java.lang.reflect.Method.invoke(Method.java:597)
14:36:51,091 ERROR [stderr]     at org.jboss.modules.Module.run(Module.java:373)
14:36:51,092 ERROR [stderr]     at org.jboss.modules.Main.main(Main.java:171)
14:36:51,092 ERROR [stderr] Caused by: java.lang.RuntimeException: Deployment root does not exist."/c:/Users/avoka/github.com/jboss-as/build/target/server/deployments/my-app.ear"
14:36:51,092 ERROR [stderr]     at org.jboss.as.model.ServerGroupDeploymentElement.activate(ServerGroupDeploymentElement.java:218)
14:36:51,092 ERROR [stderr]     at org.jboss.as.model.Standalone.activate(Standalone.java:461)
14:36:51,092 ERROR [stderr]     at org.jboss.as.server.AbstractServer.start(AbstractServer.java:101)
14:36:51,092 ERROR [stderr]     ... 9 more{quote}

So, the first thing I'd like to agree upon is whether the server should really shutdown if at least one of the deployments menetioned in its config wasn't found. If we want to follow the traditional approach then it shouldn't but should log the errors.

In fact, I can see that there is in fact some logic to log the failed services but it just doesn't get to that point because the ServiceActivator impl in this case throws a RuntimeException

{code}public void activate(final ServiceActivatorContext context) {
        final String deploymentName = key.getName() + ":" + key.getSha1HashAsHexString();
        log.info("Activating server group deployment: " + deploymentName);
        final VirtualFile deploymentRoot = VFS.getChild(getFullyQualifiedDeploymentPath(key.getName()));
        if (!deploymentRoot.exists())
            throw new RuntimeException("Deployment root does not exist." + deploymentRoot);{code}

And then AbstractServer just re-throws it

{code}catch (Throwable t) {
            throw new ServerStartException("Failed to start server", t);
        }{code}

So, I was thinking perhaps we should introduce some ServiceActivationException and hanle it so that we collect all the failed deployments which the DeploymentCallback is supposed to report.

[dmlloyd]

Alexey Loubyansky wrote:

Right now standalone fails to start because there are nonexistent deployments in its config. (BTW, it's logged only in its boot.log, not on the console. So the console logs include just JBoss Bootstrap Environment and some envvar value. And then silently returns to the shell prompt.)

The boot.log contains

[...]

So, the first thing I'd like to agree upon is whether the server should really shutdown if at least one of the deployments menetioned in its config wasn't found. If we want to follow the traditional approach then it shouldn't but should log the errors.

In fact, I can see that there is in fact some logic to log the failed services but it just doesn't get to that point because the ServiceActivator impl in this case throws a RuntimeException

[...]

And then AbstractServer just re-throws it

[...]

So, I was thinking perhaps we should introduce some ServiceActivationException and hanle it so that we collect all the failed deployments which the DeploymentCallback is supposed to report.

Yeah this is definitely wrong.

That said, there are several points in which a deployment can possibly fail.

1. A deployment can fail validation on the domain controller.  In this event, the deployment is not added to the domain and an error is thrown from the deployer API, so this does not impact the server.
2. A deployment can fail to be added to the MSC batch.  For example, a service may be attempted to be defined which already exists in the batch (in fact, at present this is the only possible error outside of outright code bugs).  In this event, the correct thing to do is to log the error and continue.
3. A failure can happen at batch resolve time.  This is currently a problem case.  In this case the entire batch is undone.  Types of failures here include: duplicate service definition (i.e. a service was defined that already exists in the container), missing required dependency (i.e. a service was defined that depends on something that isn't there), circular dependency (i.e. two or more service definitions represent circularity in terms of their dependency links).  Right now this erases the entire batch, which means that the server cannot start if any of these errors occur.  I think that circularity is quite unlikely but the other two errors are not uncommon, and should not prevent the server from starting up.  The correct behavior is, in my opinion, to log each error and continue, possibly yielding a report of problem services after the batch is resolved (it is likely, however, that one failure will lead to others).
4. A failure can happen during service start.  In this case, the exception is logged, and all the service's dependents will be trapped in a waiting state.  This type of error will typically be due to transient conditions, like an incorrectly configured IP address, disk space issues, and so on.  The correct behavior here is to log the error and let the user decide (based on the GUI or perhaps by deployment plan policy) whether to fix the issue and retry the failed service, let the error stand, or pull out the deployment unit.

[aloubyansky]

I've created the main task for deployment error handling https://jira.jboss.org/browse/JBAS-8334 (which is assigned to me for now) and subtasks to make sure it's properly done

A deployment can fail validation on the domain controller.  In this event, the deployment is not added to the domain and an error is thrown from the deployer API, so this does not impact the server.

https://jira.jboss.org/browse/JBAS-8336 (unassigned for now)

A deployment can fail to be added to the MSC batch.  For example, a service may be attempted to be defined which already exists in the batch (in fact, at present this is the only possible error outside of outright code bugs).  In this event, the correct thing to do is to log the error and continue.

https://jira.jboss.org/browse/JBAS-8335 (unassigned at the moment)

A failure can happen at batch resolve time.  This is currently a problem case.  In this case the entire batch is undone.  Types of failures here include: duplicate service definition (i.e. a service was defined that already exists in the container), missing required dependency (i.e. a service was defined that depends on something that isn't there), circular dependency (i.e. two or more service definitions represent circularity in terms of their dependency links).  Right now this erases the entire batch, which means that the server cannot start if any of these errors occur.  I think that circularity is quite unlikely but the other two errors are not uncommon, and should not prevent the server from starting up.  The correct behavior is, in my opinion, to log each error and continue, possibly yielding a report of problem services after the batch is resolved (it is likely, however, that one failure will lead to others).

https://jira.jboss.org/browse/JBAS-8337 (assigned to me)

A failure can happen during service start.  In this case, the exception is logged, and all the service's dependents will be trapped in a waiting state.  This type of error will typically be due to transient conditions, like an incorrectly configured IP address, disk space issues, and so on.  The correct behavior here is to log the error and let the user decide (based on the GUI or perhaps by deployment plan policy) whether to fix the issue and retry the failed service, let the error stand, or pull out the deployment unit.

https://jira.jboss.org/browse/JBAS-8338 (unassigned at the moment)