-
15. Re: Graceful Shutdown
"ALRubinger" wrote:
"jason.greene@jboss.com" wrote:
Let me give an example. Someone defines a service A, which supports remote invocation. There is no dependency between service A and the EJB container (Service B), because it doesn't make sense (they have nothing to do with each other). Then instance SFSB Foo of Service B *dynamically* decides to call service A during an invocation. Then during "graceful" shutdown, service A is stopped first.
Haha, that's exactly my point. There's a missing dependency between B->A. B requires A to function correctly, regardless of whether its a dynamic thing or something we support w/ injection. So if they're deployed in the same container/cluster, I'd expect there to be an explicit dep declared.
For what use case is this not possible?
The one I described. Another variant is jmx-console. You dont know what is going to be invoked until someone clicks on something. -
16. Re: Graceful Shutdown
"bstansberry@jboss.com" wrote:
Could the deployment of the acceptor be part of the deployment of the service? It's just a separate MC bean that depends on the core service.
That eliminates the issue of deployment phases.
Yeah i thought it's just a MC bean with an install callback :) It's just a question when this bean gets deployed. Basically there is no way we can eliminate deployment phases - where at the moment each Profile represents a deployment phase (mainDeployer.process()).
Just think what would happen if you move the "TomcatService" into the 'web.deployer'. This would start the connectors/acceptors/endpoints before even looking at what's in 'deploy'. Which means that the acceptor would be available, but without any web app installed. The same would happen if we have multiple smaller profile description.
Although this is not specific to graceful shutdown it could affect it depending on the way you signal the acceptors to gracefully stop. If the "central management bean" is called to signal the shutdown, before we actually stop AS then it could work. Otherwise the same would happen - we undeploy all user applications, before undeploying the acceptor deployment. -
17. Re: Graceful Shutdown
"bstansberry@jboss.com" wrote:
"emuckenhuber" wrote:
Hmm graceful shutdown should not be part of the MC bean lifecycle, as it would then always shutdown in a graceful manner. This should be an optional way to shutdown AS - triggered with a different signal or management action.
Additionally with the graceful shutdown there will most likely be a timeout which then is going just stop AS. This does not seem to fit very well with MC lifecycle actions, since we would basically need to interrupt a action during a state transition (pre_stop -> stop).
With the acceptor concept we should be able to deal with these issues even though we're essentially using the MC lifecycle.
The acceptors can all be registered with a central management bean that can set a property as to how long they should wait to return from stop(). -1, don't do anything, just return, 0 wait as long as it takes, > 0, wait that long. The default is -1 or something configurable at the server level. The management console sets to something else if a graceful shutdown is invoked.
As I wrote this it felt hacky, and it is. Emanuel, your instincts and Jason's are right; this isn't really an MC issue. It's a step in the shutdown process that occurs before the normal undeployment process starts. If we have a bunch of acceptors, whatever is controlling the shutdown can tell them to stop; we don't need the MC to do it. That is back to the original design we discussed in Westford."jason.greene@redhat.com" wrote:
Let me give an example. Someone defines a service A, which supports remote invocation. There is no dependency between service A and the EJB container (Service B), because it doesn't make sense (they have nothing to do with each other). Then instance SFSB Foo of Service B *dynamically* decides to call service A during an invocation. Then during "graceful" shutdown, service A is stopped first.
Jason, are you concerned about solving this case (eventually), or are you just using it to make a point about dependencies? Segregating internal calls from those coming via remote endpoints, and then closing off the remote endpoints before stopping/undeploying anything will handle most such cases. The only ones it won't are where Service B makes a remote call to non-HA Service A, even though both are deployed locally. Which case is IMHO out of scope. :-) -
18. Re: Graceful Shutdown
"emuckenhuber" wrote:
"bstansberry@jboss.com" wrote:
Could the deployment of the acceptor be part of the deployment of the service? It's just a separate MC bean that depends on the core service.
That eliminates the issue of deployment phases.
Yeah i thought it's just a MC bean with an install callback :) It's just a question when this bean gets deployed. Basically there is no way we can eliminate deployment phases - where at the moment each Profile represents a deployment phase (mainDeployer.process()).
Just think what would happen if you move the "TomcatService" into the 'web.deployer'. This would start the connectors/acceptors/endpoints before even looking at what's in 'deploy'. Which means that the acceptor would be available, but without any web app installed. The same would happen if we have multiple smaller profile description.
Although this is not specific to graceful shutdown it could affect it depending on the way you signal the acceptors to gracefully stop. If the "central management bean" is called to signal the shutdown, before we actually stop AS then it could work. Otherwise the same would happen - we undeploy all user applications, before undeploying the acceptor deployment.
I agree with your conclusion "if the "central management bean" is called to signal the shutdown, before we actually stop AS then it could work." But I want to clarify some re: my understanding of the acceptor and how it ties in to deployment stages.
AIUI, an acceptor is not scoped to a connector. It's scoped to the thing that handles requests for a particular component (a webapp, and EJB). In Tomcat terms, it would be the first thing in the webapp-specific Pipeline that results in a call on the servlet. -
19. Re: Graceful Shutdown
"bstansberry@jboss.com" wrote:
Jason, are you concerned about solving this case (eventually), or are you just using it to make a point about dependencies? Segregating internal calls from those coming via remote endpoints, and then closing off the remote endpoints before stopping/undeploying anything will handle most such cases. The only ones it won't are where Service B makes a remote call to non-HA Service A, even though both are deployed locally. Which case is IMHO out of scope. :-)
I was just making a point about static dependencies not being a catch-all. The remote shut-off would totally do the trick, which is why I liked your earlier suggestion about starting with just a simple solution for the web tier. -
20. Re: Graceful Shutdown
"bstansberry@jboss.com" wrote:
But I want to clarify some re: my understanding of the acceptor and how it ties in to deployment stages.
AIUI, an acceptor is not scoped to a connector. It's scoped to the thing that handles requests for a particular component (a webapp, and EJB). In Tomcat terms, it would be the first thing in the webapp-specific Pipeline that results in a call on the servlet.
Yes that's what's my understanding of an acceptor is as well, but it seems we are talking about different issues :)
AFAIR you briefly looked at the xml profile configuration prototype i did almost a year ago. The concept behind this is that all this sub-profiles are activated (deployed) before we move on to the user 'deploy' folder - to make sure that all deployers are started, installed and registered at the mainDeployer. Additionally you should combine deployer and runtime services into one consistent profile description (incl. the acceptor).
This would also mean that e.g. acceptor.start() will be called before even looking at the user deploy folder.
Doing a brief research - in the TomcatService there is a handleNotification, which actually starts and stops the connectors. So the question is if the acceptor interface should also specify a contract for this. Otherwise we could end up opening ports without any user application installed, as this is something MC also does not know about. -
21. Re: Graceful Shutdown
"emuckenhuber" wrote:
Doing a brief research - in the TomcatService there is a handleNotification, which actually starts and stops the connectors. So the question is if the acceptor interface should also specify a contract for this. Otherwise we could end up opening ports without any user application installed, as this is something MC also does not know about.
That's why i was asking if we need something like a 'acceptor' deployment phase - which is deployed last and undeployed first. So that you can just write your usual MC bean deployment thingy. In the end this is also not something ProfileService knows about, but it controls when deployments are deployed.
It most probably makes more sense to move the responsibility to the "central management bean" and the acceptor interface - where you basically have to decouple the acceptor from the MC lifecycle, or so ? -
22. Re: Graceful Shutdown
Warning: I wander all over in this post as I think things through. :-)
"emuckenhuber" wrote:
Additionally you should combine deployer and runtime services into one consistent profile description (incl. the acceptor).
This would also mean that e.g. acceptor.start() will be called before even looking at the user deploy folder.
I think the deployment of an "acceptor" can be part of the deployment of the stuff in the deploy folder; e.g. its an element in what gets deployed as part of a war. It's not a general part of the web server that exists independent of a war. For example, to handle a similar problem (url]https://jira.jboss.org/jira/browse/JBAS-6534[/url]) deployment of a clustered war includes installing a Tomcat Valve. I'm starting to experiment with a similar thing as the implementation of an "acceptor" for the web container.
When the acceptor gets called to execute it's graceful shutdown logic is a different thing from how it gets deployed. The call of course needs to happen very early in the shutdown process."emuckenhuber" wrote:
Doing a brief research - in the TomcatService there is a handleNotification, which actually starts and stops the connectors. So the question is if the acceptor interface should also specify a contract for this. Otherwise we could end up opening ports without any user application installed, as this is something MC also does not know about.
Interesting question. The way the TC Connector gets started/shutdown is pretty ad-hoc. I talk above about a "war-scoped acceptor", which may be necessary. But maybe I'm wrong and it can all be done with a single acceptor associated with the connector. As I experiment I'll try and see how that can fit.That's why i was asking if we need something like a 'acceptor' deployment phase - which is deployed last and undeployed first. So that you can just write your usual MC bean deployment thingy. In the end this is also not something ProfileService knows about, but it controls when deployments are deployed.
It most probably makes more sense to move the responsibility to the "central management bean" and the acceptor interface - where you basically have to decouple the acceptor from the MC lifecycle, or so ?
Do deployment phases fit with the shortish-term vision for the profile service? IIRC when I looked at this before, the order in which sub-profiles got deployed was based on the order in which they are specified in the xml; i.e. no real deployment phases any more.
If there are deployment phases, is it this "central management bean" that gets deployed in the last deployment phase? Acceptors are deployed with the runtime services; get injected into central management bean when it deploys last. In start(), which is now the last thing, it activates the acceptors. In stop() it tells the acceptors to do their graceful shutdown stuff.
Without deployment phases that can still work; the "central management bean" is just part of the last sub-profile in the list. -
23. Re: Graceful Shutdown
"bstansberry@jboss.com" wrote:
"emuckenhuber" wrote:
Additionally you should combine deployer and runtime services into one consistent profile description (incl. the acceptor).
This would also mean that e.g. acceptor.start() will be called before even looking at the user deploy folder.
I think the deployment of an "acceptor" can be part of the deployment of the stuff in the deploy folder; e.g. its an element in what gets deployed as part of a war. It's not a general part of the web server that exists independent of a war. For example, to handle a similar problem (url]https://jira.jboss.org/jira/browse/JBAS-6534[/url]) deployment of a clustered war includes installing a Tomcat Valve. I'm starting to experiment with a similar thing as the implementation of an "acceptor" for the web container.
When the acceptor gets called to execute it's graceful shutdown logic is a different thing from how it gets deployed. The call of course needs to happen very early in the shutdown process.
Hmm ok, i thought of an acceptor more of a facade handling both. First making sure connectors and other ports are openend once all apps are deployed and closed before undeploying. As well as handling the actually context bound to specific applications you were referring to."bstansberry@jboss.com" wrote:
"emuckenhuber" wrote:
Doing a brief research - in the TomcatService there is a handleNotification, which actually starts and stops the connectors. So the question is if the acceptor interface should also specify a contract for this. Otherwise we could end up opening ports without any user application installed, as this is something MC also does not know about.
Interesting question. The way the TC Connector gets started/shutdown is pretty ad-hoc. I talk above about a "war-scoped acceptor", which may be necessary. But maybe I'm wrong and it can all be done with a single acceptor associated with the connector. As I experiment I'll try and see how that can fit.
Well i think it would be good to handle this a better way than requiring services to listen to a jmx notification, since JMX should be optional at one point. Although we need to check how much sense it makes for other services. With the "pretty ad-hoc" startup/shutdown of TC connectors you refer to time it needs to start. Compared to starting of some clustering services?
It was more an issue i noticed - so it might does not really work trying to solve both with the same interface. So far i considered what happens in the gracefulShutdown() method as implementation detail - therefore mostly ignored that. So it really depends on your experiment what would should be done."bstansberry@jboss.com" wrote:
That's why i was asking if we need something like a 'acceptor' deployment phase - which is deployed last and undeployed first. So that you can just write your usual MC bean deployment thingy. In the end this is also not something ProfileService knows about, but it controls when deployments are deployed.
It most probably makes more sense to move the responsibility to the "central management bean" and the acceptor interface - where you basically have to decouple the acceptor from the MC lifecycle, or so ?
Do deployment phases fit with the shortish-term vision for the profile service? IIRC when I looked at this before, the order in which sub-profiles got deployed was based on the order in which they are specified in the xml; i.e. no real deployment phases any more.
If there are deployment phases, is it this "central management bean" that gets deployed in the last deployment phase? Acceptors are deployed with the runtime services; get injected into central management bean when it deploys last. In start(), which is now the last thing, it activates the acceptors. In stop() it tells the acceptors to do their graceful shutdown stuff.
Hmm, yeah DeploymentPhase was maybe a wrong term. I should have rather referred to it as DeploymentPhaseGroup or something similar. Where each Profile defines a deployment phase, but we have an additional helper defining a sort of relative ordering. Since Profiles like "deploy/" and "farm/" would be considered as user profiles they would end up at the end of this list - even without specifying any additional dependencies. So there could be a addition for an activator phase/group thingy.
As you referred to the ordering in the xml - yeah this is something we are working on to get done in a better way. There will be post about this once there this is in a better shape."bstansberry@jboss.com" wrote:
Without deployment phases that can still work; the "central management bean" is just part of the last sub-profile in the list.
Hmm i'm not really sure if the install callbacks work if you install it at the end - so this bean should be rather get deployed quite early.
First i thought we can connect that to the ProfileServiceBootstrap, but if you need an Executor created by jboss-threads this won't make much sense. Maybe we should just wait a bit on your experiment and then look again where to exactly put that? -
24. Re: Graceful Shutdown
When I used "ad-hoc" about the connector start/stop, my meaning was the same as yours -- relying on the JMX notification is not good, as it's completely outside of how all other ordering is done, can break at any time etc.
The more I think about it, the more I like combining the per-webapp thingys into a single thingy that also handles the connector start/stop. It becomes:
GracefulStart:
Start the connector.
Graceful stop:
Coordinate with the deployed webapps and their session managers to go through all the details of draining sessions and ensuring in-flight requests have completed.
Stop the connector.
My experiment is going to follow this approach until it proves unworkable.
Re: install callbacks if the "central management bean" is deployed last; I have a vague recollection that I thought that wouldn't work for some other usage, but it did. But I could be misremembering, or maybe there was some weird fluke that made it work.
If that's a problem, we can just divide the "central management bean" into two parts:
Registry: deployed in bootstrap. Gets all the install callbacks
Invocation: deployed at the end. Has registry injected. Invokes on the registered items.