6 Replies Latest reply on Apr 5, 2006 11:58 AM by safnate

Tomcat connector stop on server shutdown

dimitris Apr 4, 2006 5:18 AM

We had some cases where a normal (gracefull) server shutdown to one of the nodes of a cluster, will cause some http request to still go through, even if the server is in shutdown mode, and some report a 404 error.

I think the problem is that although the Tomcat deployer registers for the server "started" jmx notification, in order to start up its connectors *after* the server is fully started, it doesn't do the reverse on shutdown. So the shutdown server hook will just go through the MainDeployer and undeploy whatever is in the deployment list. This causes the connectors to be stopped close to the end of the shutdown.

What I propose is to register for the server "stopped" (should be really called "stopping") notification as well, and stop the tomcat connectors early, thus stop processing incoming requests at the beginning of shutdown.

1. Re: Tomcat connector stop on server shutdown

dimitris Apr 4, 2006 5:23 AM (in response to dimitris)

As a sidenote I can achieve the same effect with a BarrierController registering for the server start/stop jmx notifications, and the Tomcat deployer having a dependency on the barrier.
Actions
2. Re: Tomcat connector stop on server shutdown

anil.saldhana Apr 4, 2006 10:41 AM (in response to dimitris)

"dimitris@jboss.org" wrote:

What I propose is to register for the server "stopped" (should be really called "stopping") notification as well, and stop the tomcat connectors early, thus stop processing incoming requests at the beginning of shutdown.

I do not see any issues with your proposed approach. The connectors have to be shut down at the beginning of the shutdown process rather than towards the end. This is in line with the way the connectors are started (let all deployments deploy before servicing requests; similarly, stop servicing and then undeploy the deployments).

Scott can clarify if he sees any issues with your approach.
Actions
3. Re: Tomcat connector stop on server shutdown

robb.greathouse Apr 4, 2006 3:38 PM (in response to dimitris)

When we re-deploy a war tomcat should never return a 404 because it can't find the application.

Re-deploying the war should cause requests to be queued (when not in a cluster) or it should fail over to another machine when in a cluster. When clustered it should probably be configurable whether the requests are held while waiting for re-deployment or it fails over for that application.
Actions
4. Re: Tomcat connector stop on server shutdown

dimitris Apr 5, 2006 4:34 AM (in response to dimitris)

The problem is we don't have a "redeploy" notion. Redeploy simply translates to undeploy/deploy. So after the undeployment (which could be a normal undeployment meaning the application wont' be back) we don't know this is a transient situation.
Actions
5. Re: Tomcat connector stop on server shutdown

safnate Apr 5, 2006 11:49 AM (in response to dimitris)

Question:

What would happen if just prior to the shutdown event, a request was received. Would that request be allowed to complete before the shutdown is allowed to start (undeployment, etc...)?

Basically, I'm hoping that a shutdown event would allow two things.
1) Block any new requests
2) Allow any already running requests to complete
3) Finish shutdown.

Nate
Actions
6. Re: Tomcat connector stop on server shutdown

safnate Apr 5, 2006 11:58 AM (in response to dimitris)

"dimitris@jboss.org" wrote:
The problem is we don't have a "redeploy" notion. Redeploy simply translates to undeploy/deploy. So after the undeployment (which could be a normal undeployment meaning the application wont' be back) we don't know this is a transient situation.

One issue we're facing is that the 'High Availability' doesn't work well when doing redeployments. If we redeploy, during the window of 'undeploy/redeploy' on a particular WebApp, the request gets through to previous node, but once that request is forwarded on the requestor receives a 404 because their timing was poor and the request was made during the deployment window.

Can you think of any easy way to modify the system to add a 'redeploy' event? In this case, it would be nice if requests could be queued up until the deployment completes, and then have these requested posted to the newly deployed context?

A less ideal (but workable) solution would be to treat a redeployment like a shutdown event, so if either an undeployment or deployment event occurs, all requests to the engine would be blocked.

This would allow another node in the cluster to pick up the request (after failover), and the request would be handled in a timely fashion. Unfortunately, in the case of a clustered deployment, this would cause the requestors data to be shuffled around the nodes in a round-robin fashion, since eventually all of the nodes would block requests during the deployment.

Thanks for your time!

Nate
Actions

Go to original post