I am trying to work out a way to have requests for a particular webapp failed over to another member of the cluster when the particular node the request being sent to, does not have the specific webapp running (but other members do). I am trying to minimize 404's when a webapp is being redeployed, or JBoss is being rebooted.
Note: I only use JBoss/Jetty for dynamic content, static content is provided by Apache on the front end.
What I have come up with so far:
1. Use a custom candidacy function for mod_backhand that validates that a node has a particular webapp deployed before the request is targeted to JBoss. For this to work I would have to come up with a way to get the available contexts from Jetty and make a server app that communicates over TCP with mod_backhand. I can't begin to gauge the extra load this would entail, but I already use a function in mod_backhand to make sure the AJP port is alive before a request is sent to a node, so it might not be too bad.
2. Intercept a request in Jetty that is going to 404 and proxy it to another node which has the context available. This would entail being able to query the other nodes for their context's and working out how to proxy. The performance hit would only come into play if the proxying was required, unlike option 1.
3. Make a 'default' context that is always deployed, grab any requests that hit it and try to redirect to a node that has the context. The default context would only be called if a more specific is not available. I fear this will fail where virutal hosts are concerned.
4. Look the other way and play dumb when people ask why they received a 404 from the site :)
What is everyone else doing?
My cluster is still in pre-production so I may be worried over nothing, your advice and experience is greatly appreciated.
I guess your problem is real and should be addressed at the servlet engine level: when a re-deploy occur, the "invocation-valve" should be put in stand-by waiting that the app is restarted. You should redirect this question on the JBossWeb/jetty mailing list or add a feature request on sf.net.