I'm assuming that the following line is the real problem and it may have hung here.
12:30:11,064 INFO [Engine] StandardWrapper[/emsweb:default]: Waiting for 1 instance(s) to be deallocated
These lines which came 9 seconds later would be browser requests that came in, but the app was in the process of being undeployed...
12:30:20,364 INFO [Engine] StandardHost[localhost]: MAPPING configuration error for request URI
12:30:20,374 INFO [Engine] StandardHost[localhost]: MAPPING configuration error for request URI
Could requests coming in while an app is undeploying cause problems, or did it hang while undeploying?
It is possible that there is activity in the web application that is holding up undeployment - perhaps execution in a web application class. I'm not sure what would happen in such a case. I don't think there is a way to come back and try undeployment again on a failure.
We use "kill <jboss-pid>" to stop JBoss/Tomcat. Is this a safe way to shutdown? Does the shutdown.sh script method allow cleanup to occur?
A standard kill signal should cause JBoss to shutdown in the standard manner. You can verify this by checking the server.log output.
Yes the server.log looks fine. I compaired a kill vs a shutdown.sh and the only difference is that the shutdown.sh has 2 initial lines indicating that shtdown has started.
Another symptom of this hang during the undeployment is that it only seems to happen when JBoss has maxed out on its memory allocation. We start JBoss with 256M, and ps indicates that its consuming 300M. We may need to bump up the allocation.
The memory footprint shown by Linux is different to the heap space allocated. The VSZ/RSS includes the memory consumed by the JVM and code whereas the heap space set for your JVM (-Xmx..m)is for the storage of objects and simple variables.
You can turn on verbose:gc (in Sun Java) and get console output indicating heap memory (among other things) after GC operations. This does affect performance but is sometimes useful for test purposes. You might want to try that. Usually total use of heap memory will be signalled by "out of memory" messages or similar.