server hang during shutdown when specifying executor in connector
matlach Oct 17, 2012 10:21 AMHello everyone,
Since I've defined an executor to the connector element in the standalone.xml, my jboss as 7.1.1 server hang during shutdown.
Here is my executor configuration :
<subsystem xmlns="urn:jboss:domain:threads:1.1"> <thread-factory name="ConnectorThreadFactory" group-name="ConnectorThreadWorker" thread-name-pattern="%G-%t" priority="5"/> <unbounded-queue-thread-pool name="ConnectorThreadPool"> <max-threads count="100"/> <keepalive-time time="60" unit="seconds"/> <thread-factory name="ConnectorThreadFactory"/> </unbounded-queue-thread-pool> </subsystem>
and my connector configuration :
<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="false"> <connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp" redirect-port="8443" executor="ConnectorThreadPool"/> <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true" executor="ConnectorThreadPool"> <ssl name="https" key-alias="dev-keystore" password="XXX" certificate-key-file="../standalone/configuration/dev.keystore" protocol="TLSv1"/> </connector> <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" executor="ConnectorThreadPool"/> ... </subsystem>
after few investigations i.e. performing a thread dump after initiating shutdown I have found that few of the connector threads are not shutdown and I guess since they are not flagged as 'daemon' the application just stays alive.
"ConnectorThreadWorker-28" prio=6 tid=0x000000001cbc2000 nid=0x15e0 runnable [0x0000000020c2f000] java.lang.Thread.State: RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:1131) at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:1213) at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:451) at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:452) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:519) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) at org.jboss.threads.JBossThread.run(JBossThread.java:122) Locked ownable synchronizers: - <0x00000000cceaa7d0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
then I thought that was maybe related on how I was shutting down the server,
on my windows environment :
- I'm starting jboss with the standalone.bat script,
- then I access any webservice or rest service (through apache) defined in my deployed ear to initialize my defined executor,
- and just press ctrl+c to initiate shutdown.
on my linux environment :
- I'm starting jboss with the standalone.sh script,
- then like on windows I access any webservice or rest service
- and to initiate shutdown, I undeploy my ear, and then shutdown jboss via the cli interface.
nothing exotic here.
then after further investigation, by removing apache from the equation, I have found that the threads that were hanging was those related to the ajp connector.
here is my apache worker.properties configuration :
worker.status.type=status worker.template.type=ajp13 worker.template.lbfactor=1 worker.template.socket_connect_timeout = 5000 worker.template.max_packet_size = 65536 worker.template.socket_keepalive = True worker.template_llb.type = lb worker.template_llb.method = B worker.template_llb.sticky_session=1 worker.XXX01.reference=worker.template worker.XXX01.port=8009 worker.XXX01.host=XXX.61 worker.XXX02.reference=worker.template worker.XXX02.port=8009 worker.XXX02.host=XXX.62 worker.YYY.reference=worker.template_llb worker.YYY.balance_workers = XXX01,XXX02 worker.list = status,YYY
though even if apache was not configured properly, it is still not normal that server refuse to shutdown because of any 'client' application.
then I decided to add a breakpoint into java.util.concurrent.ThreadPoolExecutor::shutdown() and shutdownNow();
I've looked up at the thread factory thread group name to make sure either shutdown() or shutdownNow() was called upon my defined thread pool ; yep shutdown() is called via the ManagedJBossThreadPoolExecutorService::internalShutdown().
By looking at the source, we can see :
public class ManagedJBossThreadPoolExecutorService extends ManagedExecutorService implements BlockingExecutor { private final JBossThreadPoolExecutor executor; public ManagedJBossThreadPoolExecutorService(JBossThreadPoolExecutor executor) { super(executor); this.executor = executor; } void internalShutdown() { this.executor.shutdown(); }
though there's not any mechanism to call shutdownNow() if shutdown() take too long. like the following code would :
void internalShutdown() { this.executor.shutdown(); try { this.executor.awaitTerminaison(30L, TimeUnit.SECONDS); } catch (InterruptedException e) { this.executor.shutdownNow(); } }
I don't know if this is the expected behavior, though for a defined executor on a connector I'm not sure at all; I don't think it make sense to block indefinitly the server shutdown process because of any connected client.
Anyone have any idea how to fix this ? Is my worker.properties wrong ? Is my standalone.xml configuration wrong ? Is not calling shutdownNow() for connector executor wrong ?
Any help would be greatly appreciated,
Big thanks !