6 Replies Latest reply on Dec 10, 2012 10:56 PM by jaikiran

    server hang during shutdown when specifying executor in connector

    matlach

      Hello everyone,

       

      Since I've defined an executor to the connector element in the standalone.xml, my jboss as 7.1.1 server hang during shutdown.

       

      Here is my executor configuration :

       

      <subsystem xmlns="urn:jboss:domain:threads:1.1">
          <thread-factory name="ConnectorThreadFactory" group-name="ConnectorThreadWorker" thread-name-pattern="%G-%t" priority="5"/>
          <unbounded-queue-thread-pool name="ConnectorThreadPool">
              <max-threads count="100"/>
              <keepalive-time time="60" unit="seconds"/>
              <thread-factory name="ConnectorThreadFactory"/>
          </unbounded-queue-thread-pool>
      </subsystem>
      
      

       

      and my connector configuration :

       

      <subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="false">
          <connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp" redirect-port="8443" executor="ConnectorThreadPool"/>
          <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true" executor="ConnectorThreadPool">
              <ssl name="https" key-alias="dev-keystore" password="XXX" certificate-key-file="../standalone/configuration/dev.keystore" protocol="TLSv1"/>
          </connector>
          <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" executor="ConnectorThreadPool"/>
          ...
      </subsystem>
      
      

       

      after few investigations i.e. performing a thread dump after initiating shutdown I have found that few of the connector threads are not shutdown and I guess since they are not flagged as 'daemon' the application just stays alive.

       

      "ConnectorThreadWorker-28" prio=6 tid=0x000000001cbc2000 nid=0x15e0 runnable [0x0000000020c2f000]
         java.lang.Thread.State: RUNNABLE
                at java.net.SocketInputStream.socketRead0(Native Method)
                at java.net.SocketInputStream.read(SocketInputStream.java:129)
                at org.apache.coyote.ajp.AjpProcessor.read(AjpProcessor.java:1131)
                at org.apache.coyote.ajp.AjpProcessor.readMessage(AjpProcessor.java:1213)
                at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:451)
                at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:452)
                at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:519)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:662)
                at org.jboss.threads.JBossThread.run(JBossThread.java:122)
      
      
         Locked ownable synchronizers:
                - <0x00000000cceaa7d0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
      
      

       

      then I thought that was maybe related on how I was shutting down the server,

      on my windows environment :

      1. I'm starting jboss with the standalone.bat script,
      2. then I access any webservice or rest service (through apache) defined in my deployed ear to initialize my defined executor,
      3. and just press ctrl+c to initiate shutdown.

      on my linux environment :

      1. I'm starting jboss with the standalone.sh script,
      2. then like on windows I access any webservice or rest service
      3. and to initiate shutdown, I undeploy my ear, and then shutdown jboss via the cli interface.

       

      nothing exotic here.

       

      then after further investigation, by removing apache from the equation, I have found that the threads that were hanging was those related to the ajp connector.

       

      here is my apache worker.properties configuration :

       

      worker.status.type=status
      worker.template.type=ajp13
      worker.template.lbfactor=1
      worker.template.socket_connect_timeout = 5000
      worker.template.max_packet_size = 65536
      worker.template.socket_keepalive = True
      worker.template_llb.type = lb
      worker.template_llb.method = B
      worker.template_llb.sticky_session=1
      
      worker.XXX01.reference=worker.template
      worker.XXX01.port=8009
      worker.XXX01.host=XXX.61
      worker.XXX02.reference=worker.template
      worker.XXX02.port=8009
      worker.XXX02.host=XXX.62
      worker.YYY.reference=worker.template_llb
      worker.YYY.balance_workers = XXX01,XXX02
      
      worker.list = status,YYY
      
      

       

      though even if apache was not configured properly, it is still not normal that server refuse to shutdown because of any 'client' application.

       

      then I decided to add a breakpoint into java.util.concurrent.ThreadPoolExecutor::shutdown() and shutdownNow();

      I've looked up at the thread factory thread group name to make sure either shutdown() or shutdownNow() was called upon my defined thread pool ; yep shutdown() is called via the ManagedJBossThreadPoolExecutorService::internalShutdown().

       

      By looking at the source, we can see :

       

      public class ManagedJBossThreadPoolExecutorService extends ManagedExecutorService
        implements BlockingExecutor
      {
        private final JBossThreadPoolExecutor executor;
      
        public ManagedJBossThreadPoolExecutorService(JBossThreadPoolExecutor executor)
        {
          super(executor);
          this.executor = executor;
        }
      
        void internalShutdown()
        {
          this.executor.shutdown();
        }
      
      

       

      though there's not any mechanism to call shutdownNow() if shutdown() take too long. like the following code would :

       

        void internalShutdown()
        {
          this.executor.shutdown();
          try
          {
              this.executor.awaitTerminaison(30L, TimeUnit.SECONDS);
          }
          catch (InterruptedException e)
          {
              this.executor.shutdownNow();
          }
        }
      
      

       

      I don't know if this is the expected behavior, though for a defined executor on a connector I'm not sure at all; I don't think it make sense to block indefinitly the server shutdown process because of any connected client.

       

      Anyone have any idea how to fix this ? Is my worker.properties wrong ? Is my standalone.xml configuration wrong ? Is not calling shutdownNow() for connector executor wrong ?

       

      Any help would be greatly appreciated,

       

      Big thanks !