1 Reply Latest reply on Mar 8, 2010 5:04 AM by Jean-Frederic Clere

    Getting ajp_ilink_receive_failed after a few minutes

    Ned Hanks Newbie

      After a few minutes, apache/mod_cluster starts to get a time out and 'does not understand error code'.

       

      Apache

      $ ../bin/apachectl -M
      Loaded Modules:
      core_module (static)
      authn_file_module (static)
      authn_default_module (static)
      authz_host_module (static)
      authz_groupfile_module (static)
      authz_user_module (static)
      authz_default_module (static)
      auth_basic_module (static)
      include_module (static)
      filter_module (static)
      log_config_module (static)
      env_module (static)
      setenvif_module (static)
      version_module (static)
      mpm_worker_module (static)
      http_module (static)
      mime_module (static)
      status_module (static)
      autoindex_module (static)
      asis_module (static)
      cgid_module (static)
      negotiation_module (static)
      dir_module (static)
      actions_module (static)
      alias_module (static)
      rewrite_module (static)
      so_module (static)
      proxy_module (shared)
      proxy_ajp_module (shared)
      slotmem_module (shared)
      manager_module (shared)
      proxy_cluster_module (shared)
      advertise_module (shared)
      Syntax OK
      [idlapp5:/usr/local/apache2.2/conf <p42dev>]
      $ ../bin/apachectl -V
      Server version: Apache/2.2.14 (Unix)
      Server built:   Mar  4 2010 15:17:33
      Server's Module Magic Number: 20051115:23
      Server loaded:  APR 1.3.9, APR-Util 1.3.9
      Compiled using: APR 1.3.9, APR-Util 1.3.9
      Architecture:   32-bit
      Server MPM:     Worker
        threaded:     yes (fixed thread count)
          forked:     yes (variable process count)
      Server compiled with....
      -D APACHE_MPM_DIR="server/mpm/worker"
      -D APR_HAS_SENDFILE
      -D APR_HAS_MMAP
      -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
      -D APR_USE_SYSVSEM_SERIALIZE
      -D APR_USE_PTHREAD_SERIALIZE
      -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
      -D APR_HAS_OTHER_CHILD
      -D AP_HAVE_RELIABLE_PIPED_LOGS
      -D DYNAMIC_MODULE_LIMIT=128
      -D HTTPD_ROOT="/usr/local/apache2.2"
      -D SUEXEC_BIN="/usr/local/apache2.2/bin/suexec"
      -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
      -D DEFAULT_ERRORLOG="logs/error_log"
      -D AP_TYPES_CONFIG_FILE="conf/mime.types"
      -D SERVER_CONFIG_FILE="conf/httpd.conf"

       

       

      Apache2.2 httpd.conf:

      ...

      ProxyRequest off

      CreateBalancers 1

      ...

      <VirtualHost *:80>
          ServerName idlapp5.octanner.com
          ServerAdmin ned.hanks@octanner.com

       

          RewriteEngine on

       

          KeepAliveTimeout 60
          MaxKeepAliveRequests 1

       

          ServerAdvertise On http://idlapp5:80
          AdvertiseGroup 227.0.0.10:23364
          AdvertiseBindAddress 0.0.0.0:23364
          AdvertiseFrequency 5

       

          <Location /mod_cluster_manager>
              SetHandler mod_cluster-manager

       

              Order deny,allow
              Allow from all
          </Location>

       

          ProxyPass /error/ !
          ProxyPass /down/ !

       

          ProxyPass / balancer://P42DevBalancer/

       

      </VirtualHost>

       

       

      Jboss5:

      2 instances on one host, ports-default and ports-01.

       

      mod-cluster-jboss-beans.xml:

      ...

          <property name="proxyList">${jboss.modcluster.proxyList:}</property>

      ...

          <property name="stickySession">true</property>
          <property name="stickySessionForce">true</property>
          <property name="stickySessionRemove">false</property>
          <property name="maxAttempts">2</property>
          <property name="workerTimeout">5</property>

      ...

       

      Jboss started with( showing only relevant options)

      Instance 1:

      -Djboss.service.binding.set=ports-00 -Doct.jboss.web.engine.jvmRoute=p42_4_0 -Djboss.modcluster.balancer=P42DevBalancer -c p42-0 -b idlapp4.octanner.com -g P42DevCluster -u 227.0.0.10

      Instance 2:

      -Djboss.service.binding.set=ports-01 -Doct.jboss.web.engine.jvmRoute=p42_4_1  -Djboss.modcluster.balancer=P42DevBalancer -c p42-1 -b  idlapp4.octanner.com -g P42DevCluster -u 227.0.0.10

       


      http:/idlapp5.octanner.com/mod_cluster_manager:

      Auto  Refresh show  DUMP output show  INFO output

      Node p42_4_0 (ajp://172.29.70.86:8009):

      Enable  Contexts Disable  Contexts
      Balancer: P42DevBalancer,Domain: ,Flushpackets: Off,Flushwait:  10000,Ping: 10000000,Smax: 26,Ttl: 60000000,Elected: 17,Read:  123646,Transferred: 233,Connected: 0,Load: 51

      Virtual Host 1:

      Contexts:

      /, Status: ENABLED Disable
      /lineitems, Status: ENABLED Disable
      /webresources, Status: ENABLED Disable

      Aliases:

      localhost

      Node p42_4_1 (ajp://172.29.70.86:8109):

      Enable  Contexts Disable  Contexts
      Balancer: P42DevBalancer,Domain: ,Flushpackets: Off,Flushwait:  10000,Ping: 10000000,Smax: 26,Ttl: 60000000,Elected: 23,Read:  113791,Transferred: 207,Connected: 0,Load: 55

      Virtual Host 1:

      Contexts:

      /, Status: ENABLED Disable
      /lineitems, Status: ENABLED Disable
      /webresources, Status: ENABLED Disable

      Aliases:

      localhost

       

      error_log:

      [Fri Mar 05 10:55:27 2010] [debug] ajp_header.c(450): ajp_marshal_into_msgb: Done
      [Fri Mar 05 10:55:27 2010] [debug] mod_proxy_ajp.c(265): proxy: APR_BUCKET_IS_EOS
      [Fri Mar 05 10:55:27 2010] [debug] mod_proxy_ajp.c(270): proxy: data to read (max 8186 at 4)
      [Fri Mar 05 10:55:27 2010] [debug] mod_proxy_ajp.c(285): proxy: got 0 bytes of data
      [Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(687): ajp_read_header: ajp_ilink_received 04
      [Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(697): ajp_parse_type: got 04
      [Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(516): ajp_unmarshal_response: status = 200
      [Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(537): ajp_unmarshal_response: Number of headers is = 8
      [Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[0] [X-Powered-By] = [Servlet 2.5; JBoss-5.0/JBossWeb-2.1]
      [Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[1] [Set-Cookie] = [redirectURL=/home.seam]
      [Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[2] [Cache-Control] = [no-cache]
      [Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[3] [Cache-Control] = [no-store]
      [Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[4] [Cache-Control] = [must-revalidate]
      [Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[5] [Pragma] = [no-cache]
      [Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[6] [Expires] = [0]
      [Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[7] [Content-Type] = [text/html;charset=UTF-8]
      [Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(609): ajp_unmarshal_response: ap_set_content_type done
      [Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(687): ajp_read_header: ajp_ilink_received 03
      [Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(697): ajp_parse_type: got 03
      [Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(687): ajp_read_header: ajp_ilink_received 03
      [Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(697): ajp_parse_type: got 03
      [Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(687): ajp_read_header: ajp_ilink_received 03
      [Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(697): ajp_parse_type: got 03
      [Fri Mar 05 10:55:29 2010] [debug] mod_manager.c(1416): manager_trans STATUS (/)
      [Fri Mar 05 10:55:29 2010] [debug] mod_manager.c(1929): manager_handler STATUS (/) processing: "JVMRoute=p42_4_1&Load=61"
      [Fri Mar 05 10:55:29 2010] [debug] proxy_util.c(1937): proxy: ajp: retrying the worker for (172.29.70.86)
      [Fri Mar 05 10:55:29 2010] [error] proxy: ajp: disabled connection for (172.29.70.86)
      [Fri Mar 05 10:55:29 2010] [debug] mod_proxy_cluster.c(1333): proxy_cluster_isup: pingpong failed
      [Fri Mar 05 10:55:29 2010] [debug] mod_manager.c(1970): manager_handler STATUS  OK
      [Fri Mar 05 10:55:29 2010] [debug] mod_manager.c(1416): manager_trans STATUS (/)
      [Fri Mar 05 10:55:29 2010] [debug] mod_manager.c(1929): manager_handler STATUS (/) processing: "JVMRoute=p42_4_0&Load=57"
      [Fri Mar 05 10:55:29 2010] [debug] proxy_util.c(1999): proxy: ajp: has acquired connection for (172.29.70.86)
      [Fri Mar 05 10:55:29 2010] [debug] proxy_util.c(2055): proxy: connecting ajp://172.29.70.86:8009/ to 172.29.70.86:8009
      [Fri Mar 05 10:55:29 2010] [debug] proxy_util.c(2153): proxy: connected ajp://172.29.70.86:8009/ to 172.29.70.86:8009
      [Fri Mar 05 10:55:29 2010] [debug] mod_proxy_cluster.c(1194): ajp_cping_cpong: Done
      [Fri Mar 05 10:55:29 2010] [debug] proxy_util.c(2017): proxy: ajp: has released connection for (172.29.70.86)
      [Fri Mar 05 10:55:29 2010] [debug] mod_manager.c(1970): manager_handler STATUS  OK
      [Fri Mar 05 10:55:30 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:30 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:30 2010] [debug] mod_proxy_cluster.c(1115): proxy: byrequests balancer DONE (ajp://172.29.70.86:8009)
      [Fri Mar 05 10:55:30 2010] [debug] mod_proxy.c(993): Running scheme balancer handler (attempt 0)
      [Fri Mar 05 10:55:30 2010] [debug] mod_proxy_ajp.c(644): proxy: AJP: serving URL ajp://172.29.70.86:8009/block/receive-updates
      [Fri Mar 05 10:55:30 2010] [debug] proxy_util.c(1999): proxy: AJP: has acquired connection for (172.29.70.86)
      [Fri Mar 05 10:55:30 2010] [debug] proxy_util.c(2055): proxy: connecting ajp://172.29.70.86:8009/block/receive-updates to 172.29.70.86:8009
      [Fri Mar 05 10:55:30 2010] [debug] proxy_util.c(2153): proxy: connected /block/receive-updates to 172.29.70.86:8009
      [Fri Mar 05 10:55:30 2010] [debug] ajp_utils.c(31): Into ajp_handle_cping_cpong
      [Fri Mar 05 10:55:30 2010] [debug] ajp_utils.c(102): ajp_handle_cping_cpong: Done
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(224): Into ajp_marshal_into_msgb
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[0] [Host] = [idlapp5.octanner.com]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[1] [User-Agent] = [Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[2] [Accept] = [text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[3] [Accept-Language] = [en-us,en;q=0.7,ru;q=0.3]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[4] [Accept-Encoding] = [gzip,deflate]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[5] [Accept-Charset] = [ISO-8859-1,utf-8;q=0.7,*;q=0.7]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[6] [Keep-Alive] = [115]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[7] [Connection] = [keep-alive]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[8] [Content-Type] = [application/x-www-form-urlencoded; charset=UTF-8]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[9] [Referer] = [http://idlapp5.octanner.com/home.seam]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[10] [Content-Length] = [72]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[11] [Cookie] = [__utma=103199134.1337464088.1261668787.1261668787.1261668787.1; __utmz=103199134.1261668787.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); JSESSIONID=656BC5CC93874A4FEA42FC8042600B6F.p42_4_0; redirectURL=/home.seam; ice.sessions=eAWAVnAEtYcG2YV2tZzLOQ#1; updates=; ice.lease=1267809043455; bconn=eAWAVnAEtYcG2YV2tZzLOQ:1]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[12] [Pragma] = [no-cache]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[13] [Cache-Control] = [no-cache]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(450): ajp_marshal_into_msgb: Done
      [Fri Mar 05 10:55:30 2010] [debug] mod_proxy_ajp.c(265): proxy: APR_BUCKET_IS_EOS
      [Fri Mar 05 10:55:30 2010] [debug] mod_proxy_ajp.c(270): proxy: data to read (max 8186 at 4)
      [Fri Mar 05 10:55:30 2010] [debug] mod_proxy_ajp.c(285): proxy: got 72 bytes of data
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(687): ajp_read_header: ajp_ilink_received 04
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(697): ajp_parse_type: got 04
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(516): ajp_unmarshal_response: status = 200
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(537): ajp_unmarshal_response: Number of headers is = 8
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[0] [X-Powered-By] = [Servlet 2.5; JBoss-5.0/JBossWeb-2.1]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[1] [Cache-Control] = [no-cache]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[2] [Cache-Control] = [no-store]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[3] [Cache-Control] = [must-revalidate]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[4] [Pragma] = [no-cache]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[5] [Expires] = [0]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[6] [Content-Type] = [text/xml;charset=UTF-8]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(609): ajp_unmarshal_response: ap_set_content_type done
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[7] [Content-Length] = [9]
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(687): ajp_read_header: ajp_ilink_received 03
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(697): ajp_parse_type: got 03
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(687): ajp_read_header: ajp_ilink_received 05
      [Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(697): ajp_parse_type: got 05
      [Fri Mar 05 10:55:30 2010] [debug] mod_proxy_ajp.c(562): proxy: got response from 172.29.70.86:8009 (172.29.70.86)
      [Fri Mar 05 10:55:30 2010] [debug] proxy_util.c(2017): proxy: AJP: has released connection for (172.29.70.86)
      [Fri Mar 05 10:55:30 2010] [debug] mod_proxy_cluster.c(2311): proxy_cluster_post_request for (balancer://p42devbalancer) (null)
      [Fri Mar 05 10:55:31 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:31 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:31 2010] [debug] mod_proxy_cluster.c(1115): proxy: byrequests balancer DONE (ajp://172.29.70.86:8009)
      [Fri Mar 05 10:55:31 2010] [debug] mod_proxy.c(993): Running scheme balancer handler (attempt 0)
      [Fri Mar 05 10:55:31 2010] [debug] mod_proxy_ajp.c(644): proxy: AJP: serving URL ajp://172.29.70.86:8009/block/receive-updated-views
      [Fri Mar 05 10:55:31 2010] [debug] proxy_util.c(1999): proxy: AJP: has acquired connection for (172.29.70.86)
      [Fri Mar 05 10:55:31 2010] [debug] proxy_util.c(2055): proxy: connecting ajp://172.29.70.86:8009/block/receive-updated-views to 172.29.70.86:8009
      [Fri Mar 05 10:55:31 2010] [debug] proxy_util.c(2153): proxy: connected /block/receive-updated-views to 172.29.70.86:8009
      [Fri Mar 05 10:55:31 2010] [debug] ajp_utils.c(31): Into ajp_handle_cping_cpong
      [Fri Mar 05 10:55:31 2010] [debug] ajp_utils.c(102): ajp_handle_cping_cpong: Done
      [Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(224): Into ajp_marshal_into_msgb
      [Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[0] [Host] = [idlapp5.octanner.com]
      [Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[1] [User-Agent] = [Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6]
      [Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[2] [Accept] = [text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8]
      [Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[3] [Accept-Language] = [en-us,en;q=0.7,ru;q=0.3]
      [Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[4] [Accept-Encoding] = [gzip,deflate]
      [Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[5] [Accept-Charset] = [ISO-8859-1,utf-8;q=0.7,*;q=0.7]
      [Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[6] [Keep-Alive] = [115]
      [Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[7] [Connection] = [keep-alive]
      [Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[8] [Content-Type] = [application/x-www-form-urlencoded; charset=UTF-8]
      [Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[9] [Referer] = [http://idlapp5.octanner.com/home.seam]
      [Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[10] [Content-Length] = [60]
      [Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[11] [Cookie] = [__utma=103199134.1337464088.1261668787.1261668787.1261668787.1; __utmz=103199134.1261668787.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); JSESSIONID=656BC5CC93874A4FEA42FC8042600B6F.p42_4_0; redirectURL=/home.seam; ice.sessions=eAWAVnAEtYcG2YV2tZzLOQ#1; updates=; ice.lease=1267809043455; bconn=eAWAVnAEtYcG2YV2tZzLOQ:1:acquired]
      [Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[12] [Pragma] = [no-cache]
      [Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[13] [Cache-Control] = [no-cache]
      [Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(450): ajp_marshal_into_msgb: Done
      [Fri Mar 05 10:55:31 2010] [debug] mod_proxy_ajp.c(265): proxy: APR_BUCKET_IS_EOS
      [Fri Mar 05 10:55:31 2010] [debug] mod_proxy_ajp.c(270): proxy: data to read (max 8186 at 4)
      [Fri Mar 05 10:55:31 2010] [debug] mod_proxy_ajp.c(285): proxy: got 60 bytes of data
      [Fri Mar 05 10:55:38 2010] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
      [Fri Mar 05 10:55:38 2010] [error] ajp_read_header: ajp_ilink_receive failed
      [Fri Mar 05 10:55:38 2010] [debug] mod_proxy_ajp.c(525): (120006)APR does not understand this error code: ajp_read_header failed
      [Fri Mar 05 10:55:38 2010] [debug] mod_proxy_ajp.c(540): proxy: Processing of request failed backend: 1, output: 0
      [Fri Mar 05 10:55:38 2010] [error] (120006)APR does not understand this error code: proxy: dialog to 172.29.70.86:8009 (172.29.70.86) failed
      [Fri Mar 05 10:55:38 2010] [debug] proxy_util.c(2017): proxy: AJP: has released connection for (172.29.70.86)
      [Fri Mar 05 10:55:38 2010] [debug] mod_proxy_cluster.c(2311): proxy_cluster_post_request for (balancer://p42devbalancer) (null)
      [Fri Mar 05 10:55:40 2010] [debug] mod_manager.c(1416): manager_trans STATUS (/)
      [Fri Mar 05 10:55:40 2010] [debug] mod_manager.c(1929): manager_handler STATUS (/) processing: "JVMRoute=p42_4_1&Load=59"
      [Fri Mar 05 10:55:40 2010] [debug] proxy_util.c(1937): proxy: ajp: retrying the worker for (172.29.70.86)
      [Fri Mar 05 10:55:40 2010] [error] proxy: ajp: disabled connection for (172.29.70.86)
      [Fri Mar 05 10:55:40 2010] [debug] mod_proxy_cluster.c(1333): proxy_cluster_isup: pingpong failed
      [Fri Mar 05 10:55:40 2010] [debug] mod_manager.c(1970): manager_handler STATUS  OK
      [Fri Mar 05 10:55:40 2010] [debug] mod_manager.c(1416): manager_trans STATUS (/)
      [Fri Mar 05 10:55:40 2010] [debug] mod_manager.c(1929): manager_handler STATUS (/) processing: "JVMRoute=p42_4_0&Load=58"
      [Fri Mar 05 10:55:40 2010] [debug] proxy_util.c(1999): proxy: ajp: has acquired connection for (172.29.70.86)
      [Fri Mar 05 10:55:40 2010] [debug] proxy_util.c(2055): proxy: connecting ajp://172.29.70.86:8009/ to 172.29.70.86:8009
      [Fri Mar 05 10:55:40 2010] [debug] proxy_util.c(2153): proxy: connected ajp://172.29.70.86:8009/ to 172.29.70.86:8009
      [Fri Mar 05 10:55:40 2010] [debug] mod_proxy_cluster.c(1194): ajp_cping_cpong: Done
      [Fri Mar 05 10:55:40 2010] [debug] proxy_util.c(2017): proxy: ajp: has released connection for (172.29.70.86)
      [Fri Mar 05 10:55:40 2010] [debug] mod_manager.c(1970): manager_handler STATUS  OK
      [Fri Mar 05 10:55:41 2010] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
      [Fri Mar 05 10:55:41 2010] [error] ajp_read_header: ajp_ilink_receive failed
      [Fri Mar 05 10:55:41 2010] [error] (120006)APR does not understand this error code: proxy: read response failed from 172.29.70.86:8009 (172.29.70.86)

      [Fri Mar 05 10:55:41 2010] [debug] proxy_util.c(2017): proxy: AJP: has released connection for (172.29.70.86)
      [Fri Mar 05 10:55:41 2010] [debug] mod_proxy_cluster.c(2311): proxy_cluster_post_request for (balancer://p42devbalancer) (null)
      [Fri Mar 05 10:55:41 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:41 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:41 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:44 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:44 2010] [debug] mod_proxy_cluster.c(1118): proxy: byrequests balancer FAILED
      [Fri Mar 05 10:55:44 2010] [error] proxy: CLUSTER: (balancer://p42devbalancer). All workers are in error state
      [Fri Mar 05 10:55:44 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:44 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:44 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:44 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:44 2010] [debug] mod_proxy_cluster.c(1118): proxy: byrequests balancer FAILED
      [Fri Mar 05 10:55:44 2010] [error] proxy: CLUSTER: (balancer://p42devbalancer). All workers are in error state
      [Fri Mar 05 10:55:44 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:44 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:44 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:46 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:46 2010] [debug] mod_proxy_cluster.c(1118): proxy: byrequests balancer FAILED
      [Fri Mar 05 10:55:46 2010] [error] proxy: CLUSTER: (balancer://p42devbalancer). All workers are in error state
      [Fri Mar 05 10:55:46 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:46 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:46 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:48 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:48 2010] [debug] mod_proxy_cluster.c(1118): proxy: byrequests balancer FAILED
      [Fri Mar 05 10:55:48 2010] [error] proxy: CLUSTER: (balancer://p42devbalancer). All workers are in error state
      [Fri Mar 05 10:55:48 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:48 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:48 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:49 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:49 2010] [debug] mod_proxy_cluster.c(1118): proxy: byrequests balancer FAILED
      [Fri Mar 05 10:55:49 2010] [error] proxy: CLUSTER: (balancer://p42devbalancer). All workers are in error state
      [Fri Mar 05 10:55:49 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:49 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:49 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:49 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
      [Fri Mar 05 10:55:49 2010] [debug] mod_proxy_cluster.c(1118): proxy: byrequests balancer FAILED