Getting ajp_ilink_receive_failed after a few minutes

[ndhanks]

After a few minutes, apache/mod_cluster starts to get a time out and 'does not understand error code'.

Apache

$ ../bin/apachectl -M
Loaded Modules:
core_module (static)
authn_file_module (static)
authn_default_module (static)
authz_host_module (static)
authz_groupfile_module (static)
authz_user_module (static)
authz_default_module (static)
auth_basic_module (static)
include_module (static)
filter_module (static)
log_config_module (static)
env_module (static)
setenvif_module (static)
version_module (static)
mpm_worker_module (static)
http_module (static)
mime_module (static)
status_module (static)
autoindex_module (static)
asis_module (static)
cgid_module (static)
negotiation_module (static)
dir_module (static)
actions_module (static)
alias_module (static)
rewrite_module (static)
so_module (static)
proxy_module (shared)
proxy_ajp_module (shared)
slotmem_module (shared)
manager_module (shared)
proxy_cluster_module (shared)
advertise_module (shared)
Syntax OK
[idlapp5:/usr/local/apache2.2/conf <p42dev>]
$ ../bin/apachectl -V
Server version: Apache/2.2.14 (Unix)
Server built:   Mar  4 2010 15:17:33
Server's Module Magic Number: 20051115:23
Server loaded:  APR 1.3.9, APR-Util 1.3.9
Compiled using: APR 1.3.9, APR-Util 1.3.9
Architecture:   32-bit
Server MPM:     Worker
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/worker"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/usr/local/apache2.2"
-D SUEXEC_BIN="/usr/local/apache2.2/bin/suexec"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"

Apache2.2 httpd.conf:

...

ProxyRequest off

CreateBalancers 1

...

<VirtualHost *:80>
    ServerName idlapp5.octanner.com
    ServerAdmin [e-mail]

    RewriteEngine on

    KeepAliveTimeout 60
    MaxKeepAliveRequests 1

    ServerAdvertise On http://idlapp5:80
    AdvertiseGroup 227.0.0.10:23364
    AdvertiseBindAddress 0.0.0.0:23364
    AdvertiseFrequency 5

    <Location /mod_cluster_manager>
        SetHandler mod_cluster-manager

        Order deny,allow
        Allow from all
    </Location>

    ProxyPass /error/ !
    ProxyPass /down/ !

    ProxyPass / balancer://P42DevBalancer/

</VirtualHost>

Jboss5:

2 instances on one host, ports-default and ports-01.

mod-cluster-jboss-beans.xml:

...

    <property name="proxyList">${jboss.modcluster.proxyList:}</property>

...

    <property name="stickySession">true</property>
    <property name="stickySessionForce">true</property>
    <property name="stickySessionRemove">false</property>
    <property name="maxAttempts">2</property>
    <property name="workerTimeout">5</property>

...

Jboss started with( showing only relevant options)

Instance 1:

-Djboss.service.binding.set=ports-00 -Doct.jboss.web.engine.jvmRoute=p42_4_0 -Djboss.modcluster.balancer=P42DevBalancer -c p42-0 -b idlapp4.octanner.com -g P42DevCluster -u 227.0.0.10

Instance 2:

-Djboss.service.binding.set=ports-01 -Doct.jboss.web.engine.jvmRoute=p42_4_1  -Djboss.modcluster.balancer=P42DevBalancer -c p42-1 -b  idlapp4.octanner.com -g P42DevCluster -u 227.0.0.10

http:/idlapp5.octanner.com/mod_cluster_manager:

Auto  Refresh show  DUMP output show  INFO output

Node p42_4_0 (ajp://172.29.70.86:8009):

Enable  Contexts Disable  Contexts
Balancer: P42DevBalancer,Domain: ,Flushpackets: Off,Flushwait:  10000,Ping: 10000000,Smax: 26,Ttl: 60000000,Elected: 17,Read:  123646,Transferred: 233,Connected: 0,Load: 51

Virtual Host 1:

Contexts:

/, Status: ENABLED Disable
/lineitems, Status: ENABLED Disable
/webresources, Status: ENABLED Disable

Aliases:

localhost

Node p42_4_1 (ajp://172.29.70.86:8109):

Enable  Contexts Disable  Contexts
Balancer: P42DevBalancer,Domain: ,Flushpackets: Off,Flushwait:  10000,Ping: 10000000,Smax: 26,Ttl: 60000000,Elected: 23,Read:  113791,Transferred: 207,Connected: 0,Load: 55

Virtual Host 1:

Contexts:

/, Status: ENABLED Disable
/lineitems, Status: ENABLED Disable
/webresources, Status: ENABLED Disable

Aliases:

localhost

error_log:

[Fri Mar 05 10:55:27 2010] [debug] ajp_header.c(450): ajp_marshal_into_msgb: Done
[Fri Mar 05 10:55:27 2010] [debug] mod_proxy_ajp.c(265): proxy: APR_BUCKET_IS_EOS
[Fri Mar 05 10:55:27 2010] [debug] mod_proxy_ajp.c(270): proxy: data to read (max 8186 at 4)
[Fri Mar 05 10:55:27 2010] [debug] mod_proxy_ajp.c(285): proxy: got 0 bytes of data
[Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(687): ajp_read_header: ajp_ilink_received 04
[Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(697): ajp_parse_type: got 04
[Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(516): ajp_unmarshal_response: status = 200
[Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(537): ajp_unmarshal_response: Number of headers is = 8
[Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[0] [X-Powered-By] = [Servlet 2.5; JBoss-5.0/JBossWeb-2.1]
[Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[1] [Set-Cookie] = [redirectURL=/home.seam]
[Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[2] [Cache-Control] = [no-cache]
[Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[3] [Cache-Control] = [no-store]
[Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[4] [Cache-Control] = [must-revalidate]
[Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[5] [Pragma] = [no-cache]
[Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[6] [Expires] = [0]
[Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[7] [Content-Type] = [text/html;charset=UTF-8]
[Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(609): ajp_unmarshal_response: ap_set_content_type done
[Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(687): ajp_read_header: ajp_ilink_received 03
[Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(697): ajp_parse_type: got 03
[Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(687): ajp_read_header: ajp_ilink_received 03
[Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(697): ajp_parse_type: got 03
[Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(687): ajp_read_header: ajp_ilink_received 03
[Fri Mar 05 10:55:28 2010] [debug] ajp_header.c(697): ajp_parse_type: got 03
[Fri Mar 05 10:55:29 2010] [debug] mod_manager.c(1416): manager_trans STATUS (/)
[Fri Mar 05 10:55:29 2010] [debug] mod_manager.c(1929): manager_handler STATUS (/) processing: "JVMRoute=p42_4_1&Load=61"
[Fri Mar 05 10:55:29 2010] [debug] proxy_util.c(1937): proxy: ajp: retrying the worker for (172.29.70.86)
[Fri Mar 05 10:55:29 2010] [error] proxy: ajp: disabled connection for (172.29.70.86)
[Fri Mar 05 10:55:29 2010] [debug] mod_proxy_cluster.c(1333): proxy_cluster_isup: pingpong failed
[Fri Mar 05 10:55:29 2010] [debug] mod_manager.c(1970): manager_handler STATUS  OK
[Fri Mar 05 10:55:29 2010] [debug] mod_manager.c(1416): manager_trans STATUS (/)
[Fri Mar 05 10:55:29 2010] [debug] mod_manager.c(1929): manager_handler STATUS (/) processing: "JVMRoute=p42_4_0&Load=57"
[Fri Mar 05 10:55:29 2010] [debug] proxy_util.c(1999): proxy: ajp: has acquired connection for (172.29.70.86)
[Fri Mar 05 10:55:29 2010] [debug] proxy_util.c(2055): proxy: connecting ajp://172.29.70.86:8009/ to 172.29.70.86:8009
[Fri Mar 05 10:55:29 2010] [debug] proxy_util.c(2153): proxy: connected ajp://172.29.70.86:8009/ to 172.29.70.86:8009
[Fri Mar 05 10:55:29 2010] [debug] mod_proxy_cluster.c(1194): ajp_cping_cpong: Done
[Fri Mar 05 10:55:29 2010] [debug] proxy_util.c(2017): proxy: ajp: has released connection for (172.29.70.86)
[Fri Mar 05 10:55:29 2010] [debug] mod_manager.c(1970): manager_handler STATUS  OK
[Fri Mar 05 10:55:30 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:30 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:30 2010] [debug] mod_proxy_cluster.c(1115): proxy: byrequests balancer DONE (ajp://172.29.70.86:8009)
[Fri Mar 05 10:55:30 2010] [debug] mod_proxy.c(993): Running scheme balancer handler (attempt 0)
[Fri Mar 05 10:55:30 2010] [debug] mod_proxy_ajp.c(644): proxy: AJP: serving URL ajp://172.29.70.86:8009/block/receive-updates
[Fri Mar 05 10:55:30 2010] [debug] proxy_util.c(1999): proxy: AJP: has acquired connection for (172.29.70.86)
[Fri Mar 05 10:55:30 2010] [debug] proxy_util.c(2055): proxy: connecting ajp://172.29.70.86:8009/block/receive-updates to 172.29.70.86:8009
[Fri Mar 05 10:55:30 2010] [debug] proxy_util.c(2153): proxy: connected /block/receive-updates to 172.29.70.86:8009
[Fri Mar 05 10:55:30 2010] [debug] ajp_utils.c(31): Into ajp_handle_cping_cpong
[Fri Mar 05 10:55:30 2010] [debug] ajp_utils.c(102): ajp_handle_cping_cpong: Done
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(224): Into ajp_marshal_into_msgb
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[0] [Host] = [idlapp5.octanner.com]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[1] [User-Agent] = [Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[2] [Accept] = [text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[3] [Accept-Language] = [en-us,en;q=0.7,ru;q=0.3]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[4] [Accept-Encoding] = [gzip,deflate]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[5] [Accept-Charset] = [ISO-8859-1,utf-8;q=0.7,*;q=0.7]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[6] [Keep-Alive] = [115]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[7] [Connection] = [keep-alive]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[8] [Content-Type] = [application/x-www-form-urlencoded; charset=UTF-8]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[9] [Referer] = [http://idlapp5.octanner.com/home.seam]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[10] [Content-Length] = [72]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[11] [Cookie] = [__utma=103199134.1337464088.1261668787.1261668787.1261668787.1; __utmz=103199134.1261668787.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); JSESSIONID=656BC5CC93874A4FEA42FC8042600B6F.p42_4_0; redirectURL=/home.seam; ice.sessions=eAWAVnAEtYcG2YV2tZzLOQ#1; updates=; ice.lease=1267809043455; bconn=eAWAVnAEtYcG2YV2tZzLOQ:1]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[12] [Pragma] = [no-cache]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[13] [Cache-Control] = [no-cache]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(450): ajp_marshal_into_msgb: Done
[Fri Mar 05 10:55:30 2010] [debug] mod_proxy_ajp.c(265): proxy: APR_BUCKET_IS_EOS
[Fri Mar 05 10:55:30 2010] [debug] mod_proxy_ajp.c(270): proxy: data to read (max 8186 at 4)
[Fri Mar 05 10:55:30 2010] [debug] mod_proxy_ajp.c(285): proxy: got 72 bytes of data
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(687): ajp_read_header: ajp_ilink_received 04
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(697): ajp_parse_type: got 04
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(516): ajp_unmarshal_response: status = 200
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(537): ajp_unmarshal_response: Number of headers is = 8
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[0] [X-Powered-By] = [Servlet 2.5; JBoss-5.0/JBossWeb-2.1]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[1] [Cache-Control] = [no-cache]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[2] [Cache-Control] = [no-store]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[3] [Cache-Control] = [must-revalidate]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[4] [Pragma] = [no-cache]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[5] [Expires] = [0]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[6] [Content-Type] = [text/xml;charset=UTF-8]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(609): ajp_unmarshal_response: ap_set_content_type done
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[7] [Content-Length] = [9]
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(687): ajp_read_header: ajp_ilink_received 03
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(697): ajp_parse_type: got 03
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(687): ajp_read_header: ajp_ilink_received 05
[Fri Mar 05 10:55:30 2010] [debug] ajp_header.c(697): ajp_parse_type: got 05
[Fri Mar 05 10:55:30 2010] [debug] mod_proxy_ajp.c(562): proxy: got response from 172.29.70.86:8009 (172.29.70.86)
[Fri Mar 05 10:55:30 2010] [debug] proxy_util.c(2017): proxy: AJP: has released connection for (172.29.70.86)
[Fri Mar 05 10:55:30 2010] [debug] mod_proxy_cluster.c(2311): proxy_cluster_post_request for (balancer://p42devbalancer) (null)
[Fri Mar 05 10:55:31 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:31 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:31 2010] [debug] mod_proxy_cluster.c(1115): proxy: byrequests balancer DONE (ajp://172.29.70.86:8009)
[Fri Mar 05 10:55:31 2010] [debug] mod_proxy.c(993): Running scheme balancer handler (attempt 0)
[Fri Mar 05 10:55:31 2010] [debug] mod_proxy_ajp.c(644): proxy: AJP: serving URL ajp://172.29.70.86:8009/block/receive-updated-views
[Fri Mar 05 10:55:31 2010] [debug] proxy_util.c(1999): proxy: AJP: has acquired connection for (172.29.70.86)
[Fri Mar 05 10:55:31 2010] [debug] proxy_util.c(2055): proxy: connecting ajp://172.29.70.86:8009/block/receive-updated-views to 172.29.70.86:8009
[Fri Mar 05 10:55:31 2010] [debug] proxy_util.c(2153): proxy: connected /block/receive-updated-views to 172.29.70.86:8009
[Fri Mar 05 10:55:31 2010] [debug] ajp_utils.c(31): Into ajp_handle_cping_cpong
[Fri Mar 05 10:55:31 2010] [debug] ajp_utils.c(102): ajp_handle_cping_cpong: Done
[Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(224): Into ajp_marshal_into_msgb
[Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[0] [Host] = [idlapp5.octanner.com]
[Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[1] [User-Agent] = [Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6]
[Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[2] [Accept] = [text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8]
[Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[3] [Accept-Language] = [en-us,en;q=0.7,ru;q=0.3]
[Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[4] [Accept-Encoding] = [gzip,deflate]
[Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[5] [Accept-Charset] = [ISO-8859-1,utf-8;q=0.7,*;q=0.7]
[Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[6] [Keep-Alive] = [115]
[Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[7] [Connection] = [keep-alive]
[Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[8] [Content-Type] = [application/x-www-form-urlencoded; charset=UTF-8]
[Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[9] [Referer] = [http://idlapp5.octanner.com/home.seam]
[Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[10] [Content-Length] = [60]
[Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[11] [Cookie] = [__utma=103199134.1337464088.1261668787.1261668787.1261668787.1; __utmz=103199134.1261668787.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); JSESSIONID=656BC5CC93874A4FEA42FC8042600B6F.p42_4_0; redirectURL=/home.seam; ice.sessions=eAWAVnAEtYcG2YV2tZzLOQ#1; updates=; ice.lease=1267809043455; bconn=eAWAVnAEtYcG2YV2tZzLOQ:1:acquired]
[Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[12] [Pragma] = [no-cache]
[Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[13] [Cache-Control] = [no-cache]
[Fri Mar 05 10:55:31 2010] [debug] ajp_header.c(450): ajp_marshal_into_msgb: Done
[Fri Mar 05 10:55:31 2010] [debug] mod_proxy_ajp.c(265): proxy: APR_BUCKET_IS_EOS
[Fri Mar 05 10:55:31 2010] [debug] mod_proxy_ajp.c(270): proxy: data to read (max 8186 at 4)
[Fri Mar 05 10:55:31 2010] [debug] mod_proxy_ajp.c(285): proxy: got 60 bytes of data
[Fri Mar 05 10:55:38 2010] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Fri Mar 05 10:55:38 2010] [error] ajp_read_header: ajp_ilink_receive failed
[Fri Mar 05 10:55:38 2010] [debug] mod_proxy_ajp.c(525): (120006)APR does not understand this error code: ajp_read_header failed
[Fri Mar 05 10:55:38 2010] [debug] mod_proxy_ajp.c(540): proxy: Processing of request failed backend: 1, output: 0
[Fri Mar 05 10:55:38 2010] [error] (120006)APR does not understand this error code: proxy: dialog to 172.29.70.86:8009 (172.29.70.86) failed
[Fri Mar 05 10:55:38 2010] [debug] proxy_util.c(2017): proxy: AJP: has released connection for (172.29.70.86)
[Fri Mar 05 10:55:38 2010] [debug] mod_proxy_cluster.c(2311): proxy_cluster_post_request for (balancer://p42devbalancer) (null)
[Fri Mar 05 10:55:40 2010] [debug] mod_manager.c(1416): manager_trans STATUS (/)
[Fri Mar 05 10:55:40 2010] [debug] mod_manager.c(1929): manager_handler STATUS (/) processing: "JVMRoute=p42_4_1&Load=59"
[Fri Mar 05 10:55:40 2010] [debug] proxy_util.c(1937): proxy: ajp: retrying the worker for (172.29.70.86)
[Fri Mar 05 10:55:40 2010] [error] proxy: ajp: disabled connection for (172.29.70.86)
[Fri Mar 05 10:55:40 2010] [debug] mod_proxy_cluster.c(1333): proxy_cluster_isup: pingpong failed
[Fri Mar 05 10:55:40 2010] [debug] mod_manager.c(1970): manager_handler STATUS  OK
[Fri Mar 05 10:55:40 2010] [debug] mod_manager.c(1416): manager_trans STATUS (/)
[Fri Mar 05 10:55:40 2010] [debug] mod_manager.c(1929): manager_handler STATUS (/) processing: "JVMRoute=p42_4_0&Load=58"
[Fri Mar 05 10:55:40 2010] [debug] proxy_util.c(1999): proxy: ajp: has acquired connection for (172.29.70.86)
[Fri Mar 05 10:55:40 2010] [debug] proxy_util.c(2055): proxy: connecting ajp://172.29.70.86:8009/ to 172.29.70.86:8009
[Fri Mar 05 10:55:40 2010] [debug] proxy_util.c(2153): proxy: connected ajp://172.29.70.86:8009/ to 172.29.70.86:8009
[Fri Mar 05 10:55:40 2010] [debug] mod_proxy_cluster.c(1194): ajp_cping_cpong: Done
[Fri Mar 05 10:55:40 2010] [debug] proxy_util.c(2017): proxy: ajp: has released connection for (172.29.70.86)
[Fri Mar 05 10:55:40 2010] [debug] mod_manager.c(1970): manager_handler STATUS  OK
[Fri Mar 05 10:55:41 2010] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Fri Mar 05 10:55:41 2010] [error] ajp_read_header: ajp_ilink_receive failed
[Fri Mar 05 10:55:41 2010] [error] (120006)APR does not understand this error code: proxy: read response failed from 172.29.70.86:8009 (172.29.70.86)
[Fri Mar 05 10:55:41 2010] [debug] proxy_util.c(2017): proxy: AJP: has released connection for (172.29.70.86)
[Fri Mar 05 10:55:41 2010] [debug] mod_proxy_cluster.c(2311): proxy_cluster_post_request for (balancer://p42devbalancer) (null)
[Fri Mar 05 10:55:41 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:41 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:41 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:44 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:44 2010] [debug] mod_proxy_cluster.c(1118): proxy: byrequests balancer FAILED
[Fri Mar 05 10:55:44 2010] [error] proxy: CLUSTER: (balancer://p42devbalancer). All workers are in error state
[Fri Mar 05 10:55:44 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:44 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:44 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:44 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:44 2010] [debug] mod_proxy_cluster.c(1118): proxy: byrequests balancer FAILED
[Fri Mar 05 10:55:44 2010] [error] proxy: CLUSTER: (balancer://p42devbalancer). All workers are in error state
[Fri Mar 05 10:55:44 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:44 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:44 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:46 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:46 2010] [debug] mod_proxy_cluster.c(1118): proxy: byrequests balancer FAILED
[Fri Mar 05 10:55:46 2010] [error] proxy: CLUSTER: (balancer://p42devbalancer). All workers are in error state
[Fri Mar 05 10:55:46 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:46 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:46 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:48 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:48 2010] [debug] mod_proxy_cluster.c(1118): proxy: byrequests balancer FAILED
[Fri Mar 05 10:55:48 2010] [error] proxy: CLUSTER: (balancer://p42devbalancer). All workers are in error state
[Fri Mar 05 10:55:48 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:48 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:48 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:49 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:49 2010] [debug] mod_proxy_cluster.c(1118): proxy: byrequests balancer FAILED
[Fri Mar 05 10:55:49 2010] [error] proxy: CLUSTER: (balancer://p42devbalancer). All workers are in error state
[Fri Mar 05 10:55:49 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:49 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:49 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:49 2010] [debug] mod_proxy_cluster.c(924): get_balancer_by_node found context /
[Fri Mar 05 10:55:49 2010] [debug] mod_proxy_cluster.c(1118): proxy: byrequests balancer FAILED

[jfclere]

You have increase ping (due to MODCLUSTER-132) and may be nodeTimeout.