1 Reply Latest reply on Apr 3, 2013 4:34 AM by jfclere

    503 for all 4 webservers

    muthukumaran_m
    muthukumaran_m Mar 27, 2013 11:53 PM

      Hi,

       

      Versions:

                - 4 webservers running Apache 2.2.15

                - 3 app servers running Jboss EAP 5.1.0

                

      We have a very strange issue that happens in our production servers. Suddenly one fine day we recieve 503,this issue happens at random intervals.

       

      Only way we had fixed this is to restart the apache servers and issue resloves, this happened multiple times in the past months.

       

      When this issue happened I verified the following things

      - I was able to access our application going directly to jboss 8080 port

      - There was very minimal load on the Jboss servers

       

      During this time frame I see these error in the apache server logs. Please help me with any suggestions.

       

      -------------------------------------------------------------------------------------------------------

      [Wed Mar 27 18:29:23 2013] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header

      [Wed Mar 27 18:29:23 2013] [error] ajp_read_header: ajp_ilink_receive failed

      [Wed Mar 27 18:29:23 2013] [error] (120006)APR does not understand this error code: proxy: read response failed from 10.10.10.155:9009 (10.10.10.155)

      [Wed Mar 27 18:29:24 2013] [error] [client 64.214.226.150] proxy: error processing end

      [Wed Mar 27 18:29:30 2013] [error] proxy: ajp: disabled connection for (10.10.10.155)

      [Wed Mar 27 18:39:45 2013] [error] ajp_cping_cpong: apr_socket_recv failed

      [Wed Mar 27 19:25:17 2013] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header

      [Wed Mar 27 19:25:17 2013] [error] ajp_read_header: ajp_ilink_receive failed

      [Wed Mar 27 19:25:17 2013] [error] (120006)APR does not understand this error code: proxy: read response failed from 10.10.10.151:9009 (10.10.10.151)

      [Wed Mar 27 19:25:18 2013] [error] [client 204.2.199.189] proxy: error processing end

      [Wed Mar 27 19:28:45 2013] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header

      [Wed Mar 27 19:28:45 2013] [error] ajp_read_header: ajp_ilink_receive failed

      [Wed Mar 27 19:28:45 2013] [error] (120006)APR does not understand this error code: proxy: read response failed from 10.10.10.155:9009 (10.10.10.155)

      [Wed Mar 27 19:28:45 2013] [error] [client 204.2.199.79] proxy: error processing end

      [Wed Mar 27 19:30:34 2013] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header

      [Wed Mar 27 19:30:34 2013] [error] ajp_read_header: ajp_ilink_receive failed

      [Wed Mar 27 19:30:34 2013] [error] (120006)APR does not understand this error code: proxy: read response failed from 10.10.10.153:9009 (10.10.10.153)

      [Wed Mar 27 19:30:37 2013] [error] proxy: ajp: disabled connection for (10.10.10.153)

      [Wed Mar 27 19:33:02 2013] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header

      [Wed Mar 27 19:33:02 2013] [error] ajp_read_header: ajp_ilink_receive failed

      [Wed Mar 27 19:33:02 2013] [error] (120006)APR does not understand this error code: proxy: read response failed from 10.10.10.153:9009 (10.10.10.153)

      [Wed Mar 27 19:33:08 2013] [error] proxy: ajp: disabled connection for (10.10.10.153)

      [Wed Mar 27 19:33:13 2013] [error] proxy: ajp: disabled connection for (10.10.10.153)

      [Wed Mar 27 19:33:19 2013] [error] proxy: ajp: disabled connection for (10.10.10.153)

      [Wed Mar 27 19:43:31 2013] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header

      [Wed Mar 27 19:43:31 2013] [error] ajp_read_header: ajp_ilink_receive failed

      [Wed Mar 27 19:43:31 2013] [error] (120006)APR does not understand this error code: proxy: read response failed from 10.10.10.155:9009 (10.10.10.155)

      [Wed Mar 27 19:43:31 2013] [error] [client 204.93.33.45] proxy: error processing end

      [Wed Mar 27 19:49:20 2013] [error] ajp_cping_cpong: apr_socket_recv failed

      [Wed Mar 27 19:49:20 2013] [error] ajp_cping_cpong: apr_socket_recv failed

      [Wed Mar 27 19:49:30 2013] [error] ajp_cping_cpong: apr_socket_recv failed

      [Wed Mar 27 19:49:31 2013] [error] (70007)The timeout specified has expired: proxy: ajp: attempt to connect to 10.10.10.151:9009 (10.10.10.151) failed

      [Wed Mar 27 19:49:31 2013] [error] ap_proxy_connect_backend disabling worker for (10.10.10.151)

      [Wed Mar 27 19:49:35 2013] [error] ajp_cping_cpong: apr_socket_recv failed

      [Wed Mar 27 19:49:36 2013] [error] proxy: ajp: disabled connection for (10.10.10.151)

      [Wed Mar 27 19:49:36 2013] [error] ajp_cping_cpong: apr_socket_recv failed

      [Wed Mar 27 19:49:40 2013] [error] (104)Connection reset by peer: ajp_ilink_receive() can't receive header

      [Wed Mar 27 19:49:40 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed

      [Wed Mar 27 19:49:40 2013] [error] (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to 10.10.10.151:9009 (10.10.10.151)

      [Wed Mar 27 19:50:14 2013] [error] proxy: CLUSTER: (balancer://test). All workers are in error state

      -----------------------------------------------------------------------------------------------------------------------------------------------------------

       

      Thanks,

      Muthu