1 2 3 Previous Next 32 Replies Latest reply on Apr 11, 2014 3:27 PM by kevgo

    (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to

    okelet

      Hi

       

      I have the following environment:

       

      • CentOS 6.0 x86_64
      • Apache 2.2.15 compiled from source using this tutorial,as advised in this post
      • JBoss 7.1.0
      • mod_cluster 1.2.6 (native and jars)
      • 2 nodes, each one with Apache and JBoss

       

      The problem is that I get continously these type of errors:

       

      [Wed Sep 25 13:04:57 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed
      [Wed Sep 25 13:04:57 2013] [error] (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to (null) (__IP1__)
      [Wed Sep 25 13:04:57 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed
      [Wed Sep 25 13:04:57 2013] [error] (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to (null) (__IP1__)
      [Wed Sep 25 13:04:57 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed
      [Wed Sep 25 13:04:57 2013] [error] (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to __IP1__:8309 (__IP1__)
      [Wed Sep 25 13:04:57 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed
      [Wed Sep 25 13:04:57 2013] [error] (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to __IP1__:8309 (__IP1__)
      [Wed Sep 25 13:04:58 2013] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
      [Wed Sep 25 13:04:58 2013] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
      [Wed Sep 25 13:04:58 2013] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
      
      

       

      To isolate the problems I have left just 1 node (I have powered off the other one), but the problem continues (__IP1__ is the IP of node 1). That errors makes to get continous 503 errors in Apache to the users. Any advice to continue debugging this error? Regards and thanks in advance.

       

      These are my configuration files:

       

      vhosts-000-defaults.conf

       

      LoadModule slotmem_module modules/mod_slotmem.so
      LoadModule manager_module modules/mod_manager.so
      LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
      LoadModule advertise_module modules/mod_advertise.so
      
      UseCanonicalName Off
      ServerAdmin sistemas.sscc@domain.es
      ServerSignature Off
      ServerTokens Minor
      
      NameVirtualHost *:80
      
      <VirtualHost *:80>
          ErrorLog /var/log/httpd/error_log
          CustomLog /var/log/httpd/access_log combined
          ProxyPass / balancer://mycluster/ stickysession=JSESSIONID|jsessionid nofailover=On
          ProxyPassReverse / balancer://mycluster/
      </VirtualHost>
      
      <VirtualHost *:443>
          SSLEngine On
          ErrorLog /var/log/httpd/error-ssl_log
          CustomLog /var/log/httpd/access-ssl_log combined
          RewriteEngine On
          RewriteRule ^/$ http://www.intranet.es [R,L]
      </VirtualHost>
      
      

      vhosts-010-admin.conf

       

      Listen 8880
      NameVirtualHost *:8880
      
      <VirtualHost *:8880>
      
      
          ErrorLog /var/log/httpd/error-admin_log
          CustomLog /var/log/httpd/access-admin_log combined
      
          ServerAdvertise On
          EnableMCPMReceive
          AllowDisplay On
      
          RewriteRule ^/mcm$ /mcm/ [R,L]
          <Location "/mcm/">
              SetHandler mod_cluster-manager
              Order allow,deny
              Allow from all
          </Location>
      
      </VirtualHost>
      
      

      vhosts-050-myapp.conf

       

      <VirtualHost *:80>
          ServerName myapp.px.domain.es
          ErrorLog /var/log/httpd/error-myapp_log
          CustomLog /var/log/httpd/access-myapp_log combined
          RewriteEngine On
          RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [L,R]
      </VirtualHost>
      
      <VirtualHost *:443>
          SSLEngine On
          ServerName myapp.px.sacyl.es
          ErrorLog /var/log/httpd/error-myapp_ssl_log
          CustomLog /var/log/httpd/access-myapp_ssl_log combined
          RewriteEngine On
          RewriteRule ^/$ /myapp [R,L]
          RewriteRule ^/myapp$ /myapp/ [R,L]
          ProxyPass /vere/ balancer://mycluster/vere/ stickysession=JSESSIONID|jsessionid
          ProxyPassReverse /vere/ balancer://mycluster/vere/
      </VirtualHost>
      
      
        • 1. Re: (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to
          jfclere

          LogLevel debug will probably helps.

           

          It looks like AJP connection between httpd and Jboss has problems... Firewall?

          • 2. Re: Re: (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to
            okelet

            Hi

             

            Thanks for the response. Firewall is disabled in all hosts, and also selinux. After setting loglevel to debug in Apache, in the virtual host that serves the app, these messages are written in the log (an extract):

             

            [Thu Sep 26 10:34:08 2013] [error] (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to __IP1__:8309 (__IP1__)
            [Thu Sep 26 10:34:08 2013] [debug] proxy_util.c(2036): proxy: AJP: has released connection for (__IP1__)
            [Thu Sep 26 10:34:08 2013] [debug] mod_proxy_cluster.c(3157): cluster: Using route 506a3aed-21d1-3ee9-a833-562c00c3e42a
            [Thu Sep 26 10:34:08 2013] [debug] proxy_util.c(1956): proxy: BALANCER: retrying the worker for (__IP1__)
            [Thu Sep 26 10:34:08 2013] [debug] mod_proxy_cluster.c(2231): proxy: byrequests balancer FAILED
            [Thu Sep 26 10:34:09 2013] [debug] mod_proxy_cluster.c(2231): proxy: byrequests balancer FAILED
            [Thu Sep 26 10:34:09 2013] [debug] mod_proxy_cluster.c(2231): proxy: byrequests balancer FAILED
            [Thu Sep 26 10:34:10 2013] [debug] mod_proxy_cluster.c(2231): proxy: byrequests balancer FAILED
            [Thu Sep 26 10:34:11 2013] [debug] mod_proxy_cluster.c(2231): proxy: byrequests balancer FAILED
            [Thu Sep 26 10:34:11 2013] [debug] mod_proxy_cluster.c(2231): proxy: byrequests balancer FAILED
            [Thu Sep 26 10:34:12 2013] [debug] mod_proxy_cluster.c(2231): proxy: byrequests balancer FAILED
            [Thu Sep 26 10:34:12 2013] [debug] mod_proxy_cluster.c(2231): proxy: byrequests balancer FAILED
            [Thu Sep 26 10:34:13 2013] [debug] mod_proxy_cluster.c(2231): proxy: byrequests balancer FAILED
            [Thu Sep 26 10:34:14 2013] [debug] mod_proxy_cluster.c(2231): proxy: byrequests balancer FAILED
            [Thu Sep 26 10:34:14 2013] [debug] mod_proxy_cluster.c(2231): proxy: byrequests balancer FAILED
            [Thu Sep 26 10:34:15 2013] [debug] mod_proxy_cluster.c(2231): proxy: byrequests balancer FAILED
            [Thu Sep 26 10:34:15 2013] [debug] mod_proxy_cluster.c(2221): proxy: byrequests balancer DONE (ajp://__IP1__:8309)
            [Thu Sep 26 10:34:15 2013] [debug] mod_proxy.c(1036): Running scheme balancer handler (attempt 1)
            [Thu Sep 26 10:34:15 2013] [debug] mod_proxy_http.c(1964): proxy: HTTP: declining URL ajp://__IP1__:8309/myapp/ptesAutorizarDisplayAction.action
            [Thu Sep 26 10:34:15 2013] [debug] mod_proxy_ajp.c(726): proxy: AJP: serving URL ajp://__IP1__:8309/myapp/ptesAutorizarDisplayAction.action
            [Thu Sep 26 10:34:15 2013] [debug] proxy_util.c(2018): proxy: AJP: has acquired connection for (__IP1__)
            [Thu Sep 26 10:34:15 2013] [debug] proxy_util.c(2074): proxy: connecting ajp://__IP1__:8309/myapp/ptesAutorizarDisplayAction.action to __IP1__:8309
            [Thu Sep 26 10:34:15 2013] [debug] proxy_util.c(2200): proxy: connected /myapp/ptesAutorizarDisplayAction.action to __IP1__:8309
            [Thu Sep 26 10:34:15 2013] [debug] proxy_util.c(2451): proxy: AJP: fam 2 socket created to connect to __IP1__
            [Thu Sep 26 10:34:15 2013] [debug] ajp_utils.c(31): Into ajp_handle_cping_cpong
            [Thu Sep 26 10:34:15 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed
            [Thu Sep 26 10:34:15 2013] [debug] ajp_utils.c(102): ajp_handle_cping_cpong: Done
            [Thu Sep 26 10:34:15 2013] [error] (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to __IP1__:8309 (__IP1__)
            [Thu Sep 26 10:34:15 2013] [debug] proxy_util.c(2074): proxy: connecting ajp://__IP1__:8309/myapp/ptesAutorizarDisplayAction.action to __IP1__:8309
            [Thu Sep 26 10:34:15 2013] [debug] proxy_util.c(2200): proxy: connected /vere/ptesAutorizarDisplayAction.action to __IP1__:8309
            [Thu Sep 26 10:34:15 2013] [debug] proxy_util.c(2451): proxy: AJP: fam 2 socket created to connect to __IP1__
            [Thu Sep 26 10:34:15 2013] [debug] ajp_utils.c(31): Into ajp_handle_cping_cpong
            

             

            I don't know if this can help, but I can't see any strange in the logs that could say something about the errors... Stuck :S.

             

            Regards and thanks in advance.

            • 3. Re: Re: (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to
              erasmomarciano

              Hi

              You are using mod_cluster with ajp connector?

              • 4. Re: (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to
                okelet

                Yes, I think the AJP connector is better than HTTP, isn't it?

                • 5. Re: (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to
                  erasmomarciano

                  You have add this tag in httpd.conf

                   

                  <Location /modcluster-manager>

                                 SetHandler mod_cluster-manager

                                 Order deny,allow

                                 Deny from all

                                 Allow from 127.0.0.1

                          </Location>

                   

                  Then you open your broswer and call http://X.X.X.X/modcluster-manager

                   

                  You should see a web page that says if it's working mod_cluster.



                  • 6. Re: (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to
                    okelet

                    The mod_cluster manager page works fine, even the app is accesed fine, but most of the times, Apache gives a 503 error, due to the previous errors.

                    • 7. Re: (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to
                      erasmomarciano

                      Error 503 in http signify

                      "The server is currently unavailable (because it is overloaded or down for maintenance)"

                       

                      When you get this error, you have check that Jboss it works correctly.

                       

                      You have done this test

                      • 8. Re: (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to
                        okelet

                        Yes, the app is working fine. I am testing the app with siege, and if I lauch the tests directly against JBoss, port 8080 (in this case port 8380, because of the offset), the app works fine always, but if I launch the tests against Apache, most of the times I get a 503 error.

                        • 9. Re: (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to
                          pandes3

                          Hi Juan,

                           

                          Did you find any fix for this problem ?

                           

                          Regards

                          • 10. Re: (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to
                            okelet

                            Hi Santy. No, the errors continue happening.

                            • 11. Re: (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to
                              jfclere

                              "but most of the times" does it mean that a some point you can use the webapp through Apache httpd?

                              in the error_log so you have something STATUS or request that are OK and other that aren't?

                              • 12. Re: (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to
                                pandes3

                                Hi Jean,

                                 

                                I see the following error messages in the apache error log every night, but I think the application work fine. Again next morning these messages disappear

                                 

                                [Tue Oct 01 03:03:39 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed

                                [Tue Oct 01 03:03:39 2013] [error] (70007)The timeout specified has expired: proxy: AJP: cping/cpong failed to 192.168.39.180:8009 (192.168.39.180)

                                [Tue Oct 01 03:16:02 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed

                                [Tue Oct 01 03:16:02 2013] [error] (70007)The timeout specified has expired: proxy: AJP: cping/cpong failed to 192.168.39.181:8009 (192.168.39.181)

                                [Tue Oct 01 04:02:19 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed

                                [Tue Oct 01 04:02:19 2013] [error] (70007)The timeout specified has expired: proxy: AJP: cping/cpong failed to 192.168.39.181:8009 (192.168.39.181)

                                [Tue Oct 01 04:03:26 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed

                                [Tue Oct 01 04:03:26 2013] [error] (70007)The timeout specified has expired: proxy: AJP: cping/cpong failed to 192.168.39.180:8009 (192.168.39.180)

                                [Tue Oct 01 04:12:19 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed

                                [Tue Oct 01 04:12:19 2013] [error] (70007)The timeout specified has expired: proxy: AJP: cping/cpong failed to 192.168.39.181:8009 (192.168.39.181)

                                [Tue Oct 01 04:13:07 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed

                                [Tue Oct 01 04:13:07 2013] [error] (70007)The timeout specified has expired: proxy: AJP: cping/cpong failed to 192.168.39.181:8009 (192.168.39.181)

                                [Tue Oct 01 04:17:40 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed

                                [Tue Oct 01 04:17:40 2013] [error] (70007)The timeout specified has expired: proxy: AJP: cping/cpong failed to 192.168.39.180:8009 (192.168.39.180)

                                [Tue Oct 01 04:19:17 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed

                                [Tue Oct 01 04:19:17 2013] [error] (70007)The timeout specified has expired: proxy: AJP: cping/cpong failed to 192.168.39.180:8009 (192.168.39.180)

                                [Tue Oct 01 04:20:02 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed

                                [Tue Oct 01 04:20:02 2013] [error] (70007)The timeout specified has expired: proxy: AJP: cping/cpong failed to 192.168.39.180:8009 (192.168.39.180)

                                [Tue Oct 01 04:23:07 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed

                                [Tue Oct 01 04:23:07 2013] [error] (70007)The timeout specified has expired: proxy: AJP: cping/cpong failed to 192.168.39.181:8009 (192.168.39.181)

                                [Tue Oct 01 04:28:58 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed

                                [Tue Oct 01 04:28:58 2013] [error] (70007)The timeout specified has expired: proxy: AJP: cping/cpong failed to 192.168.39.181:8009 (192.168.39.181)

                                [Tue Oct 01 04:34:09 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed

                                [Tue Oct 01 04:34:09 2013] [error] (70007)The timeout specified has expired: proxy: AJP: cping/cpong failed to 192.168.39.180:8009 (192.168.39.180)

                                [Tue Oct 01 04:40:45 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed

                                 

                                On weekend I see these messages continuously and in the end the I start seeing 503 and 500 errors in the log, I think when there is no load on the web I start seeing these errors.

                                 

                                30 10:05:20 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed

                                [Mon Sep 30 10:05:20 2013] [error] (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to 192.168.39.180:8009 (192.168.39.180)

                                [Mon Sep 30 10:05:20 2013] [error] (104)Connection reset by peer: ajp_ilink_receive() can't receive header

                                [Mon Sep 30 10:05:20 2013] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed

                                [Mon Sep 30 10:05:20 2013] [error] (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to 192.168.39.180:8009 (192.168.39.180)

                                [Mon Sep 30 10:05:20 2013] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state

                                [Mon Sep 30 10:05:22 2013] [error] proxy: ajp: disabled connection for (192.168.39.181)

                                [Mon Sep 30 10:05:22 2013] [error] proxy: ajp: disabled connection for (192.168.39.182)

                                [Mon Sep 30 10:05:23 2013] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state

                                [Mon Sep 30 10:05:25 2013] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state

                                 

                                My environment is 3 apache (2.2.15) servers and 3 Jboss (7.1.1) standalone server cluster in Mod_Cluster configuration hosted on redhat Linux 6 environment. There is a vshield firewall between apache and Jboss servers and this is passing traffic from apache to jboss (TCP 8009) and from Jboss to Apache (80).

                                 

                                I am not sure whether this is due to environment or configuration of Jboss and Apache ?

                                 

                                Any help would be appreciated.

                                 

                                Regards

                                 

                                Santy

                                 

                                 

                                • 13. Re: (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to
                                  jfclere

                                  your problem is very weird it doesn't look to be related to any configuration problem but it looks like the Jboss application is hanging during the nights and the week-end. Anything in the server.log file?

                                  • 14. Re: Re: (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to
                                    pandes3

                                    Hi Jean,

                                     

                                    I am monitoring the Jboss application and it looks good (Heap, threads etc), I don't see any errors in Jboss logs with respect to proxy config or configuration reset.

                                     

                                    I see the following error continuously but this seems to be a bug in Jboss (I am seeing this errors after I enabled Jboss management traffic on my management IP for administration).

                                     

                                    16:43:28,874 ERROR [org.jboss.as.controller.management-operation] (HttpManagementService-threads - 363) JBAS014612: Operation ("read-attribute") failed - address: ([
                                    16:43:29,934 ERROR [org.jboss.as.controller.management-operation] (HttpManagementService-threads - 363) JBAS014612: Operation ("read-attribute") failed - address: ([
                                    16:43:30,945 ERROR [org.jboss.as.controller.management-operation] (HttpManagementService-threads - 363) JBAS014612: Operation ("read-attribute") failed - address: ([
                                    
                                    
                                    

                                     

                                    I start my jboss with following command ( standalone.sh -b production_ip -bmanagement management_ip )

                                     

                                    If I start Jboss with production IP for app and management then I don't see these errors ( standalone.sh -b production_ip -bmanagement production_ip)

                                     

                                    I have attached the jboss and apache error log (Since last night), please can you have a look in to this.

                                     

                                    Regards

                                     

                                    Santy

                                    1 2 3 Previous Next