3 Replies Latest reply on Feb 25, 2010 6:32 AM by jfclere

    AJP error? or how to configure mod_cluster

    sverker

      I'm having major problems with a setup with apache + mod_cluster on the same host as a jboss instance. Very frequently I get 500 internal server error. This is how the error log look like:

       

      [Tue Feb 23 17:39:15 2010] [error] ajp_read_header: ajp_ilink_receive failed
      [Tue Feb 23 17:39:15 2010] [error] (120006)APR does not understand this error code: proxy: read response failed from 0.0.0.0:8009 (0.0.0.0)
      [Tue Feb 23 17:39:18 2010] [error] proxy: ajp: disabled connection for (0.0.0.0)
      [Tue Feb 23 17:39:28 2010] [error] proxy: ajp: disabled connection for (0.0.0.0)
      [Tue Feb 23 17:39:38 2010] [error] proxy: ajp: disabled connection for (0.0.0.0)
      [Tue Feb 23 17:39:48 2010] [error] proxy: ajp: disabled connection for (0.0.0.0)
      [Tue Feb 23 17:39:58 2010] [error] proxy: ajp: disabled connection for (0.0.0.0)
      [Tue Feb 23 17:40:08 2010] [error] proxy: ajp: disabled connection for (0.0.0.0)
      [Tue Feb 23 18:08:27 2010] [error] [client 81.227.174.54] proxy: error processing body, referer: http://www.taxijournalen.se/export.do?method=prepare
      [Tue Feb 23 18:08:27 2010] [error] proxy: dialog to 0.0.0.0:8009 (0.0.0.0) failed
      [Tue Feb 23 18:30:15 2010] [error] [client 83.250.164.185] proxy: error processing body
      [Tue Feb 23 18:30:15 2010] [error] proxy: dialog to 0.0.0.0:8009 (0.0.0.0) failed
      [Tue Feb 23 18:30:20 2010] [error] [client 83.250.164.185] proxy: error processing body
      [Tue Feb 23 18:30:20 2010] [error] proxy: dialog to 0.0.0.0:8009 (0.0.0.0) failed
      [Tue Feb 23 18:30:23 2010] [error] [client 83.250.164.185] proxy: error processing body
      [Tue Feb 23 18:30:23 2010] [error] proxy: dialog to 0.0.0.0:8009 (0.0.0.0) failed
      [Tue Feb 23 18:30:24 2010] [error] [client 83.250.164.185] proxy: error processing body
      [Tue Feb 23 18:30:24 2010] [error] proxy: dialog to 0.0.0.0:8009 (0.0.0.0) failed
      [Tue Feb 23 18:30:28 2010] [error] [client 83.250.164.185] proxy: error processing body
      [Tue Feb 23 18:30:28 2010] [error] proxy: dialog to 0.0.0.0:8009 (0.0.0.0) failed
      [Tue Feb 23 19:55:22 2010] [error] [client 81.224.201.201] proxy: error processing body
      [Tue Feb 23 19:55:22 2010] [error] proxy: dialog to 0.0.0.0:8009 (0.0.0.0) failed
      [Tue Feb 23 21:49:56 2010] [error] [client 81.231.250.245] proxy: error processing body
      [Tue Feb 23 21:49:56 2010] [error] proxy: dialog to 0.0.0.0:8009 (0.0.0.0) failed
      [Tue Feb 23 21:49:57 2010] [error] [client 81.231.250.245] proxy: error processing body
      [Tue Feb 23 21:49:57 2010] [error] proxy: dialog to 0.0.0.0:8009 (0.0.0.0) failed
      [Tue Feb 23 21:50:02 2010] [error] [client 81.231.250.245] proxy: error processing body, referer: http://www.taxijournalen.se/viewschedule.do?startYear=2010&month=3&width=1259
      [Tue Feb 23 21:50:02 2010] [error] proxy: dialog to 0.0.0.0:8009 (0.0.0.0) failed
      [Tue Feb 23 21:50:07 2010] [error] [client 81.231.250.245] proxy: error processing body, referer: http://www.taxijournalen.se/viewschedule.do?startYear=2010&month=3&width=1259
      [Tue Feb 23 21:50:07 2010] [error] proxy: dialog to 0.0.0.0:8009 (0.0.0.0) failed
      [Tue Feb 23 21:50:07 2010] [error] [client 81.231.250.245] proxy: error processing body, referer: http://www.taxijournalen.se/viewschedule.do?startYear=2010&month=3&width=1259
      [Tue Feb 23 21:50:07 2010] [error] proxy: dialog to 0.0.0.0:8009 (0.0.0.0) failed
      [Tue Feb 23 21:50:09 2010] [error] [client 81.231.250.245] proxy: error processing body, referer: http://www.taxijournalen.se/viewschedule.do?startYear=2010&month=3&width=1259
      [Tue Feb 23 21:50:09 2010] [error] proxy: dialog to 0.0.0.0:8009 (0.0.0.0) failed
      [Tue Feb 23 21:50:09 2010] [error] [client 81.231.250.245] proxy: error processing body, referer: http://www.taxijournalen.se/viewschedule.do?startYear=2010&month=3&width=1259
      [Tue Feb 23 21:50:09 2010] [error] proxy: dialog to 0.0.0.0:8009 (0.0.0.0) failed
      [Tue Feb 23 23:43:13 2010] [error] [client 87.106.90.44] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
      [Wed Feb 24 00:02:46 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
      [Wed Feb 24 00:33:52 2010] [error] ajp_cping_cpong: apr_socket_recv failed
      [Wed Feb 24 00:34:12 2010] [error] ajp_cping_cpong: apr_socket_recv failed
      [Wed Feb 24 00:39:54 2010] [error] ajp_read_header: ajp_ilink_receive failed
      [Wed Feb 24 00:39:54 2010] [error] (120006)APR does not understand this error code: proxy: read response failed from 0.0.0.0:8009 (0.0.0.0)
      [Wed Feb 24 00:39:56 2010] [error] proxy: ajp: disabled connection for (0.0.0.0)
      [Wed Feb 24 00:40:06 2010] [error] proxy: ajp: disabled connection for (0.0.0.0)
      [Wed Feb 24 00:40:16 2010] [error] proxy: ajp: disabled connection for (0.0.0.0)
      [Wed Feb 24 00:40:26 2010] [error] proxy: ajp: disabled connection for (0.0.0.0)
      [Wed Feb 24 00:40:36 2010] [error] proxy: ajp: disabled connection for (0.0.0.0)
      [Wed Feb 24 00:40:46 2010] [error] proxy: ajp: disabled connection for (0.0.0.0)
      [Wed Feb 24 00:41:40 2010] [error] [client 85.228.31.13] proxy: error processing body, referer: http://www.taxijournalen.se/viewschedule.do?startYear=2010&month=2&width=1154
      [Wed Feb 24 00:41:40 2010] [error] proxy: dialog to 0.0.0.0:8009 (0.0.0.0) failed
      [Wed Feb 24 00:41:52 2010] [error] ajp_read_header: ajp_ilink_receive failed
      [Wed Feb 24 00:41:52 2010] [error] (120006)APR does not understand this error code: proxy: read response failed from 0.0.0.0:8009 (0.0.0.0)
      [Wed Feb 24 00:41:56 2010] [error] proxy: ajp: disabled connection for (0.0.0.0)
      [Wed Feb 24 00:42:06 2010] [error] proxy: ajp: disabled connection for (0.0.0.0)
      [Wed Feb 24 00:42:16 2010] [error] proxy: ajp: disabled connection for (0.0.0.0)
      [Wed Feb 24 00:42:26 2010] [error] proxy: ajp: disabled connection for (0.0.0.0)
      [Wed Feb 24 00:42:36 2010] [error] proxy: ajp: disabled connection for (0.0.0.0)
      [Wed Feb 24 00:42:46 2010] [error] proxy: ajp: disabled connection for (0.0.0.0)

       

      My httpd configuration looks like this:

       

      LoadModule proxy_module modules/mod_proxy.so
      LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
      LoadModule slotmem_module modules/mod_slotmem.so
      LoadModule manager_module modules/mod_manager.so
      LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
      LoadModule advertise_module modules/mod_advertise.so

      CreateBalancers 1

      <VirtualHost *:80>
              ServerName www.taxijournalen.se
              ServerAlias taxijournalen.se *.taxijournalen.se *.taxijournalen.com
              ErrorLog logs/taxijournalen-error_log
              CustomLog logs/taxijournalen-access_log combined

              ProxyPass / balancer://mycluster/
              <Directory />
                      Order deny,allow
                      Deny from all
                      Allow from 172.18.60.
                      Allow from 127.0.0.1
              </Directory>
              KeepAliveTimeout 60
              MaxKeepAliveRequests 0

              ManagerBalancerName mycluster
              AdvertiseFrequency 5
      </VirtualHost>

      <Location /mod_cluster-manager>
          SetHandler mod_cluster-manager

          Order deny,allow
          Deny from all
          Allow from 127.0.0.1
      </Location>

      I don't really get the hang of this. It seam that it doesn't matter what I set as ServerName nor ServerAlias here as it finds that out from the cluster nodes somehow. If I didn't set CreateBalancers 1 then other virtualhosts on the apache didn't work but all was sent to jboss.

       

      I was thinking that maybe the problem is in AJP, but how to configure so that http is used instead? Reading the doc it looks like that I just should load mod_proxy_http instead of mod_proxy_ajp but then it doesn't find the worker node.

        • 1. Re: AJP error? or how to configure mod_cluster
          jfclere

          That is MODCLUSTER-91

          To work-around use Address="address or name" in <Connector/>

          • 2. Re: AJP error? or how to configure mod_cluster
            sverker

            Hi Jean-Frederic

            I'm not so sure of that as I read MODCLUSTER-91. I get these failures occationally, and I've got a feeling that it's when the request take a bit long time and I allways see it when trying to start an applet, it fails to download it's classes. The application is an old EJB2 based app so it's not so fast.

             

            Anyway, I tested to set address in connector and get the same problem:

             

            [Wed Feb 24 12:30:13 2010] [error] ajp_cping_cpong: apr_socket_recv failed
            [Wed Feb 24 12:30:17 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
            [Wed Feb 24 12:30:19 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
            [Wed Feb 24 12:30:22 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
            [Wed Feb 24 12:30:34 2010] [error] ajp_cping_cpong: apr_socket_recv failed
            [Wed Feb 24 12:30:35 2010] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed
            [Wed Feb 24 12:30:35 2010] [error] (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to (null) (172.18.60.10)
            [Wed Feb 24 12:30:45 2010] [error] ajp_handle_cping_cpong: ajp_ilink_receive failed
            [Wed Feb 24 12:30:45 2010] [error] (120006)APR does not understand this error code: proxy: AJP: cping/cpong failed to 172.18.60.10:8009 (172.18.60.10)
            [Wed Feb 24 12:31:53 2010] [error] ajp_read_header: ajp_ilink_receive failed
            [Wed Feb 24 12:31:53 2010] [error] (120006)APR does not understand this error code: proxy: read response failed from 172.18.60.10:8009 (172.18.60.10)
            [Wed Feb 24 12:31:58 2010] [error] proxy: ajp: disabled connection for (172.18.60.10)
            [Wed Feb 24 12:31:58 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
            [Wed Feb 24 12:32:01 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
            [Wed Feb 24 12:32:05 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
            [Wed Feb 24 12:32:08 2010] [error] proxy: ajp: disabled connection for (172.18.60.10)
            [Wed Feb 24 12:32:09 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
            [Wed Feb 24 12:32:18 2010] [error] proxy: ajp: disabled connection for (172.18.60.10)
            [Wed Feb 24 12:32:28 2010] [error] proxy: ajp: disabled connection for (172.18.60.10)
            [Wed Feb 24 12:32:38 2010] [error] proxy: ajp: disabled connection for (172.18.60.10)
            [Wed Feb 24 12:32:48 2010] [error] proxy: ajp: disabled connection for (172.18.60.10)

            • 3. Re: AJP error? or how to configure mod_cluster
              jfclere

              That looks be something else....

              Try it increase the ping parameter.