1 2 Previous Next 15 Replies Latest reply on Jul 24, 2017 4:30 PM by gsandeepry

    Error 503 using mod_cluster

    nichele

      Hi All,

      i'm using mod_cluster in production but i'm having some 503.

      I tried replacing mod_cluster with mod_proxy_balancer and in such case i didn't have any 503 so i'm thinking to a misconfiguration of mod_cluster.

       

      My environemnt is composed by an instance of apache httpd ver. 2.2.19, and of 4 instances of tomcat 6.0.13 with mod_cluster 1.1.2.

       

      In any server.xml file i added:

       

        <Listener className="org.jboss.modcluster.catalina.ModClusterListener"

                  advertise="false"

                  proxyList="my-proxy:6666"

                  maxAttempts="3"

                  nodeTimeout="600"

                  ping="30"

                  loadMetricClass="org.jboss.modcluster.load.metric.impl.BusyConnectorsLoadMetric" />

       

      In my httpd conf file i have:

       

      Listen *:6666

      <VirtualHost *:6666>

       

         <Directory />

            Order deny,allow

            #Deny from all

            Allow from all

         </Directory>

       

         KeepAliveTimeout 60

         MaxKeepAliveRequests 0

       

         ManagerBalancerName mycluster

         ServerAdvertise Off

       

      </VirtualHost>

       

      <Location /cluster-manager>

          SetHandler mod_cluster-manager

          Order deny,allow

          Deny from all

          Allow from xx.xx.xx.xx

      </Location>

       

      ProxyStatus On


      In the apache error log i have:

       

      Sep  5 06:54:10 httpd[26073]: [debug] mod_proxy_cluster.c(622): update_workers_node starting

      Sep  5 06:54:10 httpd[26073]: [debug] mod_proxy_cluster.c(637): update_workers_node done

      Sep  5 06:54:20 httpd[26073]: [error] (70007)The timeout specified has expired: proxy: AJP: attempt to connect to 10.214.14.209:8001 (10.214.14.209) failed

      Sep  5 06:54:20 httpd[26073]: [error] ap_proxy_connect_backend disabling worker for (10.214.14.209)

      Sep  5 06:54:20 httpd[26073]: [error] proxy: AJP: failed to make connection to backend: 10.214.14.209

      Sep  5 06:54:20 httpd[26073]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE02)

      Sep  5 06:54:20 httpd[26073]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE02)

      Sep  5 06:54:21 httpd[26073]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE02)

      Sep  5 06:54:21 httpd[25976]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE02)

      Sep  5 06:54:21 httpd[26073]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE02)

      Sep  5 06:54:21 httpd[25976]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE02)

      Sep  5 06:54:22 httpd[25976]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE02)

       

      For completeness, this is the mod_proxy_balancer configuration i tested:

       

      <Proxy balancer://mycluster>

       

             BalancerMember ajp://node01:8001/ route=NODE01

             BalancerMember ajp://node02:8001/ route=NODE02

             BalancerMember ajp://node03:8001/ route=NODE03

             BalancerMember ajp://node04:8001/ route=NODE04

       

             ProxySet lbmethod=bybusyness

             ProxySet stickysession=JSESSIONID|jsessionid

      </Proxy>

       

      ProxyPass / balancer://mycluster/

      ProxyPassReverse / balancer://mycluster/

       

      ProxyStatus On

       

      Do you have any suggestion ? I tried to play with the timeouts but i was not able to find any working values.

       

      As usual thanks in advance

      Stef

        • 1. Re: Error 503 using mod_cluster
          jfclere

          It looks like to me that the node is not sending the right address to httpd, to say it another way: 10.214.14.20 is not the address of node1 (or any of the nodes in the BalancerMember list). You have to provide the address either while starting the node whith -b node1 for example or adding address=node1 in the ajp connector.

          • 2. Re: Error 503 using mod_cluster
            nichele

            Hi Jean-Frederic,

            i already have the address in the ajp connector:

             

                <Connector port="8001" protocol="AJP/1.3" redirectPort="8441" address="10.214.14.209" maxThreads="200" />

             

            and usually the system works fine.

            This is how the log continues if it can be useful (as you can see after some seconds apache is able to created worker to 10.214.14.209

             

            Sep  5 06:54:29 httpd[26261]: [debug] proxy_util.c(1806): proxy: grabbed scoreboard slot 0 in child 26261 for worker proxy:reverse

            Sep  5 06:54:29 httpd[26261]: [debug] proxy_util.c(1825): proxy: worker proxy:reverse already initialized

            Sep  5 06:54:29 httpd[26261]: [debug] proxy_util.c(1902): proxy: initialized worker 0 in child 26261 for (*) min=0 max=30 smax=30

            Sep  5 06:54:30 httpd[26261]: [debug] mod_proxy_cluster.c(416): add_balancer_node: Create balancer balancer://mycluster

            Sep  5 06:54:30 httpd[26261]: [debug] mod_proxy_cluster.c(228): Created: worker for ajp://10.214.14.209:8001

            Sep  5 06:54:30 httpd[26261]: [debug] mod_proxy_cluster.c(334): proxy: initialized worker 1 in child 26261 for (10.214.14.209) min=0 max=30 smax=30

            Sep  5 06:54:30 httpd[26261]: [debug] mod_proxy_cluster.c(392): Created: worker for ajp://10.214.14.209:8001 1 (status): 1

            Sep  5 06:54:30 httpd[26261]: [debug] mod_proxy_cluster.c(228): Created: worker for ajp://10.194.187.15:8001

            .....

             

            But sometime there is a 503 (not always on the same host) with a timeout in the apache server log.

             

             

            Ste

            • 3. Re: Error 503 using mod_cluster
              rhusar

              I remember in some case CreateBalancers 0/1/2 helped, I don't remember why and how unfortunately. Here are the docs for that:

               

              http://docs.jboss.org/mod_cluster/1.1.0/html/native.config.html#d0e490

              • 4. Re: Error 503 using mod_cluster
                jfclere

                "But sometime there is a 503 (not always on the same host) with a timeout in the apache server log."

                 

                That could be a problem of gc in the JVM too.

                 

                Try to increase the ping and the nodeTimeout values or check the application behaviour.

                • 5. Re: Error 503 using mod_cluster
                  rhusar

                  +1 In situations like that, for undestanding what is wrong make sure to check the servers behind the load balancer by accessing the standard HTTP too.

                  • 6. Re: Error 503 using mod_cluster
                    nichele

                    Hi,

                    i'm trying increasing timeout but just now i have found another error in my apache log that causes at the end a 503:

                     

                    Sep  6 17:35:07 httpd[29260]: [error] ajp_marshal_into_msgb - No such method

                    Sep  6 17:35:07 httpd[29260]: [error] ajp_send_header: ajp_marshal_into_msgb failed

                    Sep  6 17:35:07 httpd[29260]: [error] (120010)APR does not understand this error code: proxy: AJP: request failed to 10.194.187.15:8001 (10.194.187.15)

                    Sep  6 17:35:07 httpd[25446]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE03)

                    Sep  6 17:35:08 httpd[25446]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE03)

                    Sep  6 17:35:08 httpd[29260]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE03)

                    Sep  6 17:35:08 httpd[29260]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE03)

                    Sep  6 17:35:09 httpd[29260]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE03)

                     

                    Thoughts ?

                     

                    Thanks

                    ste

                    • 7. Re: Error 503 using mod_cluster
                      jfclere

                      weird... do you have a little more trace? - before the "ajp_marshal_into_msgb - No such method"

                      It looks like you have a weird request being processed... Unfornatunatly it marks the node in error instead just reject the request.

                      • 8. Re: Error 503 using mod_cluster
                        rhusar

                        "No such method" sounds like messed up versions :-)

                        • 9. Re: Error 503 using mod_cluster
                          nichele

                          I was able to reproduce this last issue:

                           

                          1. open a shell

                          2. type "telnet host post"

                          3. type <SPACE><SPACE><SPACE><ENTER>

                           

                          that's all. The request is malformed and in the apache log there is:

                           

                          Sep  7 15:07:35 httpd[15164]: [error] ajp_marshal_into_msgb - No such method

                          Sep  7 15:07:35 httpd[15164]: [error] ajp_send_header: ajp_marshal_into_msgb failed

                          Sep  7 15:07:35 httpd[15164]: [error] (120010)APR does not understand this error code: proxy: AJP: request failed to 10.254.234.112:8001 (my-node04)

                          Sep  7 15:07:38 httpd[15164]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE04)

                          Sep  7 15:07:39 httpd[15164]: [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state for route (NODE04)

                          Sep  7 15:07:45 httpd[18651]: [debug] proxy_util.c(1806): proxy: grabbed scoreboard slot 0 in child 18651 for worker proxy:reverse

                           

                          and some 503 are returned (i think 2 in this case since we had 2 workers).

                           

                          Im supposing this is bug, doesn't it ?

                          The same scenario doesn't cause issues with mod_proxy (just the request is rejected)

                           

                          (still testing 503 caused by timeout)

                           

                          Thanks

                          ste

                          • 10. Re: Error 503 using mod_cluster
                            jfclere

                            "I'm supposing this is bug, doesn't it ?"

                             

                            Yes. In fact the bug is in mod_proxy_ajp the request should be rejected with not such method (bad request). I have patch httpd-trunk and proposed the patch for back port.

                            • 11. Re: Error 503 using mod_cluster
                              nichele

                              Coud you provide me the patch for 2.2.15  (the version bundled with mod_cluster) ? or for any other version if available....

                               

                              But i have not undestood why with mod_proxy_balancer (and mod_proxy_apj since i'm using ajp as protocol) the issue doesn't occur.

                               

                              thanks

                              ste

                              • 12. Re: Error 503 using mod_cluster
                                jfclere

                                You should upgrade to 2..2.21 see http://www.apache.org/dist/httpd/CHANGES_2.2

                                • 13. Re: Error 503 using mod_cluster
                                  nichele

                                  Thanks Jean-Frederic,

                                  one things i do not understand yet...

                                   

                                  Why mod_cluster releases modules for apache ? I mean, i can use apache 2.2.21 but then i have to overwrite for instance mod_proxy.so with the one coming from mod_cluster...is this correct ? am I not loosing some fixes done in the apache code in this way ?

                                   

                                  I'm asking this since using official httpd modules without to overwrite them with the mod_cluster ones (of course i copied the ones not available in apache) i have some issues (in particular it seems apache doesn't do stickysession).

                                   

                                  thanks

                                  ste

                                  • 14. Re: Error 503 using mod_cluster
                                    jfclere

                                    mod_cluster delivers bundles to make it more easy to test mod_cluster.

                                    we deliver so files too in that case you will have to copy only:

                                    mod_slotmem.so

                                    mod_proxy_cluster.so

                                    mod_manager.so

                                    mod_advertise.so

                                     

                                    Or wait for the next release of mod_cluster

                                    1 2 Previous Next