8 Replies Latest reply on Jan 10, 2012 4:24 AM by jfclere

    mod_cluster and Proxypass directives for hosts that are down results in 503 error

    ishai

      Dear mod_cluster community member

       

      I am facing the following issue after switching from mod_jk 1.2.31 to mod_cluster 1.1.2 (could not work with 1.1.3 due to the this issue) on Apache 2.2.17 RHEL: JK set-up works OK even if one or multiple reverse proxied hosts are down. With mod_cluster 503 error is reported if some of the reverse proxied hosts are unavailable or non-responsive. Commenting out such reverse proxy directives restores the system to its normal behavior. In both scenarious same cluster of 2 JBoss 5.1.0.GA machines is utilized.

       

      As much as I enjoyed working with mod_cluster and its seemless configuration I am at the brink of going back to mod_jk. Any help will be greatly appreciated.

       

      Irena

       

      In the configrations below host1 is down and the application on host2 is not responsive

       

      JK configuration

      httpd.conf

      LoadModule proxy_module modules/mod_proxy.so

      LoadModule proxy_balancer_module modules/mod_proxy_balancer.so

      LoadModule jk_module modules/mod_jk-1.2.31-httpd-2.2.x.so

      JkWorkersFile /etc/httpd/conf/workers.properties

      JkShmFile     /var/log/httpd/mod_jk.shm

      JkLogFile     /var/log/httpd/mod_jk.log

      JkLogLevel    info

      JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "

      JkMount  /jboss-content-1/* jboss

      JkMount  /jboss-content-1* jboss

      JkMount  /jboss-content-2/* jboss

      JkMount  /jboss-content-2* jboss

      ...

      ProxyPass /content1/ http://host1/content1/

      ProxyPassReverse /content1/ http://host1/content1/

       

      ProxyPass /content2/ https://host2/content2/

      ProxyPassReverse /content2/ https://host2/content2/

       

      ProxyPass /content3/ http://host3/content1/

      ProxyPassReverse /content3/ http://host3/content1/

      workers.properties

       

      worker.list=jboss

       

      worker.worker1.port=8009

      worker.worker1.host=jboss-1

      worker.worker1.type=ajp13

      worker.worker1.lbfactor=1

      worker.worker1.ping_mode=A

      worker.worker1.fail_on_status=400,404,500,503

      worker.worker1.recovery_options=28

       

      worker.worker2.port=8009

      worker.worker2.host=jboss-2

      worker.worker2.type=ajp13

      worker.worker2.lbfactor=1

      worker.worker2.ping_mode=A

      worker.worker2.fail_on_status=400,404,500,503

      worker.worker2.recovery_options=28

       

      # Define the LB worker

      worker.jboss.type=lb

      worker.jboss.balance_workers=worker1,worker2

      worker.jboss.sticky_session=1

       

      mod_cluster configuration

      httpd.conf

      # MOD CLUSTER - START

      LoadModule proxy_module modules/mod_proxy.so

      LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

      LoadModule slotmem_module modules/mod_slotmem.so

      LoadModule manager_module modules/mod_manager.so

      LoadModule proxy_cluster_module modules/mod_proxy_cluster.so

      #LoadModule advertise_module modules/mod_advertise.so

      #LoadModule proxy_module modules/mod_proxy.so

      #LoadModule proxy_balancer_module modules/mod_proxy_balancer.so

       

      Listen *:7777

      <VirtualHost *:7777>

       

        <Directory />

          Order deny,allow

          Deny from all

          Allow from some.ip

        </Directory>

       

        CreateBalancers 0

        KeepAliveTimeout 60

        MaxKeepAliveRequests 0

        LogLevel debug

       

       

      ProxyPassMatch / !

      </VirtualHost>

       

      # MOD CLUSTER - END

      ...

      ProxyPass /content1/ http://host1/content1/

      ProxyPassReverse /content1/ http://host1/content1/

       

      ProxyPass /content2/ https://host2/content2/

      ProxyPassReverse /content2/ https://host2/content2/

       

      ProxyPass /content3/ http://host3/content1/

      ProxyPassReverse /content3/ http://host3/content1/

       

       

       

        • 1. Re: mod_cluster and Proxypass directives for hosts that are down results in 503 error
          rhusar

          Hi Irena,

           

          so do I understand correctly that /content1,2,3 is not available if a node goes down but /jboss-content-1,2,3 works okay?

           

          (Let me note that mod_jk's fail_on_status=400,404,500,503 doesnt work in mod_cluster. Some of those are non-retryable status codes thus break the HTTP spec on retrying.)

           

          Rado

          • 2. Re: mod_cluster and Proxypass directives for hosts that are down results in 503 error
            ishai

            Hi Rado

             

            Following contents are supposed to be reverse proxied from the same Apache instance:

            1. Content{1,2} are either host unavailable or unresponsive (non-jboss)
            2. Content3 is responsive (non-jboss)
            3. /jboss-content-{1,2} is serverd from 2 node JBoss cluster either via mod_jk or via mod_cluster.
              1. With mod_jk those are serviced OK regardless of issues with content{1,2}.
              2. With mod_cluster 503 is returned unless directives for reverse proxying content{1,2} is commented out in httpd.conf

             

            Hope that helps (let me know if still unclear),

            Irena

             

            UPDATED Dec 27, 2011:

            I ran with the offending proxy pass directives and without those with the LogLevel debug. Here is the snippets of activities on updates from JBoss to the manager. Looks like proxy_util is attempting to establish connection to <host>:<port> which is down while it actually supposed to open an ajp connection to the <ip> of the registering JBoss. Will try looking into the proxy_util code (line 1999). Please share any ideas, insight, etc.

             

            Irena

             

            working

            [Tue Dec 27 12:55:23 2011] [debug] mod_manager.c(1653): manager_trans STATUS (/)

            [Tue Dec 27 12:55:23 2011] [debug] mod_manager.c(2296): manager_handler STATUS (/) processing: "JVMRoute=worker1&Load=84"

            [Tue Dec 27 12:55:23 2011] [debug] mod_manager.c(1394): Processing STATUS

            [Tue Dec 27 12:55:23 2011] [debug] proxy_util.c(1999): proxy: ajp: has acquired connection for (<ip>)

            [Tue Dec 27 12:55:23 2011] [debug] proxy_util.c(2055): proxy: connecting ajp://<ip>:8009/ to <ip>:8009

            [Tue Dec 27 12:55:23 2011] [debug] proxy_util.c(2153): proxy: connected ajp://<ip>:8009/ to <ip>:8009

            [Tue Dec 27 12:55:23 2011] [debug] mod_proxy_cluster.c(716): ajp_cping_cpong: Done

            [Tue Dec 27 12:55:23 2011] [debug] proxy_util.c(2017): proxy: ajp: has released connection for (<ip>)

            [Tue Dec 27 12:55:23 2011] [debug] mod_manager.c(2339): manager_handler STATUS  OK

            non-working

            [Tue Dec 27 12:52:14 2011] [debug] mod_manager.c(1653): manager_trans STATUS (/)

            [Tue Dec 27 12:52:14 2011] [debug] mod_manager.c(2296): manager_handler STATUS (/) processing: "JVMRoute=worker2&Load=80"

            [Tue Dec 27 12:52:14 2011] [debug] mod_manager.c(1394): Processing STATUS

            [Tue Dec 27 12:52:14 2011] [debug] proxy_util.c(1999): proxy: http: has acquired connection for (<host>)

            [Tue Dec 27 12:52:14 2011] [debug] proxy_util.c(2055): proxy: connecting http://<host>:<port>/ to <host>:<port>

            [Tue Dec 27 12:52:14 2011] [debug] proxy_util.c(2153): proxy: connected http://<host>:<port>/ to <host>:<port>

            [Tue Dec 27 12:52:14 2011] [debug] proxy_util.c(2244): proxy: http: backend socket is disconnected.

            [Tue Dec 27 12:52:14 2011] [debug] proxy_util.c(2308): proxy: http: fam 2 socket created to connect to <host>

            [Tue Dec 27 12:52:14 2011] [debug] proxy_util.c(2017): proxy: http: has released connection for (<host>)

            [Tue Dec 27 12:52:14 2011] [debug] mod_manager.c(2339): manager_handler STATUS  OK

            • 3. Re: mod_cluster and Proxypass directives for hosts that are down results in 503 error
              jfclere

              The STATUS looks OK but it didn't check much as Rado explained it.

              We need a bit more log information... Like the CONFIG and ENABLE message as well that the failed request traces.

              • 4. Re: mod_cluster and Proxypass directives for hosts that are down results in 503 error
                ishai

                Hi Jean-Frederic

                 

                Those are the requests that resulted in 503:

                incoming.ip - - [27/Dec/2011:12:50:48 -0500] "GET /jboss-content-1/ui/ext-js/ext-all-debug.js HTTP/1.1" 503 405
                incoming.ip - - [27/Dec/2011:12:50:48 -0500] "GET /jboss-content-1/ui/ext-js/resources/css/ext-all.css HTTP/1.1" 503 405
                incoming.ip - - [27/Dec/2011:12:50:48 -0500] "GET /jboss-content-1/ui/ca-css/css/ca-all.css HTTP/1.1" 503 405
                incoming.ip - - [27/Dec/2011:12:50:48 -0500] "GET /jboss-content-1/ui/ext-js/adapter/ext/ext-base.js HTTP/1.1" 503 405
                incoming.ip - - [27/Dec/2011:12:50:48 -0500] "GET /jboss-content-1/ui/subcontent2/logo-r20-blue.png HTTP/1.1" 503 405
                incoming.ip - - [27/Dec/2011:12:50:51 -0500] "GET /jboss-content-2 HTTP/1.1" 503 405
                

                The rest of the logs (complete) can be found here.

                 

                Irena

                • 5. Re: mod_cluster and Proxypass directives for hosts that are down results in 503 error
                  jfclere

                  Hm it seems there is a bug somewhere... I have reproduced the problem and created a JIRA MODCLUSTER-274

                  1 of 1 people found this helpful
                  • 6. Re: mod_cluster and Proxypass directives for hosts that are down results in 503 error
                    ishai

                    Thank you Jean-Frederic for the update.

                    • 7. Re: mod_cluster and Proxypass directives for hosts that are down results in 503 error
                      ishai

                      Jean-Frederic

                       

                      When will the fix be available for dowload as an official version?

                       

                      Thanks,

                      Irena

                      • 8. Re: mod_cluster and Proxypass directives for hosts that are down results in 503 error
                        jfclere

                        A 1.2.0 should be available for testing soon.