9 Replies Latest reply on Jun 6, 2013 10:02 AM by thinksteep thinksteep

    mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED

    thinksteep thinksteep Novice

      Hi Community,

       

      I am having hard time in finding a solution for this issue.

       

      I have two JBoss 7.1.1.Final AS systems configured in cluster using mod_cluster +httpd (on Linux machine).

       

      On restart, one the node not registering (NODE2 and ONLY happening with this NODE) with Apache and all requests are returning 503 error.

       

      Here are the messages I am seeing in apache error_log on restart of the NODE2.

       

      Wed May 29 16:36:02 2013] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done

      [Wed May 29 16:36:02 2013] [debug] proxy_util.c(2044): proxy: ajp: has released connection for (jbosscluster.sightlinesystems.local)

      [Wed May 29 16:36:05 2013] [debug] mod_proxy_cluster.c(1954): cluster: Found value t3dRT37a3mApMZB-4H6DTaDM.newnode2 for stickysession JSESSIONID|jsessionid

      [Wed May 29 16:36:05 2013] [debug] mod_proxy_cluster.c(2338): cluster: Using route newnode2

      [Wed May 29 16:36:05 2013] [debug] mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED

      [Wed May 29 16:36:05 2013] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state

       

      I tried to restart NODE1 and not seeing this behaviour there. Any input would be really appreciated.

       

      Thank you for your time and help.

        • 1. Re: mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED
          Michal Karm Babacek Apprentice

          Hi user, this is your community speaking:

           

          Could you post your configs here please? mod_cluster.conf and the modcluster subsystem element from NODE and NODE2?

          Furthermore, paste the whole error_log, or at least more of it please. I wonder whether all the workers are not in the error state, judging by the internal_find_best_byrequests function.

          • 2. Re: mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED
            thinksteep thinksteep Novice

            Hi Michal,

             

            Thanks for quick reply!

             

            Here is web sub-system configuration:

            <server name="newnode2" xmlns="urn:jboss:domain:1.2">

                .......................................

                   <subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">

                        <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>

                        <connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp"/>

                        <virtual-server name="default-host" enable-welcome-root="true">

                            <alias name="localhost"/>

                            <alias name="example.com"/>

                        </virtual-server>

                    </subsystem>

             

            And mod cluster sub system:

             

            <subsystem xmlns="urn:jboss:domain:modcluster:1.0">

                        <mod-cluster-config advertise-socket="modcluster" proxy-list="apacheIP:8080" balancer="mycluster" advertise-security-key="secret">

                            <dynamic-load-provider>

                                <load-metric type="busyness"/>

                            </dynamic-load-provider>

                        </mod-cluster-config>

            </subsystem>

             

            Apache httpd.conf entries:

             

            LoadModule slotmem_module modules/mod_slotmem.so

            LoadModule manager_module modules/mod_manager.so

            LoadModule proxy_cluster_module modules/mod_proxy_cluster.so

            LoadModule advertise_module modules/mod_advertise.so

             

            Listen apacheMachineIP:8080

            <VirtualHost apacheMachineIP:8080>

             

                <Location />

                     Order deny,allow

                     Allow from all

                </Location>

             

                <Location /mod_cluster-manager>

                SetHandler mod_cluster-manager

                Order deny,allow

                Allow from all

                </Location>

             

             

                KeepAliveTimeout 60

                #KeepAlive Off

                MaxKeepAliveRequests 0

             

             

                ManagerBalancerName mycluster

                ServerAdvertise On

                AdvertiseFrequency 5

                EnableMCPMReceive

             

             

            </VirtualHost>

             

             

            Attached error_log from apache (it is trimmed down version which has entries for initial start---> shutdown --->Start again (Which is when I am seeing above entries and all requests are resulting in 503 error).

             

            Even though I have two nodes, I am just playing with Node2 only because this is the one causing issue.

             

            Please let me know if any information required.

             

            Thank you,

            • 3. Re: mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED
              Michal Karm Babacek Apprentice

              Hey Steep, it looks like one of our nodes uses an IPv6 and the other one IPv4 interface, is this intentional?

              First, make everything IPv4 only and test it.

              Furthermore, try to bind both balancer and workers to IPv6 interfaces and test again.

              If you need to work with IPv6, it would be best for you to upgrade to mod_cluster 1.2.3+ as there were some IPv6 related bugs in 1.2.0 if my memory doesn't fail me.

              Are you up to any exotic network setup, like an IPv6/IPv4 hybrid cluster? Why?

              • 4. Re: mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED
                thinksteep thinksteep Novice

                Hi Michal,

                 

                Thanks for your time on this.

                 

                1) AFAIK, We are using IPv4 only. How were you able to figure out IPv4 & IPv6 combination? Could you provide traces? That helps as backup while discussing with network admin.

                 

                2) Log attached in previous reply is from only one node (Which is NODE2). Other node is down and I am not using it at this moment because issue it is working fine on restarts.

                 

                Thank you for your time and help.

                • 5. Re: mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED
                  thinksteep thinksteep Novice

                  One more observation is, on restart of APACHE (Don't need to restart NODE2 after above steps), further requests are being processed succesfully (No more 503 errors).

                   

                  Thank you,

                  • 6. Re: mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED
                    Michal Karm Babacek Apprentice

                    Well, it look like your worker "secondnodeIP" - if it is not just something you searched/replaced in the log, is coming from an IPv6:

                    [Wed May 29 16:33:57 2013] [debug] mod_proxy_ajp.c(616): proxy: got response from [foo:yada:yada:yada:yada:yada:yada:dbbc]:8009 (secondnodeIP)
                    

                    I can't tell from

                    Listen apacheMachineIP:8080
                    

                    what address are you listening on...yet, this looks like your client (browser) address: 192.168.1.79,

                    suggesting that the balancer is listening on IPv4 as well, since it tried to serve you a non existing favicon.

                     

                    Your workers might pick up balancer's advertising (ou don't have advertise="false" configures) or they use what you configured in proxy-list.

                    What I see as a problem in this setup is that balancer can't talk back to IPv6 addresses and contact workers, hence:

                    [Wed May 29 16:35:42 2013] [debug] proxy_util.c(2471): (70007)The timeout specified has expired: proxy: ajp: attempt to connect to [foo:yada:yada:yada:yada:yada:yada:dbbc]:8009 (secondnodeIP) failed
                    

                     

                    Suggestion: Try everything on IPv4 first and if it works, move to IPv6.

                     

                    EDIT: I've seen your latest remark regarding Apache's restart: What address it bound to after it had started? What the debug log looks like now? Is the balancer able to talk back to the workers now?

                    • 7. Re: mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED
                      thinksteep thinksteep Novice

                      Hi Michal,

                       

                      Thanks for your time on this.

                       

                      I have a request, can you please mask the IP in your answer? I don't want to make it public.

                       

                      Yes, you are correct, I did search & replace.

                       

                      Today we disabled IPv6 on both NODE2 & Apache server. Now they have only IPv4 enabled on both machines. Still same issue.

                       

                      Attached full log. Sequence of operations are:

                       

                      1) Started Node2

                      2) Browser request to application (Browser/Client is also on Node2 machine)--->Sucess

                      3) Stopped Node2

                      4) Started Node2

                      5) Browser request to application  ---> 503 error

                      6) Restart the Node2 again

                      7) Browser request to application  ----> 503 error

                      8) Restart Apache server

                      9) Browser request to application -----> Sucess

                       

                      This would be the sequence you see in attached log. Please suggest your thoughts.

                       

                      Thanks again for your time and help.

                      • 8. Re: mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED
                        Michal Karm Babacek Apprentice

                        I had tried to reproduce it with the same httpd version and I failed. I'll try it again... stay tuned.

                        • 9. Re: mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED
                          thinksteep thinksteep Novice

                          Hi Michal,

                           

                          Thanks for your time and help.

                           

                          Any luck on this?

                           

                          Thank you,