11 Replies Latest reply on Feb 19, 2014 5:21 AM by jfclere

    jboss wont register available contexts in mod_cluster after restart

    gurrish
    gurrish Feb 13, 2014 10:07 AM

      Hi!

       

      I have a problem with mod_cluster. The versions are modcluster 1.2.0 on apache 2.2.23 and jboss AS7 jboss-as-7.1.3.Final, java 1.7_04

       

      I run jboss in domain mode and my setup is like this:

       

      I have 2 modclusters, 2 slaves and 1 master.

       

      First i start my jboss master, then the mod_clusters and finally the slaves.

       

      If I check the modcluster access log I can see that traffic is only sent to one of the slaves and if i check the modcluster_manager GUI I can only see registered contexts on one of the slaves.

       

      If I stop the slaves and the master, remove all the deployments from the master node and then start everything up again, the applications will be registered as available in modcluster. But if I restart one of the slaves after this, it will never register the contexts.

       

      What should I do? Has anyone ever experienced this before as well?

        • 1. Re: jboss wont register available contexts in mod_cluster after restart
          gurrish
          gurrish Feb 13, 2014 10:16 AM (in response to gurrish)

          httpd.conf

          ServerRoot "/opt/modcluster"

          Listen 8181

           

           

          DefaultType None

          ErrorLog logs/error_log

          LogFormat "%h %l %u %t \"%r\" %>s %b %{BALANCER_WORKER_ROUTE}e" common

          CustomLog logs/access_log common

          ServerName modcluster:8181

           

           

          # mod_cluster modules

          LoadModule proxy_module modules/mod_proxy.so

          LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

          LoadModule slotmem_module modules-local/mod_slotmem.so

          LoadModule manager_module modules-local/mod_manager.so

          LoadModule proxy_cluster_module modules-local/mod_proxy_cluster.so

          # authentication module

          LoadModule authz_host_module modules/mod_authz_host.so

          # logging module

          LoadModule log_config_module modules/mod_log_config.so

           

           

          #rewrite module

          LoadModule rewrite_module modules/mod_rewrite.so

           

           

          # mod_cluster configuration

          Maxhost 100

          Maxnode 100

          Maxcontext 1000

           

           

          Listen modcluster:7777

          ManagerBalancerName modcluster

           

           

          RewriteEngine On

           

           

          <VirtualHost modcluster:7777>

              KeepAliveTimeout 300

              MaxKeepAliveRequests 0

              EnableMCPMReceive

              CheckNonce Off

           

           

              <Location />

                  Order Deny,Allow

                  #Deny from all

                  #Allow from 127.0.0

                  # NOTE! Only for testing

                  Allow from all

              </Location>

           

           

             <Location /mod_cluster_manager>

                SetHandler mod_cluster-manager

                Order Deny,Allow

                #Deny from all

                #Allow from 127.0.0

                #Note! Only for testing

                Allow from all

             </Location>

          </VirtualHost>

           

          I will post the jboss conf as well.

            • Actions
          • 2. Re: jboss wont register available contexts in mod_cluster after restart
            mbabacek
            mbabacek Feb 13, 2014 10:23 AM (in response to gurrish)

            Please, post your error_log on LogLevel Debug, so as we can see the workers coming and going...

              • Actions
            • 3. Re: jboss wont register available contexts in mod_cluster after restart
              jfclere
              jfclere Feb 14, 2014 6:45 AM (in response to gurrish)

              modcluster 1.2.0 (that looks fishy) Are you sure? If that is the case use a more recent version.

                • Actions
              • 4. Re: jboss wont register available contexts in mod_cluster after restart
                gurrish
                gurrish Feb 14, 2014 6:47 AM (in response to mbabacek)

                Hi!

                 

                I did what you asked for. I made sure that no traffic was sent during the time that monitored the error_log on debug level. This is what I see.

                 

                [Fri Feb 14 11:44:44 2014] [debug] proxy_util.c(2013): proxy: ajp: has acquired connection for (jboss-slave1)

                [Fri Feb 14 11:44:44 2014] [debug] proxy_util.c(2069): proxy: connecting ajp://jboss-slave1:9009/ to jboss-slave1:9009

                [Fri Feb 14 11:44:44 2014] [debug] proxy_util.c(2195): proxy: connected ajp://jboss-slave1:9009/ to jboss-slave1:9009

                [Fri Feb 14 11:44:44 2014] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done

                [Fri Feb 14 11:44:44 2014] [debug] proxy_util.c(2031): proxy: ajp: has released connection for (jboss-slave1)

                [Fri Feb 14 11:44:44 2014] [debug] mod_manager.c(2366): manager_handler STATUS  OK

                [Fri Feb 14 11:44:45 2014] [debug] proxy_util.c(2013): proxy: ajp: has acquired connection for (jboss-slave1)

                [Fri Feb 14 11:44:45 2014] [debug] proxy_util.c(2069): proxy: connecting ajp://jboss-slave1:9009/ to jboss-slave1:9009

                [Fri Feb 14 11:44:45 2014] [debug] proxy_util.c(2195): proxy: connected ajp://jboss-slave1:9009/ to jboss-slave1:9009

                [Fri Feb 14 11:44:45 2014] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done

                [Fri Feb 14 11:44:45 2014] [debug] proxy_util.c(2031): proxy: ajp: has released connection for (jboss-slave1)

                [Fri Feb 14 11:44:47 2014] [debug] proxy_util.c(2013): proxy: ajp: has acquired connection for (jboss-slave2)

                [Fri Feb 14 11:44:47 2014] [debug] proxy_util.c(2069): proxy: connecting ajp://jboss-slave2:9009/ to jboss-slave2:9009

                [Fri Feb 14 11:44:47 2014] [debug] proxy_util.c(2195): proxy: connected ajp://jboss-slave2:9009/ to jboss-slave2:9009

                [Fri Feb 14 11:44:47 2014] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done

                [Fri Feb 14 11:44:47 2014] [debug] proxy_util.c(2031): proxy: ajp: has released connection for (jboss-slave2)

                [Fri Feb 14 11:44:50 2014] [debug] proxy_util.c(2013): proxy: ajp: has acquired connection for (jboss-slave1)

                [Fri Feb 14 11:44:50 2014] [debug] proxy_util.c(2069): proxy: connecting ajp://jboss-slave1:9009/ to jboss-slave1:9009

                [Fri Feb 14 11:44:50 2014] [debug] proxy_util.c(2195): proxy: connected ajp://jboss-slave1:9009/ to jboss-slave1:9009

                [Fri Feb 14 11:44:50 2014] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done

                [Fri Feb 14 11:44:50 2014] [debug] proxy_util.c(2031): proxy: ajp: has released connection for (jboss-slave1)

                [Fri Feb 14 11:44:51 2014] [debug] mod_manager.c(1667): manager_trans STATUS

                [Fri Feb 14 11:44:51 2014] [debug] mod_manager.c(2323): manager_handler STATUS processing: "JVMRoute=slave2-application-group-1-slave&Load=100"

                [Fri Feb 14 11:44:51 2014] [debug] mod_manager.c(1404): Processing STATUS

                [Fri Feb 14 11:44:51 2014] [debug] proxy_util.c(2013): proxy: ajp: has acquired connection for (jboss-slave2)

                [Fri Feb 14 11:44:51 2014] [debug] proxy_util.c(2069): proxy: connecting ajp://jboss-slave2:9009/ to jboss-slave2:9009

                [Fri Feb 14 11:44:51 2014] [debug] proxy_util.c(2195): proxy: connected ajp://jboss-slave2:9009/ to jboss-slave2:9009

                [Fri Feb 14 11:44:51 2014] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done

                [Fri Feb 14 11:44:51 2014] [debug] proxy_util.c(2031): proxy: ajp: has released connection for (jboss-slave2)

                [Fri Feb 14 11:44:51 2014] [debug] mod_manager.c(2366): manager_handler STATUS  OK

                [Fri Feb 14 11:44:52 2014] [debug] proxy_util.c(2013): proxy: ajp: has acquired connection for (jboss-slave2)

                [Fri Feb 14 11:44:52 2014] [debug] proxy_util.c(2069): proxy: connecting ajp://jboss-slave2:9009/ to jboss-slave2:9009

                [Fri Feb 14 11:44:52 2014] [debug] proxy_util.c(2195): proxy: connected ajp://jboss-slave2:9009/ to jboss-slave2:9009

                [Fri Feb 14 11:44:52 2014] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done

                [Fri Feb 14 11:44:52 2014] [debug] proxy_util.c(2031): proxy: ajp: has released connection for (jboss-slave2)

                [Fri Feb 14 11:44:54 2014] [debug] mod_manager.c(1667): manager_trans STATUS

                [Fri Feb 14 11:44:54 2014] [debug] mod_manager.c(2323): manager_handler STATUS processing: "JVMRoute=slave1-application-group-1-slave&Load=99"

                [Fri Feb 14 11:44:54 2014] [debug] mod_manager.c(1404): Processing STATUS

                [Fri Feb 14 11:44:54 2014] [debug] proxy_util.c(2013): proxy: ajp: has acquired connection for (jboss-slave1)

                [Fri Feb 14 11:44:54 2014] [debug] proxy_util.c(2069): proxy: connecting ajp://jboss-slave1:9009/ to jboss-slave1:9009

                [Fri Feb 14 11:44:54 2014] [debug] proxy_util.c(2195): proxy: connected ajp://jboss-slave1:9009/ to jboss-slave1:9009

                [Fri Feb 14 11:44:54 2014] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done

                [Fri Feb 14 11:44:54 2014] [debug] proxy_util.c(2031): proxy: ajp: has released connection for (jboss-slave1)

                [Fri Feb 14 11:44:54 2014] [debug] mod_manager.c(2366): manager_handler STATUS  OK

                [Fri Feb 14 11:44:55 2014] [debug] proxy_util.c(2013): proxy: ajp: has acquired connection for (jboss-slave1)

                [Fri Feb 14 11:44:55 2014] [debug] proxy_util.c(2069): proxy: connecting ajp://jboss-slave1:9009/ to jboss-slave1:9009

                [Fri Feb 14 11:44:55 2014] [debug] proxy_util.c(2195): proxy: connected ajp://jboss-slave1:9009/ to jboss-slave1:9009

                [Fri Feb 14 11:44:55 2014] [debug] proxy_util.c(2446): proxy: ajp: fam 2 socket created to connect to jboss-slave1

                [Fri Feb 14 11:44:55 2014] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done

                [Fri Feb 14 11:44:55 2014] [debug] proxy_util.c(2031): proxy: ajp: has released connection for (jboss-slave1)

                [Fri Feb 14 11:44:57 2014] [debug] proxy_util.c(2013): proxy: ajp: has acquired connection for (jboss-slave2)

                [Fri Feb 14 11:44:57 2014] [debug] proxy_util.c(2069): proxy: connecting ajp://jboss-slave2:9009/ to jboss-slave2:9009

                [Fri Feb 14 11:44:57 2014] [debug] proxy_util.c(2195): proxy: connected ajp://jboss-slave2:9009/ to jboss-slave2:9009

                [Fri Feb 14 11:44:57 2014] [debug] mod_proxy_cluster.c(773): ajp_cping_cpong: Done

                [Fri Feb 14 11:44:57 2014] [debug] proxy_util.c(2031): proxy: ajp: has released connection for (jboss-slave2)

                  • Actions
                • 5. Re: jboss wont register available contexts in mod_cluster after restart
                  mbabacek
                  mbabacek Feb 14, 2014 6:59 AM (in response to gurrish)

                  Hmm, could you grep for CONFIG message in that debug log? One should be able to see contexts reported by the worker nodes (you can anonymize the actual names of the contexts, of course). The point is to see which nodes actually registered contexts with that message and which nodes didn't send anything.   BTW: Yes, as JFC suggested, mod_cluster 1.2.0 is quite an obsolete one, a huge pile of stuff was made better for 1.2.6 version...

                  1 of 1 people found this helpful
                    • Actions
                  • 6. Re: jboss wont register available contexts in mod_cluster after restart
                    gurrish
                    gurrish Feb 14, 2014 7:52 AM (in response to mbabacek)

                    I guess that the problem is within the JBoss

                     

                    When I grep for config I can see that jboss-slave1

                    [Fri Feb 14 12:14:55 2014] [debug] mod_manager.c(2366): manager_handler CONFIG  OK

                     

                    Which is followed by

                     

                    alot of theese

                     

                    [Fri Feb 14 12:14:55 2014] [debug] mod_manager.c(2323): manager_handler ENABLE-APP  processing: "JVMRoute=slave1-application-group-1-slave&Alias=default-host%2Clocalhost&Context=%2Fapp1-1"

                    [Fri Feb 14 12:14:55 2014] [debug] mod_manager.c(2366): manager_handler ENABLE-APP  OK

                     

                     

                    The second one just does this

                     

                    [Fri Feb 14 12:15:02 2014] [debug] mod_manager.c(1667): manager_trans CONFIG

                    [Fri Feb 14 12:15:02 2014] [debug] mod_manager.c(2323): manager_handler CONFIG  processing: "JVMRoute=slave2-application-group-1-slave&Host=jboss-slave2&Maxattempts=1&Port=9009&StickySessionForce=No&Type=ajp&ping=10"

                    [Fri Feb 14 12:15:02 2014] [debug] mod_manager.c(2366): manager_handler CONFIG  OK

                    [Fri Feb 14 12:15:02 2014] [debug] mod_manager.c(1667): manager_trans STATUS

                    [Fri Feb 14 12:15:02 2014] [debug] mod_manager.c(2323): manager_handler STATUS  processing: "JVMRoute=slave2-application-group-1-slave&Load=100"

                    [Fri Feb 14 12:15:02 2014] [debug] mod_manager.c(1404): Processing STATUS

                      • Actions
                    • 7. Re: jboss wont register available contexts in mod_cluster after restart
                      gurrish
                      gurrish Feb 17, 2014 4:20 AM (in response to mbabacek)

                      Is it hard to upgrade from 1.2.0 to 1.2.6? What do I have to do except replacing the modules in apache? Is there some kind of migration guide that I can use?

                        • Actions
                      • 8. Re: jboss wont register available contexts in mod_cluster after restart
                        mbabacek
                        mbabacek Feb 17, 2014 4:35 AM (in response to gurrish)

                        Dear Gustaf, one can just replace the modules:

                         

                        • mod_proxy_cluster.so
                        • mod_slotmem.so
                        • mod_manager.so
                        • mod_advertise.so

                         

                        there is really nothing to fear with migrating from 1.2.0 to 1.2.6. No update to your configuration is necessary.

                        1 of 1 people found this helpful
                          • Actions
                        • 9. Re: jboss wont register available contexts in mod_cluster after restart
                          gurrish
                          gurrish Feb 17, 2014 5:25 AM (in response to mbabacek)

                          As I mentioned in the beginning, I'm using apache 2.2.23, do you also recommend an upgrade to a more recent apache? I know that there are improvements made to mod_proxy_ajp.

                            • Actions
                          • 10. Re: jboss wont register available contexts in mod_cluster after restart
                            gurrish
                            gurrish Feb 18, 2014 9:04 AM (in response to mbabacek)

                            While at it, I did upgrade my apache server as well. Now when I'm running a 2.4 with mpm event instead of worker (should I recompile and use worker instead?) I wan't to know which parameters that I need to consider when I want my cluster to be able to serve more parallel requests. The limit seems to be a problem for me now, I can only handle about 50 parallel connections, the rest of the clients will receive a 502 response.

                              • Actions
                            • 11. Re: jboss wont register available contexts in mod_cluster after restart
                              jfclere
                              jfclere Feb 19, 2014 5:21 AM (in response to gurrish)

                              When using httpd-2.4.x you should use master for the moment.

                                • Actions