3 Replies Latest reply on Jan 20, 2014 4:52 AM by Jean-Frederic Clere

    Jboss AS 7.1.1Final  + managed domain + modcluster - cluster nodes are not discovered by advertising.

    Sebastian Choina Newbie

      Hello

      I want to setup a basic cluster.

      I have one domain controller on my local host (192.168.2.142 windows) and two virtualboxed slaves (192.168.2.159, 192.168.2.160 puppy linux). As for the domain itself, everything works just fine. Project deployed via CLI on domain controller populates slaves and work just fine one both of them.

      I have configured mod_cluster 1.2.6 final for windows downloaded from mod_cluster 1.2.6.Final bin Downloads - JBoss Community. It is deployed on the same machine as domain controller (192.168.2.142).

      I have checked network configuration for local and virtual instances and it is just fine. I have checked, that UDP brodcasts can be received on virtual systems.

       

      Unfortunetly none of cluster nodes are resolved for load balancing. What am I missing? Acording to official howto's it should work due to fact, that JBoss 7.1.1 has modcluster supperort working out-of-the-box.

       

      Here are some of my configuration files:

       

      httpd.conf

       

        Listen 192.168.2.142:6666
        ManagerBalancerName breeze-cluster
        <VirtualHost 192.168.2.142:6666>
          <Location />
           Order deny,allow
           Deny from all
           Allow from all
          </Location>
      
          KeepAliveTimeout 300
          MaxKeepAliveRequests 0
          ServerAdvertise on http://192.168.2.142:6666
          AdvertiseFrequency 5
          #AdvertiseSecurityKey secret
          #AdvertiseGroup 224.0.1.105:23364
          EnableMCPMReceive
      
          <Location /mod_cluster_manager>
             SetHandler mod_cluster-manager
             Order deny,allow
             Deny from all
             Allow from all
          </Location>
      
        </VirtualHost>
      
      </
      
      

       

       

      domain.xml

       

      <server-groups>
        <server-group name="breeze-cluster" profile="ha">
          <socket-binding-group ref="ha-sockets"/>
          <deployments>
            <deployment name="WebApiPort.war" runtime-name="WebApiPort.war"/>
          </deployments>
        </server-group>
      </server-groups>
      

       

       

      And this is raport from cluster manager:

       

      screen.png

        • 1. Re: Jboss AS 7.1.1Final  + managed domain + modcluster - cluster nodes are not discovered by advertising.
          Michal Karm Babacek Apprentice

          Hello Seb,

          could you go through this basic sanity-check list and report back the results, please?

           

          • Are you sure you've got ha or full-ha profile active for your hosts? (It looks like so, form the snippet you provided though...)
          • Could you copy-paste here the modcluster module configuration that is active? (either from ha or full-ha profile)
          • What can you see in the host's log? Is there anything like "Listening to proxy advertisements on"?
          • Let's not trust UDP multicast in VirtualBox for a moment, and type in the proxy-list="192.168.2.142:6666" as an attribute to modcluster module configuration element (ha or full-ha profile). Does it change anything?
          • Use LogLevel Debug in your httpd.conf. What can you see in the error_log? Aren't there any suspicious messages about balancer not being able to reach its workers?

           

          _K-

           

          EDIT: Obviously, it's proxy-list, not proxy-lisy :-)

          1 of 1 people found this helpful
          • 2. Re: Re: Jboss AS 7.1.1Final  + managed domain + modcluster - cluster nodes are not discovered by advertising.
            Sebastian Choina Newbie

            Debug LVL apache error_log

             

            [Fri Jan 17 22:01:06 2014] [warn] module proxy_cluster_module is already loaded, skipping

            [Fri Jan 17 22:01:06 2014] [warn] module manager_module is already loaded, skipping

            [Fri Jan 17 22:01:06 2014] [warn] module slotmem_module is already loaded, skipping

            [Fri Jan 17 22:01:06 2014] [warn] module advertise_module is already loaded, skipping

            [Fri Jan 17 22:01:06 2014] [debug] mod_advertise.c(589): [10640 - 10776] in child post config hook

            [Fri Jan 17 22:01:06 2014] [notice] Child 10640: Child process is running

            [Fri Jan 17 22:01:06 2014] [debug] mpm_winnt.c(398): Child 10640: Retrieved our scoreboard from the parent.

            [Fri Jan 17 22:01:06 2014] [info] Parent: Duplicating socket 244 and sending it to child process 10640

            [Fri Jan 17 22:01:06 2014] [debug] mpm_winnt.c(595): Parent: Sent 1 listeners to child 10640

            [Fri Jan 17 22:01:06 2014] [debug] mpm_winnt.c(554): Child 10640: retrieved 1 listeners from parent

            [Fri Jan 17 22:01:06 2014] [debug] proxy_util.c(1818): proxy: grabbed scoreboard slot 0 in child 10640 for worker proxy:reverse

            [Fri Jan 17 22:01:06 2014] [debug] proxy_util.c(1914): proxy: initialized worker 0 in child 10640 for (*) min=0 max=64 smax=64

            [Fri Jan 17 22:01:06 2014] [notice] Child 10640: Acquired the start mutex.

            [Fri Jan 17 22:01:06 2014] [notice] Child 10640: Starting 64 worker threads.

            [Fri Jan 17 22:01:06 2014] [notice] Child 10640: Starting thread to listen on port 6666.

            [Fri Jan 17 22:01:08 2014] [debug] mod_proxy_cluster.c(1010): update_workers_node starting

            [Fri Jan 17 22:01:08 2014] [debug] mod_proxy_cluster.c(1025): update_workers_node done

            [Fri Jan 17 22:01:08 2014] [debug] mod_proxy_cluster.c(1010): update_workers_node starting

            [Fri Jan 17 22:01:08 2014] [debug] mod_proxy_cluster.c(1025): update_workers_node done

            [Fri Jan 17 22:01:14 2014] [debug] mod_manager.c(2322): manager_info request:nonce=d36af0df-1642-8542-9d1d-66b36a2428c8&refresh=10

             

             

             

            nothing suspicious here

            domain.xml

             

            <subsystem xmlns="urn:jboss:domain:modcluster:1.0">
                            <mod-cluster-config advertise-socket="modcluster">
                                <dynamic-load-provider>
                                    <load-metric type="busyness"/>
                                </dynamic-load-provider>
                            </mod-cluster-config>
                        </subsystem>
            
            
            
            
            

             

            socketd binding - ha-sockets

             

            <socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>
            
            
            
            
            

             

            HOWEVER

             

            after changing line in domain.xml to

             

             

            and restartting domain host + nodes, something is going on. I got following errors on nodes (not host) instances:

             

            22:20:51,789 ERROR [org.apache.catalina.core.ContainerBase] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Exception invoking periodic operation: : java.lang.IllegalArgumentException: Node: [3],Name: REMOVED,Balancer: breeze-cluster,LBGroup: ,Host: 192.168.2.159,Port: 8009,Type: ajp,Flushpackets: Off,Flushwait: 10,Ping: 10,Smax: 65,Ttl: 60,Elected: 13,Read: 3146,Transfered: 4589,Connected: 0,Load: 100

            Vhost: [2:1:1], Alias: example.com

            Vhost: [2:1:2], Alias: localhost

            Vhost: [2:1:3], Alias: default-host

            Vhost: [3:1:4], Alias: default-host

            Vhost: [3:1:5], Alias: localhost

            Vhost: [3:1:6], Alias: example.com

            Context: [2:1:1], Context: /WebApiPort, Status: ENABLED

            Context: [3:1:2], Context: /WebApiPort, Status: ENABLED

             

                at org.jboss.modcluster.mcmp.impl.DefaultMCMPResponseParser.parseInfoResponse(DefaultMCMPResponseParser.java:96)

                at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:376)

                at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:347)

                at org.jboss.modcluster.ModClusterService.status(ModClusterService.java:460)

                at org.jboss.modcluster.container.catalina.CatalinaEventHandlerAdapter.lifecycleEvent(CatalinaEventHandlerAdapter.java:239)

                at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:115) [jbossweb-7.0.13.Final.jar:]

                at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1337) [jbossweb-7.0.13.Final.jar:]

                at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1601) [jbossweb-7.0.13.Final.jar:]

                at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590) [jbossweb-7.0.13.Final.jar:]

                at java.lang.Thread.run(Unknown Source) [rt.jar:1.7.0_51]

             

            22:21:01,818 ERROR [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Error [MEM: MEM: Can't read node: {4}] sending command STATUS to proxy 192.168.2.142/192.168.2.142:6666, configuration will be reset

             

            What is going on now?

             

            EDIT right now it all looks the same as described here mod_cluster - Error [MEM: MEM: Can't read node: {4}], but i see there is no answer for this.

             

            After aplying step-2 of domain configuration from http://middlewaremagic.com/jboss/?p=2147 error is gone and cluster started to work as expected, both nodes are exposing webservice interface via singe entry point.

             

            @Michal Thank you for pointing out proxy-list property.

            • 3. Re: Re: Jboss AS 7.1.1Final  + managed domain + modcluster - cluster nodes are not discovered by advertising.
              Jean-Frederic Clere Master

              It looks like an old bug where the node was removed and some context were remaining. Is temporary or does it last long?