Jboss AS 7.1.1Final + managed domain + modcluster - cluster nodes are not discovered by advertising.

[sebastian.choina]

Hello

I want to setup a basic cluster.

I have one domain controller on my local host (192.168.2.142 windows) and two virtualboxed slaves (192.168.2.159, 192.168.2.160 puppy linux). As for the domain itself, everything works just fine. Project deployed via CLI on domain controller populates slaves and work just fine one both of them.

I have configured mod_cluster 1.2.6 final for windows downloaded from mod_cluster 1.2.6.Final bin Downloads - JBoss Community. It is deployed on the same machine as domain controller (192.168.2.142).

I have checked network configuration for local and virtual instances and it is just fine. I have checked, that UDP brodcasts can be received on virtual systems.

Unfortunetly none of cluster nodes are resolved for load balancing. What am I missing? Acording to official howto's it should work due to fact, that JBoss 7.1.1 has modcluster supperort working out-of-the-box.

Here are some of my configuration files:

httpd.conf

  Listen 192.168.2.142:6666
  ManagerBalancerName breeze-cluster
  <VirtualHost 192.168.2.142:6666>
    <Location />
     Order deny,allow
     Deny from all
     Allow from all
    </Location>

    KeepAliveTimeout 300
    MaxKeepAliveRequests 0
    ServerAdvertise on http://192.168.2.142:6666
    AdvertiseFrequency 5
    #AdvertiseSecurityKey secret
    #AdvertiseGroup 224.0.1.105:23364
    EnableMCPMReceive

    <Location /mod_cluster_manager>
       SetHandler mod_cluster-manager
       Order deny,allow
       Deny from all
       Allow from all
    </Location>

  </VirtualHost>

</

domain.xml

<server-groups>
  <server-group name="breeze-cluster" profile="ha">
    <socket-binding-group ref="ha-sockets"/>
    <deployments>
      <deployment name="WebApiPort.war" runtime-name="WebApiPort.war"/>
    </deployments>
  </server-group>
</server-groups>

And this is raport from cluster manager:

[mbabacek]

Hello Seb,

could you go through this basic sanity-check list and report back the results, please?

• Are you sure you've got ha or full-ha profile active for your hosts? (It looks like so, form the snippet you provided though...)
• Could you copy-paste here the modcluster module configuration that is active? (either from ha or full-ha profile)
• What can you see in the host's log? Is there anything like "Listening to proxy advertisements on"?
• Let's not trust UDP multicast in VirtualBox for a moment, and type in the proxy-list="192.168.2.142:6666" as an attribute to modcluster module configuration element (ha or full-ha profile). Does it change anything?
• Use LogLevel Debug in your httpd.conf. What can you see in the error_log? Aren't there any suspicious messages about balancer not being able to reach its workers?

_K-

EDIT: Obviously, it's proxy-list, not proxy-lisy :-)

[sebastian.choina]

Debug LVL apache error_log

[Fri Jan 17 22:01:06 2014] [warn] module proxy_cluster_module is already loaded, skipping

[Fri Jan 17 22:01:06 2014] [warn] module manager_module is already loaded, skipping

[Fri Jan 17 22:01:06 2014] [warn] module slotmem_module is already loaded, skipping

[Fri Jan 17 22:01:06 2014] [warn] module advertise_module is already loaded, skipping

[Fri Jan 17 22:01:06 2014] [debug] mod_advertise.c(589): [10640 - 10776] in child post config hook

[Fri Jan 17 22:01:06 2014] [notice] Child 10640: Child process is running

[Fri Jan 17 22:01:06 2014] [debug] mpm_winnt.c(398): Child 10640: Retrieved our scoreboard from the parent.

[Fri Jan 17 22:01:06 2014] [info] Parent: Duplicating socket 244 and sending it to child process 10640

[Fri Jan 17 22:01:06 2014] [debug] mpm_winnt.c(595): Parent: Sent 1 listeners to child 10640

[Fri Jan 17 22:01:06 2014] [debug] mpm_winnt.c(554): Child 10640: retrieved 1 listeners from parent

[Fri Jan 17 22:01:06 2014] [debug] proxy_util.c(1818): proxy: grabbed scoreboard slot 0 in child 10640 for worker proxy:reverse

[Fri Jan 17 22:01:06 2014] [debug] proxy_util.c(1914): proxy: initialized worker 0 in child 10640 for (*) min=0 max=64 smax=64

[Fri Jan 17 22:01:06 2014] [notice] Child 10640: Acquired the start mutex.

[Fri Jan 17 22:01:06 2014] [notice] Child 10640: Starting 64 worker threads.

[Fri Jan 17 22:01:06 2014] [notice] Child 10640: Starting thread to listen on port 6666.

[Fri Jan 17 22:01:08 2014] [debug] mod_proxy_cluster.c(1010): update_workers_node starting

[Fri Jan 17 22:01:08 2014] [debug] mod_proxy_cluster.c(1025): update_workers_node done

[Fri Jan 17 22:01:08 2014] [debug] mod_proxy_cluster.c(1010): update_workers_node starting

[Fri Jan 17 22:01:08 2014] [debug] mod_proxy_cluster.c(1025): update_workers_node done

[Fri Jan 17 22:01:14 2014] [debug] mod_manager.c(2322): manager_info request:nonce=d36af0df-1642-8542-9d1d-66b36a2428c8&refresh=10

nothing suspicious here

domain.xml

<subsystem xmlns="urn:jboss:domain:modcluster:1.0">
                <mod-cluster-config advertise-socket="modcluster">
                    <dynamic-load-provider>
                        <load-metric type="busyness"/>
                    </dynamic-load-provider>
                </mod-cluster-config>
            </subsystem>

socketd binding - ha-sockets

<socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>

HOWEVER

after changing line in domain.xml to

and restartting domain host + nodes, something is going on. I got following errors on nodes (not host) instances:

22:20:51,789 ERROR [org.apache.catalina.core.ContainerBase] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Exception invoking periodic operation: : java.lang.IllegalArgumentException: Node: [3],Name: REMOVED,Balancer: breeze-cluster,LBGroup: ,Host: 192.168.2.159,Port: 8009,Type: ajp,Flushpackets: Off,Flushwait: 10,Ping: 10,Smax: 65,Ttl: 60,Elected: 13,Read: 3146,Transfered: 4589,Connected: 0,Load: 100

Vhost: [2:1:1], Alias: example.com

Vhost: [2:1:2], Alias: localhost

Vhost: [2:1:3], Alias: default-host

Vhost: [3:1:4], Alias: default-host

Vhost: [3:1:5], Alias: localhost

Vhost: [3:1:6], Alias: example.com

Context: [2:1:1], Context: /WebApiPort, Status: ENABLED

Context: [3:1:2], Context: /WebApiPort, Status: ENABLED

    at org.jboss.modcluster.mcmp.impl.DefaultMCMPResponseParser.parseInfoResponse(DefaultMCMPResponseParser.java:96)

    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:376)

    at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.status(DefaultMCMPHandler.java:347)

    at org.jboss.modcluster.ModClusterService.status(ModClusterService.java:460)

    at org.jboss.modcluster.container.catalina.CatalinaEventHandlerAdapter.lifecycleEvent(CatalinaEventHandlerAdapter.java:239)

    at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:115) [jbossweb-7.0.13.Final.jar:]

    at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1337) [jbossweb-7.0.13.Final.jar:]

    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1601) [jbossweb-7.0.13.Final.jar:]

    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1590) [jbossweb-7.0.13.Final.jar:]

    at java.lang.Thread.run(Unknown Source) [rt.jar:1.7.0_51]

22:21:01,818 ERROR [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Error [MEM: MEM: Can't read node: {4}] sending command STATUS to proxy 192.168.2.142/192.168.2.142:6666, configuration will be reset

What is going on now?

EDIT right now it all looks the same as described here mod_cluster - Error [MEM: MEM: Can't read node: {4}], but i see there is no answer for this.

After aplying step-2 of domain configuration from http://middlewaremagic.com/jboss/?p=2147 error is gone and cluster started to work as expected, both nodes are exposing webservice interface via singe entry point.

@Michal Thank you for pointing out proxy-list property.

[jfclere]

It looks like an old bug where the node was removed and some context were remaining. Is temporary or does it last long?