14 Replies Latest reply on Sep 5, 2014 2:17 AM by sewatech

    Ugly behavior with default-interface and mod_cluster

    sewatech

      Hi,

       

      I’m trying to setting a cluster of WildFly 8.1.0.Final, with mod_cluster as a front end, and all of this in docker containers. I’ve customized the network configuration of my WildFly, in the standalone-ha.xml. I have 2 interfaces : all and public :

          <interfaces>
               <interface name="public">
                 <nic name="eth0"/>
               </interface>
               <interface name="all">
                 <any-address/>
               </interface>
           </interfaces>
      

      I have explicitly declared the interface of all my socket-bindings :

              <socket-binding name="management-http" port="${jboss.management.http.port:9990}" interface="public" />
               <socket-binding name="management-https" port="${jboss.management.https.port:9993}" interface="public" />
               <socket-binding name="ajp" port="${jboss.ajp.port:8009}" interface="public"/>
               <socket-binding name="http" port="${jboss.http.port:8080}" interface="public"/>
               <socket-binding name="https" port="${jboss.https.port:8443}" interface="public"/>
               <socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700" interface="public"/>
               <socket-binding name="jgroups-tcp" port="7600" interface="public"/>
               <socket-binding name="jgroups-tcp-fd" port="57600" interface="public"/>
               <socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688" interface="public"/>
               <socket-binding name="jgroups-udp-fd" port="54200" interface="public"/>
               <socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364" interface="public"/>
               <socket-binding name="txn-recovery-environment" port="4712" interface="public"/>
              <socket-binding name="txn-status-manager" port="4713" interface="public"/>
      

       

      Now, when I declare the default-interface to all, mod_cluster can discover my instance.

      <socket-binding-group name="standard-sockets" default-interface="all" port-offset="${jboss.socket.binding.port-offset:0} »>
      

      But if I declare the default interface to public, mod_cluster cannot discover my instance.

      <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
      

       

      I thought that the default-interface attribute was the default value for the interface attribute of all the socket-bindings. So why does it has an impact whereas I've declare the interface on all the socket-bindings ?

       

      Alexis

        • 1. Re: Ugly behavior with default-interface and mod_cluster
          jaikiran

          So it's using a wrong interface for the socket binding? Is this just for mod_cluster service or any service relying on the socket binding interface?

          • 2. Re: Ugly behavior with default-interface and mod_cluster
            sewatech

            I really don't know where it come from. I don't see any difference with "netstat -pan".

             

            The problem is only with mod_cluster.

            • 3. Re: Ugly behavior with default-interface and mod_cluster
              rhusar

              Hi Alexis,

               

              Just trying to make sense of this, so the only difference is whether you explicitly or implicitly define interface for the modcluster socket binding? At least it looks like this is trivial to workaround.

               

              Can you try with lastest upstream master as well?

              • 4. Re: Ugly behavior with default-interface and mod_cluster
                sewatech

                No, in both cases the interface of modcluster is explicit. The only difference is on the default-interface.

                 

                I'll give a try to the upstream and come back soon.

                • 5. Re: Ugly behavior with default-interface and mod_cluster
                  rhusar

                  The code actually does get the default interfce (from the SocketBindingManager) if none is specified for that particular binding:

                   

                  The code checks if the interface is configured expliticly, and only then uses the default one.

                      public InetAddress getAddress() {

                          return networkInterface != null ? networkInterface.getAddress() : socketBindings.getDefaultInterfaceAddress();

                      }

                  • 6. Re: Ugly behavior with default-interface and mod_cluster
                    sewatech

                    This piece of code makes sense. But it does not fit to my test. The default interface is probably used somewhere else.

                     

                    Same behaviour with wildfly-9.0.0.Alpha1-SNAPSHOT from master.

                     

                    I can share Docker images if it helps.

                    • 7. Re: Ugly behavior with default-interface and mod_cluster
                      rhusar

                      So I tried to reproduce the problem, but all works as expected.

                       

                      I have set modcluster apache module to advertise on my tunnel interface (tun) by configuring AdvertiseBindAddress. I have configured another interface local (lo). Multicasts from one don't get forwarded to another.

                       

                      Then tried several combinations of setting interface in WildFly 8.1:

                       

                      • default-interface=lo + no mod_cluster socket binding interface => no discovery
                      • default-interface=lo + mod_cluster socket binding interface=tun => discovery OK
                      • default-interface=tun + no mod_cluster socket binding interface => discovery OK
                      • default-interface=tun + mod_cluster socket binding interface=tun => discovery OK

                       

                      All works as expected. Does this test reenact your test?

                      • 8. Re: Ugly behavior with default-interface and mod_cluster
                        sewatech

                        I do not have a tun interface but I'm using eth0. I have lo too (loopback, 127.0.0.1). This is based on the default network configuration of docker.

                        • default-interface=lo + no mod_cluster socket binding interface => no discovery
                        • default-interface=lo + mod_cluster socket binding interface=eth0 => no discovery
                        • default-interface=eth0 + no mod_cluster socket binding interface => no discovery
                        • default-interface=eth0 + mod_cluster socket binding interface=eth0 => no discovery

                         

                        The only way to have discovery is to have eth0+lo as default interface.

                        • default-interface=eth0+lo + no mod_cluster socket binding interface => discovery OK
                        • default-interface=eth0+lo + mod_cluster socket binding interface=eth0 => discovery OK
                        • 9. Re: Ugly behavior with default-interface and mod_cluster
                          rhusar

                          Can you elaborate what you mean by

                          default-interface=eth0+lo

                          ?

                          • 10. Re: Re: Ugly behavior with default-interface and mod_cluster
                            sewatech

                            eth0 + lo means

                             

                                    <interface name="all">
                                      <nic name="eth0"/>
                                      <nic name="lo"/>
                                    </interface>
                            

                            or

                             

                                    <interface name="all">
                                      <any-address/>
                                    </interface>
                            
                            • 11. Re: Re: Ugly behavior with default-interface and mod_cluster
                              rhusar

                              eth0 + lo means

                               

                               

                              1.         <interface name="all"> 
                              2.           <nic name="eth0"/> 
                              3.           <nic name="lo"/> 
                              4.         </interface> 

                               

                              I am afraid this is same as specifying one nic being "lo". I need to double check this, specically talking about org.jboss.as.server.parsing.CommonXml#parseSimpleInterfaceCriterion, but unless you are using nested <any> or <not> the parsing logic seems to just look for the configuration elements and the resulting model will only pick up the last one which overrides any previous values. This looks like a usability bug that it doesn't tell you that this configuration is not doing what you might have expected.

                               

                              Try to look at the resulting model.

                               

                              [standalone@localhost:9990 /] /interface=all:read-resource

                              {

                                  "outcome" => "success",

                                  "result" => {

                                      "any" => undefined,

                                      "any-address" => undefined,

                                      "any-ipv4-address" => undefined,

                                      "any-ipv6-address" => undefined,

                                      "inet-address" => expression "${jboss.bind.address:127.0.0.1}",

                                      "link-local-address" => undefined,

                                      "loopback" => undefined,

                                      "loopback-address" => undefined,

                                      "multicast" => undefined,

                                      "name" => "all",

                                      "nic" => "lo",

                                      "nic-match" => undefined,

                                      "not" => undefined,

                                      "point-to-point" => undefined,

                                      "public-address" => undefined,

                                      "site-local-address" => undefined,

                                      "subnet-match" => undefined,

                                      "up" => undefined,

                                      "virtual" => undefined

                                  }

                              }

                               

                              Nevertheless, this shouldn't be affecting the problem we are talking about here with mod_cluster not picking the right interface.

                              • 12. Re: Re: Ugly behavior with default-interface and mod_cluster
                                sewatech

                                You're totally right. lo or eth0 + lo are the same. Have to replay my tests...

                                 

                                Updated result :

                                • default-interface=lo + no mod_cluster socket binding interface => no discovery
                                • default-interface=lo + mod_cluster socket binding interface=eth0 => discovery OK
                                • default-interface=eth0 + no mod_cluster socket binding interface => no discovery
                                • default-interface=eth0 + mod_cluster socket binding interface=eth0 => no discovery
                                • 13. Re: Re: Ugly behavior with default-interface and mod_cluster
                                  rhusar

                                  Can you post your XML with the socket bindings? It looks like it's the problem when you have multiple <nic>s in your <interface> and thus only the last one is effectively used.

                                  • 14. Re: Re: Re: Ugly behavior with default-interface and mod_cluster
                                    sewatech

                                    My initial config (see first post) didn't have multiple nic. Neither my latest :

                                     

                                        <interfaces>
                                            <interface name="public">
                                                <nic name="eth0"/>
                                            </interface>
                                            <interface name="local">
                                                <loopback/>
                                            </interface>
                                        </interfaces>
                                    
                                    
                                        <socket-binding-group name="standard-sockets" default-interface="local" port-offset="${jboss.socket.binding.port-offset:0}">
                                            <socket-binding name="management-http" interface="public" port="${jboss.management.http.port:9990}"/>
                                            <socket-binding name="management-https" interface="local" port="${jboss.management.https.port:9993}"/>
                                            <socket-binding name="ajp" interface="public" port="${jboss.ajp.port:8009}"/>
                                            <socket-binding name="http" interface="local" port="${jboss.http.port:8080}"/>
                                            <socket-binding name="https" interface="local" port="${jboss.https.port:8443}"/>
                                            <socket-binding name="jgroups-mping" interface="local" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
                                            <socket-binding name="jgroups-tcp" interface="local" port="7600"/>
                                            <socket-binding name="jgroups-tcp-fd" interface="local" port="57600"/>
                                            <socket-binding name="jgroups-udp" interface="public" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
                                            <socket-binding name="jgroups-udp-fd" interface="public" port="54200"/>
                                            <socket-binding name="modcluster" interface="public" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>
                                            <socket-binding name="txn-recovery-environment" interface="local" port="4712"/>
                                            <socket-binding name="txn-status-manager" interface="local" port="4713"/>
                                            <outbound-socket-binding name="mail-smtp">
                                                <remote-destination host="localhost" port="25"/>
                                            </outbound-socket-binding>
                                        </socket-binding-group>