7 Replies Latest reply on Apr 15, 2010 4:33 AM by timfox

    Can we get auto discovery across subnets?

    drkirwin

      Folks--

       

      In short: I'm hoping there's a way to get the multicast auto-discovery stuff to cross subnet boundaries.

       

      Some context:

       

      I have a production set up in which we have two subnets.

       

      1. HornetQ clients
      2. HornetQ itself

       

      The reason we have two subnets is because we're transmitting sensitive data and our security folks require that there be a firewall between the clients that transmit the data and the points where the data might be written down (including databases). Our assumption is that under certain conditions, HornetQ may need to write down the messages.

       

      (For the most part, we need to be able to demonstrate certain things to certain review bodies, so this isn't really a discussion about whether we really, really need to do this. There's no arguing with certain industry watchdogs over the "spirit" of a constraint rather than the "letter".)

       

      Anyway, we're using multicast in another context in this same environment, and the way we get it to work (in Java) is that we set TTL on the multicast socket to 32.

       

      Here's a capture from the Hornet Q socket capture:

       

      IP:   ----- IP Header -----
      IP:   
      IP:   Version = 4
      IP:   Header length = 20 bytes
      IP:   Type of service = 0x00
      IP:         xxx. .... = 0 (precedence)
      IP:         ...0 .... = normal delay
      IP:         .... 0... = normal throughput
      IP:         .... .0.. = normal reliability
      IP:         .... ..0. = not ECN capable transport
      IP:         .... ...0 = no ECN congestion experienced
      IP:   Total length = 1164 bytes
      IP:   Identification = 1523
      IP:   Flags = 0x0
      IP:         .0.. .... = may fragment
      IP:         ..0. .... = last fragment
      IP:   Fragment offset = 2960 bytes
      IP:   Time to live = 1 seconds/hops
      IP:   Protocol = 17 (UDP)
      IP:   Header checksum = a34e
      IP:   Source address = 192.16.18.159, 192.16.18.159
      IP:   Destination address = 231.7.7.7, 231.7.7.7
      IP:   No options
      IP:   
      UDP:  [1144 byte(s) of data, continuation of IP ident=1523]
      

       

      According to my colleague in Ops: "Here is the IP header from a multicast packet originating from the hornetq service.  Looking at the time to live header line you see it’s set to 1. This is the java default and we need to modify that so multicast travels across gateways.  Currently in our locally developed apps we set this to 32."

       

      Here is where I see the multicast socket creation in the source code.

       

      Line 111 in file ./src/main/org/hornetq/core/cluster/impl/DiscoveryGroupImpl.java

       

         socket = new MulticastSocket(groupPort);
      

       

      In our apps, we "fix" this problem by doing something like:

       

         socket.setTimeToLive($ttl_integer);
      

       

      Is there a way we can add this as part of the cluster configuration?

       

      Is there a way to get this working immediately in the GA release?

       

      Keith

        • 1. Re: Can we get auto discovery across subnets?
          ataylor

          we could probably support adding extra params in the config something like:

           

          <discovery-groups>
                <discovery-group name="my-discovery-group">
                   <group-address>231.7.7.7</group-address>
                   <group-port>9876</group-port>
                   <refresh-timeout>10000</refresh-timeout>
                   <socket-params>
                      <param name="timeToLive" value="99999" type="Integer"/>
                   </socket-params>
                </discovery-group>
             </discovery-groups>

           

           

          supporting setting extra params on the socket.

           

          Feel free to raise a jira altho we are pretty tied up at the minute so i am not sure when it would get implemented. You could always provide the patch yourselves and contribute?

          • 2. Re: Can we get auto discovery across subnets?
            timfox

            I am actually working in this area at the moment, so perhaps I can add this param too while I am here.

            • 3. Re: Can we get auto discovery across subnets?
              drkirwin

              Cool, guys. Thanks. If you're already working in that area, that's great. I don't think I could come up to speed on the configuration system soon enough to contribute a patch right away. Then again, I've not even looked at it. Maybe it's super easy!

               

              Right now, I have a build where I hard coded in a TTL of 32, and we're going to test it to see if it even works like we think it will.

               

              I'll report back here when we've completed the test.

               

              Thanks again!

               

              Keith

              • 4. Re: Can we get auto discovery across subnets?
                drkirwin

                We set the "setTimeToLive" method to "32", but when we do a tcpdump, we see that TTL is set to 1, still. Very odd.

                 

                Keith

                • 5. Re: Can we get auto discovery across subnets?
                  drkirwin

                  Okay, so the DiscoveryGroupImpl class is for receiving discovery information. The class uses a MulticastSocket to accomplish its task.

                   

                  The BroadcastGroupImpl class is for sending it. This class uses the DatagramSocket to send packets. Unfortunately, this class doesn't allow the setting of a TTL value. The DatagramSocket class has an underlying DatagramSocketImpl which has the value and a setting, but it can't be accessed from outside the socket class itself. It's an abstract class, so it can't be subclassed either.

                   

                  Hm.

                   

                  I wonder why HornetQ doesn't use a MulticastSocket for broadcasting?

                  • 6. Re: Can we get auto discovery across subnets?
                    drkirwin

                    Okay, I got it all to work replacing DatagramSocket in BroadcastGroupImpl with MulticastSocket, and then set TimeToLive to 32, and I can now see that value broadcast using tcpdump.

                     

                    I suppose I can make up a patch, but it's bound to not quite fit in with the code's culture. Should I attach a patch here just for illustration (i.e., less proper configuration)?

                     

                    Keith

                    • 7. Re: Can we get auto discovery across subnets?
                      timfox

                      Feel free to open a JIRA and add a patch, but it needs also to include a means to configure the setting from the broad cast group configuration in hornetq-configuration.xml - that's where most of the work is, and writing a test.

                       

                      Just changing the code to use a MulticastSocket is the easy bit and probably not worth posting a patch for