JBAS-2394 - multicast addr not working

[adrian.brock]

I ran Branch_4_0 with -u 228.1.2.4 and looked at the debug logging
here's the output and config:

It claims it is using the config:

2005-10-21 11:07:34,841 DEBUG [org.jboss.ha.framework.server.ClusterPartition] Setting JGProps from xml to: UDP(ip_mcast=true;ip_ttl=8;loopback=false;mcast_addr=228.1.
2.4;mcast_port=45566;mcast_recv_buf_size=150000;mcast_send_buf_size=800000;ucast_recv_buf_size=150000;ucast_send_buf_size=800000):PING(down_thread=true;num_initial_mem
bers=3;timeout=2000;up_thread=true):MERGE2(max_interval=20000;min_interval=10000):FD(down_thread=true;max_tries=5;shun=true;timeout=2500;up_thread=true):VERIFY_SUSPECT
(down_thread=true;num_msgs=3;timeout=3000;up_thread=true):pbcast.NAKACK(down_thread=true;gc_lag=50;max_xmit_size=8192;retransmit_timeout=300,600,1200,2400,4800;up_thre
ad=true):UNICAST(down_thread=true;min_threshold=10;timeout=300,600,1200,2400,4800;window_size=100):pbcast.STABLE(desired_avg_gossip=20000;down_thread=true;up_thread=tr
ue):FRAG(down_thread=true;frag_size=8192;up_thread=true):pbcast.GMS(join_retry_timeout=2000;join_timeout=5000;print_local_addr=true;shun=true):pbcast.STATE_TRANSFER(do
wn_thread=true;up_thread=true)

and then started Branch_3_2 (with ports-01)

11:14:44,293 INFO [UDP] unicast sockets will use interface 65.244.175.186
11:14:44,299 INFO [UDP] socket information:
local_addr=dev22:32805 (additional data: 19 bytes), mcast_addr=228.1.2.3:45566, bind_addr=/65.244.175.186, ttl=8
sock: bound to 65.244.175.186:32805, receive buffer size=131071, send buffer size=131071
mcast_recv_sock: bound to 65.244.175.186:45566, send buffer size=131071, receive buffer size=131071
mcast_send_sock: bound to 65.244.175.186:32806, send buffer size=131071, receive buffer size=131071
11:14:44,301 INFO [STDOUT]
-------------------------------------------------------
GMS: address is dev22:32805 (additional data: 19 bytes)
-------------------------------------------------------
11:14:46,377 INFO [DefaultPartition] Number of cluster members: 2
11:14:46,377 INFO [DefaultPartition] Other members: 1
11:14:46,377 INFO [DefaultPartition] Fetching state (will wait for 60000 milliseconds):
11:14:46,378 INFO [DefaultPartition] New cluster view for partition DefaultPartition: 1 ([65.244.175.186:1099, 65.244.175.186:1199] delta: 0)
11:14:46,386 INFO [DefaultPartition] I am (null) received membershipChanged event:
11:14:46,386 INFO [DefaultPartition] Dead members: 0 ([])
11:14:46,386 INFO [DefaultPartition] New Members : 0 ([])
11:14:46,386 INFO [DefaultPartition] All Members : 2 ([65.244.175.186:1099, 65.244.175.186:1199])

So why are these forming a cluster?

[adrian.brock]

From the bug report:

More bug info:

It works if you change the *port* for the 3.2.5 instance. So, this implies that in Linux at least, it doesn't matter which multicast IP you use, the messages will go to all IPs that bind to that port. This means we need another option on the command line like '-up' for the port.

You can't put everything on one port if they share the same address.

It is more likely that the address is not propogating down to the socket config
or we are both making the same stupid mistake?

[dsicore]

After discussing this issue with Ben Wang, he claims that he only sees this problem on Linux.


Adrian what OS are you using?

[adrian.brock]

After discussing this issue with Ben Wang, he claims that he only sees this problem on Linux.


Adrian what OS are you using?

adrian@dev22:~/jboss-4.0> uname -a
Linux dev22 2.6.5-7.201-smp #1 SMP Thu Aug 25 06:20:45 UTC 2005 x86_64 x86_64 x86_64 GNU/Linux

[adrian.brock]

This is not a jgroups issue, or if it is, it is a common problem.
I also see HAJNDI discovery (which is just multicast)
incorrectly responding on the default group (230.0.0.4)

12:11:15,686 INFO [DetachedHANamingService$AutomaticDiscovery] Listening on /0.0.0.0:1102, group=228.1.2.4, HA-JNDI address=65.244.175.186:1100

[ben.wang]

Since I don't use Linux myslef, I only run into this once or twice. :-) I think Bela is also aware of this issue as well, if I remeber it correctly. So he should have an explanation.

-Ben

[adrian.brock]

I'm going to close the bug report as "won't fix" because of this:
http://bugs.sun.com/bugdatabase/view_bug.do;:YfiG?bug_id=4701650
It would be interesting to see what extra information Bela has.

I'll raise a feature request to look into an easily configurable port option.
But I suspect this is less easy to do in a backwardly compatible way if we
have mutliple services specifying the same address?

And a comment on that bug report suggests that even the port might not work
on all platforms :-(

With JDK 1.4 on Linux Kernel 2.4, binding to 0.0.0.0 with a
port results in all packets destined for that port
(regardless of address) being received by the client.

Maybe I can find time to do some Linux Kernel Hacking ;-)

[adrian.brock]

For future reference, the issue is that if two java applications on the same
machine join different groups, they see each other's traffic.
But only if they have the same port.

I tried setting different interfaces but that makes no difference.
So the port change feature request makes sense.

I can't find any reference to this behaviour except on Sun's bug report.
So I wonder whether this is really a Linux issue and not a JVM issue?

[belaban]

So you have appl A1 joining 230.1.2.3:7500 and appl A2 joining 230.5.6.7:7500. Both are on the same host and the interface used is 192.168.0.10.

When you multicast to 230.1.2.3:7500 only A1 will receive those multicasts, there is *no way* that A2 will receive it. The hashing of the class D address on the NIC level may be imperfect, so that a multicast packet is received, where it shouldn't, but the IP layer will discard the packet if it is not destined for the multicast address we joined on the local host.
Note, however, that if you send the same packet to 192.168.0.10:7500, then both applications (or only 1 of them) may receive the packet, depending on the IP implementation.

[belaban]

Adrian, you're running Linux 2.6 on *dev22*. I didn't find dev22 in the Atlanta labs, so I assume this is your own box ?
If you give me an account on it, I can run my JGroups test programs to verify this (would be horrible if true)...

[adrian.brock]

Yes, it is "my" box. I can get you an account if you like or just ask Eric.

Like I said, I tried using different interfaces. It made no difference.

The issue is not really about two listeners. It is about the sender and receiver
joining different groups but the groups being "cummulative" - quoted from Sun's bug
report.

[belaban]

The bug report is for JDK 1.4, maybe this was fixed for 5 (although the report mentions Mustang...).
Okay, so if we bind to the wildcard 0.0.0.0 address then instead of the OS picking the interface, we receive packets from *all* interfaces. Hmm, in this case we need to communicate this to our customers and tell them to always pick different ports, not just different mcast addresses.
I'll put this on the wiki and let support know.
Thanks,

[belaban]

This is captured in http://wiki.jboss.org/wiki/Wiki.jsp?page=PromiscuousTraffic

[dsicore]

Just for the record:

I've tested this using 1.4 running the 3.2.6 AS and the second node being 1.5 running 4.0.3 final.

In the 1.4/3.2.6 node, the cluster service is using the:

bind_addr="67.19.124.42" in the cluster-service.xml

In the 1.5/4.0.3 node, I use the '-b' but, I don't explicitly set the bind_addr in any of the multiple jgroups configs. Is this what you mean, Bela, by binding to 0.0.0.0?

Also, is there any benefit to adding a bind_addr="${jboss.bind.address:0.0.0.0}" in each of the jgroups configurations, and having that be the default that ships with the server? It seems like I'm always adding it anyway with each node deployment I do.

[adrian.brock]

All interface configuration should be using that pattern!