View Changed JGroupsAddress wrong address
Hi, I have a problem with the ViewChanged notification providing a 'wrong' address. In general, addresses are provided in the following format, e.g., ip-10-81-128-189-29581, but occassionally we see UUID like addresses 0fbd6496-9b37-f2ca-3c93-590d6b54324e. We are then attempting to convert this to an IP address using JGroups, which fails
DefaultCacheManager dcm = (DefaultCacheManager)CacheManager.getInstance().getCacheManager();
JGroupsTransport jgt = (JGroupsTransport)dcm.getTransport();
Object addr = jgt.getChannel().getProtocolStack().down(new Event(Event.GET_PHYSICAL_ADDRESS, ((JGroupsAddress)jgroupsAddress).getJGroupsAddress()));
This is occurring in a cluster with 3+ members, and when a member joins, one other member of the cluster will often get an address as above. Subsequently, if another node joins the cluster, the cluster view that originally received the UUID address, will then have the address of that node (represented by the UUID) in the normal format.
Is this a bug? if not, how can the UUID address be converted to an IP address?
Our environment is:
- CentOS 6.2
- We are using a GossipRouter to manage our cluster.
- Infinispan 5.1.3.FINAL (JGroups 3.0.8.FINAL).
- Java 1.6.30
It actually looks like an issue with JGroups, but as we are interfacing with it via Infinispan I am posting here.
Our gossip router config is:
<?xml version="1.0" encoding="UTF-8"?>
<config>
<TCP bind_port="7900"/>
<TCPGOSSIP timeout="3000" initial_hosts="10.81.0.227[8800]" num_initial_members="3"/>
<MERGE2 max_interval="30000" min_interval="10000"/>
<FD_SOCK/>
<FD timeout="10000" max_tries="5"/>
<VERIFY_SUSPECT timeout="1500"/>
<pbcast.NAKACK use_mcast_xmit="false" retransmit_timeout="300,600,1200,2400,4800" discard_delivered_msgs="true"/>
<UNICAST timeout="300,600,1200,2400,3600"/>
<pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000" max_bytes="400000"/>
<pbcast.GMS print_local_addr="true" join_timeout="3000" view_bundling="true"/>
<UFC max_credits="2000000" min_threshold="0.10"/>
<MFC max_credits="2000000" min_threshold="0.10"/>
<FRAG2 frag_size="60000"/>
</config>
Here is some pared down logging from our servers:
10.81.0.227
2012-04-11 22:17:58,123 [notification-thread-0] INFO [JmsCluster.viewChanged() 375] - ip-10-81-128-189-29581 [NODE|ATTACHED]
2012-04-11 22:17:58,125 [notification-thread-0] INFO [JmsCluster.getInetAddress() 457] - getInetAddress ip-10-81-128-189-29581 => 10.81.128.189
2012-04-11 22:18:20,054 [notification-thread-0] INFO [JmsCluster.viewChanged() 388] - ip-10-81-208-97-19626 [NODE|DETACHED]
2012-04-11 22:18:20,073 [notification-thread-0] INFO [JmsCluster.getInetAddress() 457] - getInetAddress ip-10-81-208-97-19626 => 10.81.208.97
2012-04-11 22:18:45,515 [notification-thread-0] INFO [JmsCluster.viewChanged() 375] - ip-10-81-208-97-20403 [NODE|ATTACHED]
2012-04-11 22:18:45,518 [notification-thread-0] INFO [JmsCluster.getInetAddress() 457] - getInetAddress ip-10-81-208-97-20403 => 10.81.208.97
10.81.128.189
2012-04-11 22:18:04,686 [notification-thread-0] INFO [JmsCluster.viewChanged() 375] - ip-10-81-128-189-29581 [NODE|ATTACHED]
2012-04-11 22:18:04,689 [notification-thread-0] INFO [JmsCluster.getInetAddress() 457] - getInetAddress ip-10-81-128-189-29581 => 10.81.128.189
2012-04-11 22:18:20,402 [notification-thread-0] INFO [JmsCluster.viewChanged() 388] - ip-10-81-208-97-19626 [NODE|DETACHED]
2012-04-11 22:18:20,404 [notification-thread-0] INFO [JmsCluster.getInetAddress() 457] - getInetAddress ip-10-81-208-97-19626 => 10.81.208.97
2012-04-11 22:18:45,640 [notification-thread-0] INFO [JmsCluster.viewChanged() 375] - 0fbd6496-9b37-f2ca-3c93-590d6b54324e [NODE|ATTACHED]
2012-04-11 22:18:45,640 [notification-thread-0] INFO [JmsCluster.getInetAddress() 437] - getInetAddress 0fbd6496-9b37-f2ca-3c93-590d6b54324e
2012-04-11 22:18:45,641 [notification-thread-0] INFO [JmsCluster.getInetAddress() 441] - getInetAddress, DefaultCacheManager view: members: [ip-10-81-0-227-2930, ip-10-81-128-189-29581, 0fbd6496-9b37-f2ca-3c93-590d6b54324e]
2012-04-11 22:18:45,641 [notification-thread-0] INFO [JmsCluster.getInetAddress() 444] - getInetAddress, JGroupsTransport view: members: [ip-10-81-0-227-2930, ip-10-81-128-189-29581, 0fbd6496-9b37-f2ca-3c93-590d6b54324e], addresses: [10.81.128.189:7800]
2012-04-11 22:18:45,642 [notification-thread-0] INFO [JmsCluster.getInetAddress() 457] - getInetAddress 0fbd6496-9b37-f2ca-3c93-590d6b54324e => null
2012-04-11 22:37:05,509 [notification-thread-0] INFO [JmsCluster.viewChanged() 388] - ip-10-81-0-227-2930 [NODE|DETACHED]
2012-04-11 22:37:05,512 [notification-thread-0] INFO [JmsCluster.getInetAddress() 437] - getInetAddress ip-10-81-0-227-2930
2012-04-11 22:37:05,512 [notification-thread-0] INFO [JmsCluster.getInetAddress() 441] - getInetAddress, DefaultCacheManager view: members: [ip-10-81-128-189-29581, ip-10-81-208-97-20403]
2012-04-11 22:37:05,513 [notification-thread-0] INFO [JmsCluster.getInetAddress() 444] - getInetAddress, JGroupsTransport view: members: [ip-10-81-128-189-29581, ip-10-81-208-97-20403], addresses: [10.81.128.189:7800]
2012-04-11 22:37:05,514 [notification-thread-0] INFO [JmsCluster.getInetAddress() 457] - getInetAddress ip-10-81-0-227-2930 => 10.81.0.227
the later logging shows that 0fbd6496-9b37-f2ca-3c93-590d6b54324e represents ip-10-81-208-97-20403
10.81.208.97
2012-04-11 22:18:53,257 [notification-thread-0] INFO [JmsCluster.viewChanged() 375] - ip-10-81-208-97-20403 [NODE|ATTACHED]
2012-04-11 22:18:53,260 [notification-thread-0] INFO [JmsCluster.getInetAddress() 457] - getInetAddress ip-10-81-208-97-20403 => 10.81.208.97
2012-04-11 22:37:18,178 [notification-thread-0] INFO [JmsCluster.viewChanged() 388] - ip-10-81-128-189-29581 [NODE|DETACHED]
2012-04-11 22:37:18,180 [notification-thread-0] INFO [JmsCluster.getInetAddress() 457] - getInetAddress ip-10-81-128-189-29581 => 10.81.128.189