When a detector sends out its detection message, it will query the InvokerRegistry (via getServerInvokers()) to get all the server invokers living within its VM (since InvokerRegistry is static, will get server invoker for both Connectors).
ADDED will be fired the first time a detection message is received from a particular detector. If the detection message from that detector ever changes (e.g. a new server invoker is added), an UPDATED event will be fired.
Hope that helps.
How will I know if an invoker was removed or added after that initial detection message?
Example: c1 and c2 again. c1 comes up - I get ADDED message. c2 comes up, I get UPDATED message. If I understand your last post, if c1 goes down, I will get an UPDATED notification again, *not* REMOVED. How do I know what went down?
In sequence it would be:
1) ADDED - notification has invoker array consisting of [c1]
2) UPDATED - notification has invoker array consisting of [c1,c2]
3) UPDATED - notification has invoker array consisting of [c2]
That second UPDATED notification only has c2 in the list - but I would have to remember from a previously fired notification that c1 originally was in that list so I could find out what's missing from the new list (and once I figure out c1 is missing and therefore removed, I can do something about it).
Is that how it works?
I was hoping to find a way that easily, quickly tells me that c1 was REMOVED, rather than me maintain state within my listener (although that's not difficult to do and it seems as though that's what I should do).
I think my misunderstanding stems from what this network registry is registering and what it means when a "server" is ADDED, REMOVED, UPDATED. When I read "server", I was thinking in the context of a Connector. But I think it means "server" as in a JVM server (e.g. the "JBossAS server"). So a "server" JVM is ADDED when it is detected and it is registered under its Identity - any number of connectors can be associated with a "server" (i.e. associated with an Identity). When that JVM goes down, it is REMOVED from the registry. I thought when a Connector goes down it is REMOVED from the registry. But I think I get it now, the new JVM is added and its initial Conenctor comes in witih the ADDED message. But as I add/remove Connectors from that JVM server, UPDATED messages come in.
So, this means I need to maintain a map of Identy's in my notification object so I can figure out what connectors came up and went down when I get these UPDATED messages.
OK, I think it should be easy to do what I think needs to be done. That is, my network notification listener will now maintain a Map of servers (where a "server" is a "server VM that has one or more Connectors).
The Map is keyed on Identity and the value of the map entries is a List of InvokerLocators.
Now - the next question is - when I get a REMOVED notification, will that notification's getLocator() ALWAYS return ALL invoker locators for that server? I think it will because it should - if a server is removed, that would have to mean all of its invokers have also been removed and so the full array of all known invokers should be returned by getLocator().
Similarly, when I get an UPDATED notification, will its getLocator() return ALL invokers currently running in the server? Or will it only give me those invokers that were added or removed? I would think it would have to be the former - you'd have to get the whole list. Getting just those invokers that changed wouldn't help 'cause you wouldn't know if the invoker was removed from the server or added to the server.
I guess I'm just looking for confirmation on this. Making sure that this is really what is supposed to happen.
You are right about how remoting detection views a "server". It is a view of the entire VM instances (regardless the number of mbeanservers and connectors). So when a detector sends its detection message out, it will included every server invoker it could find from the InvokerRegistry (which should be all that exist in that VM). The main reason for doing this was to reduce the amount of network traffic (as if had 10 Connectors running on a server, would be 10 different detection messages otherwise). The problem, as you noted, is will have to manually maintain a list to detect any changes in the number of Connectors on the "server".
So, like you said, if a new Connector is added or one is removed, this will cause an UPDATED event to be fired with the new list of all the Connectors that exist on the "server" at that time.
The REMOVED is a little different (and maybe not best way to do this, so open to suggestions). Periodically, the detector will check its internal list of "servers" and if has not received a detection message in a while from a particular server (which basically acts as a heartbeat), it will perform a ping on that server to check that it is still alive. If this ping fails, it will generate a REMOVED event, which will contain a list of the Connectors it knew about before the failure and this list will be included in the event (basically indicating which Connectors it thinks are now dead).
OK, so that's how I understood it now. The REMOVED means the "server VM" has gone down and all its invokers (that the detector knew about before) will be in the REMOVED notification. That's what I think I need.
Is there a ping timeout configured somewhere? For example, when a "server VM" is killed, the REMOVED notification won't go out until that ping is attempted. How often is that ping sent and is it configurable?
Is any of this documented somewhere? I didn't see it in the NetworkNotification javadocs - should probably add this information in there at least.