What problem are we trying to solve?
Currently when using TCP stack in domain mode, we need to specify all servers in JGroups configuration. This is unnecessary since the Domain Controller hold full list of servers in a domain. More information might be found in WFLY-1066 .
- JGroups subsystem need to receive notifications about newly registered (and unregistered) servers per server group
- Notifications might have some delay (but not too big, probably less than 30 seconds)
- Notifications should work properly during HA failover (manual failover is fine)
- A server should be able to pull a full list of servers (let's call it members list) within a domain on demand (e.g. a server crashed and we have no guarantee what is the current state; we should be able to pull a new members list from a Domain Controller)
- Pulled members list doesn't have to be persistent
- Notifications as well as members list should contain expanded and resolved list of IP addresses (not symbolic names)
Why the simplest approach won't work?
- The simplest approach is to query Domain Controller and ask for a list. This approach has serious scalability problems is large domains (10k nodes). Network traffic overhead will be tremendous.
Design proposal #1: The service registry
- We will create a registry on the top of Domain Model
- The registry will be a a key/value map
- Each server will maintain its local copy of the registry
- Since Domain Model is already replicated to all nodes, adding a new registry won't affect scalability too much. However there is always a concern what will happen if a subsystem will want to replicate large objects. We probably should introduce some kind of soft (or hard limit there)
- There is also a problem with deadlock (e.g. an op comes in starting new server /host=a/server=1). A newly started server should not send any updates to the registry until it's fully operational
- The registry should have the following tree-based structure: root -> server group -> capability -> custom objects
- Initially, the registry would store only topology information (IPs, ports etc)
- Marshalling and serialization is not an issue here - modules are responsible for marshalling (and unmarshalling) data
- The implementation should be based management-client-content. The module already solves most of the problems
- Each server adds its own host details (IP, port) to the domain managed service registry
- Newly added information is replicated to other servers
- JGroups subsystem extracts topology information and injects members list into TCPPING
- pro: a newly created registry could be reused by many subsystems
- con: a lot of clustering stuff would have to be implemented (and most of them has been already solved in Infinispan project)
Design proposal #2: Hardcoded topology information and Infinispan based registry
- The design idea #1 is simply a replicated cache. We could use Infinispan for that.
- However there is a still chicken or the egg problem because we need topology information to start ISPN cluster in TCP mode.
- We could hardcode updating topology information code into the Domain Model (just like we do with other bits)
- The registry would be populated only by a Domain Controller
- Infinispan subsystem would consume this information and form a cluster
- We could create a domain registry using infinispan subsystem
- We could implement it similarly to web session cache
- pro: much simpler implementation and much less clustering implementation in registry area
- con: A dependency on Infinispan in core?