We want to use HornetQ to implement a basic level of fairness between message flows. I'd like to hear if anybody on the forum has a good idea on implementing this.
- The application, a multi-tenant system, processes flows of messages, one per tenant.
- Cluster A produces messages into HornetQ.
- Cluster B consumes messages from HornetQ.
- The number of message streams from A to B is dynamic. The number of active tenants varies. At any time, there are between 100-10,000 active message flows.
- Message volume varies by 1-2 orders of magnitude between tenants.
- Message volume within a tenant spikes occasionally.
Scenario 1: Load spike in one message stream
Ocassionally, one of our tenants misbehaves or sends us a large amount of data. The node(s) in cluster B consuming the tenants queue will not be able to keep up. The desired behavior in this case is for the particular tenant to receive back pressure, and their messages to build up in HornetQ. Once the memory allocated to the tenant is full, HornetQ should BLOCK. Other tenants should be unaffected by this.
Scenario 2: Cluster B stopped
When Cluster B is unavailable to consume any messages, HornetQ should store messages until Cluster B comes back online. It should do so fairly, allocating comparable amounts of space to all tenants.
How would one best setup a set of HornetQ addresses, diverts, queues, groups, etc (using Core API), such that:
- An overload condition (more incoming vs. consumed messages) in one stream does not affect the other streams.
- Memory in HornetQ is distributed fairly between the message streams, without wasting any memory in a global overload/back pressure situation.
My (not-so-great) ideas:
- Use one address/queue per stream. As I understand, this requires me to manually manage memory across the addresses, which is tricky. (example, I have 10GB memory, with 10 streams, I'd like to assign 1GB to each stream, but as we grow to 100 streams, I'd need to shrink all the addresses to 100MB each to leave room).
- Use rate limited flow control. Monitor address utilization and adjust rate limits on an ongoing basis. This seems very indirect and clumsy.
Is there any obvious approach I'm missing? Any other good ideas on implementing fairness?