The first thing to do is ensure your <redistribution-delay> is small enough (e.g. 0) to ensure quick message redistribution. Then you should make sure any consumers which are routinely slower than the other uses a smaller consumer-window-size than the others so they buffer fewer messages.
already got those settings, they don't help balance the load after messages was loaded to queues.
AFAIK redistribution-delay can only help when no consumers are connected, however in my case all consumers are present all the time, they're just slow.
I can think about simple cron script that can redistribute messages across all nodes to keep queues about same size until they're empty
but this seems counter intuitive to do it manually. I would expect such behaviour to be already included in Hornet, but if its not I can go for a cron job without any further hesitation.
Assuming your slow consumers are arbitrary then they should be randomly distributed across the cluster (assuming you're using the default load-balancing-policy). If they are all clumped up on a single server then this problem could definitely occur. At this point I see 2 options:
- Make sure your slow consumers are distributed across the cluster and not clumped up on one node. Like I said, I would expect that to happen anyway, but the way you are connecting may prevent that.
- If your consumers are really slow then you could close the consumer while the message is being processed. That would allow redistribution to happen.