RHQ - Support for multiple data centers?

[genman]

I've been exploring using RHQ in multiple sites. Although it seems like there is support for this, e.g. assigning affinity groups, etc., practically speaking there are limitations:

1. RHQ server can't really work efficiently with a database in a different datacenter. See Bug 1032192. Performance for inventory merge, availability changes, etc. is just too slow.

2. Every sever still needs to reach every agent. This isn't a major limitation, but requires network changes.

3. The storage cluster does not know how to setup network replication. In fact, StorageNodeOperationsHandlerBean.updateReplicationFactor may in fact overwrite any custom replication strategy with SimpleStrategy. https://bugzilla.redhat.com/show_bug.cgi?id=1032199

[genman]

As for (1), performance is not good but seems good enough once inventory is imported and there aren't a lot of inventory or availability changes. There basically needs to be optimizations to fetching data so that Hibernate does not do N+1 selects.

I've played around with batch sizes and whatnot, but I'm not sure how much it helps.

(2) I suppose can be solved once RHQ supports NAT connections. Basically everything would be fine if RHQ agents connected to a specific server and the servers sent messages through a message bus.

(3) I've gotten multiple DCs to work with Cassandra. It requires this patch: https://bugzilla.redhat.com/show_bug.cgi?id=1033943...Otherwise, inserting metrics can be too slow, causing agents to time out. If the ping time from one DC to another is 100ms, then inserting 1000 metrics at a time then takes 200 seconds, give or take. Locally, 200 metric inserts happen in under a second.