1 Reply Latest reply on Dec 2, 2013 3:18 PM by Elias Ross

    RHQ - Support for multiple data centers?

    Elias Ross Master

      I've been exploring using RHQ in multiple sites. Although it seems like there is support for this, e.g. assigning affinity groups, etc., practically speaking there are limitations:

       

      1. RHQ server can't really work efficiently with a database in a different datacenter. See Bug 1032192. Performance for inventory merge, availability changes, etc. is just too slow.

      2. Every sever still needs to reach every agent. This isn't a major limitation, but requires network changes.

      3. The storage cluster does not know how to setup network replication. In fact, StorageNodeOperationsHandlerBean.updateReplicationFactor may in fact overwrite any custom replication strategy with SimpleStrategy. https://bugzilla.redhat.com/show_bug.cgi?id=1032199

        • 1. Re: RHQ - Support for multiple data centers?
          Elias Ross Master

          As for (1), performance is not good but seems good enough once inventory is imported and there aren't a lot of inventory or availability changes. There basically needs to be optimizations to fetching data so that Hibernate does not do N+1 selects.

           

          I've played around with batch sizes and whatnot, but I'm not sure how much it helps.

           

          (2) I suppose can be solved once RHQ supports NAT connections. Basically everything would be fine if RHQ agents connected to a specific server and the servers sent messages through a message bus.

           

          (3) I've gotten multiple DCs to work with Cassandra. It requires this patch: https://bugzilla.redhat.com/show_bug.cgi?id=1033943...Otherwise, inserting metrics can be too slow, causing agents to time out. If the ping time from one DC to another is 100ms, then inserting 1000 metrics at a time then takes 200 seconds, give or take. Locally, 200 metric inserts happen in under a second.