RHQ.next Inventory Design Proposals

[lkrejci]

Hi all,

I just wanted to let everyone know about the design proposals for the inventory in RHQ.next.

The designs build on these two articles:

Proposal - Poly-agent RHQ - RHQ - Project Documentation Editor

Thoughts on Inventory for RHQ.next

The design docs are here:

https://docs.google.com/document/d/1GNsKZvCebauYFg1ML3FUXZhXF3BPOip9fJiE55x4Hjo/edit

http://docs.rhqnextinventory.apiary.io/

The second is the REST API prototype.

Both documents are very much WIP but you should be able to comment on both. I look forward to your insights and comments.

[brmeyer]

One alternative to consider is something we've discussed with Thomas and Heiko: http://www.projectoverlord.io/s-ramp

Frankly, the RHQ Inventory requirements and articles I've seen, so far, read like S-RAMP overview guides.  Many of the posed questions are the very reason the spec and our implementation exist.  Some of the thoughts are described here: https://docs.google.com/document/d/1WiI0i0uq8wIOS8w13aYIsAt0zLYAZO77W0DKFP4O1UU/edit?usp=sharing

I had also put together the following, describing the structure and nature of our tree, as there were several questions and concerns about using a hierarchical graph.  S-RAMP's tree structure design

[lkrejci]

Thanks, Brett for the links.

I've known S-RAMP for a year or so but frankly, at the start, it didn't occur to me as a possible candidate for inventory storage. But you're right, there's a large overlap of concepts that we need supported. Certainly at least at the storage side of things, S-RAMP will be taken into consideration when we get to the point of deciding on technology (right now, we're just coming up with the usecases and workflows really).

One question I have about S-RAMP though is how much does it scale - can it handle cca 500k "documents" with each having at least 3 relationships (a rough approximation of the larger inventories we anticipate that I just pulled out of thin air)?

[brmeyer]

One question I have about S-RAMP though is how much does it scale - can it handle cca 500k "documents" with each having at least 3 relationships (a rough approximation of the larger inventories we anticipate that I just pulled out of thin air)?

Absolutely!  Realistically, the S-RAMP project itself is extremely lightweight.  Most performance considerations lean on our use of ModeShape/Infinispan.  The above link describes our tree structure, which was designed with ModeShape's help and with performance/scalability as the primary concern.  The specific approach is described in detail here: http://modeshape.wordpress.com/2014/08/14/improving-performance-with-large-numbers-of-child-nodes/

[lkrejci]

The design doc on google docs has just been updated with additional details. We're getting close to have the design "compiled" (but NOT finished at all - this is a starting point to discussion).

[brmeyer]

Hey guys, I've been discussing the following with Lukas and Thomas -- capturing it here.

As mentioned, the feature overlap between Artificer (formerly "S-RAMP") and Inventory seems to be fairly significant.  Missing metadata, queries, etc. can be quickly addressed, for the most part.

There's been a lot of discussions about graph databases.  Although I'm confident JCR/ModeShape, and Artificer's tree structure, can handle the proposed loads (I'll soon be working on performance tests), here's something to consider.  Artificer is purposefully modularized, including abstraction for adding additional storage solutions.  Although JCR/ModeShape is our current one, Artificer is setup such that we could add support for a graph backend.  It would obviously be a decent amount of work, but in the end, it may be more worthwhile than attempting to write all your features from-scratch.  So, if the backing storage is the only hold up, adding support for graph databases within Artificer may be important to explore, especially considering the rest of the overlap and OOTB capabilities that already exist.

Just my $.02.

[jayshaughnessy]

I know I've seen mention of TitanDB as a graph DB fronting Cassandra.  I have not seen any evidence that we've actually tried using it.  But the idea that it already leverages a Cassandra cluster is attractive given that hawkular-metrics is already using Cassandra.  I guess my [naive] question is whether the proposed work for Artificer to to "support a graph backend" means that it would be a layer on top of something like TitanDB or would replace, for example, TitanDB?  And if that latter, would it be able to leverage the Cassandra repository or does it add another repository solution to the mix?

[lkrejci]

Artificer would I assume be a layer on top of Titan which would be a layer on top of Cassandra. We have some rudimentary perf tests of Titan that Jirka has written up: https://github.com/Jiri-Kremser/rhq-titan-poc but we need a) more work on those tests themselves and b) compare them against something. It would be interesting to see how Titan would compare against Artificer with its current Modeshape backend.

[lkrejci]

There's a question about Artificer that I've wanted to ask. Does it support transactions?

[brmeyer]

> I guess my [naive] question is whether the proposed work for Artificer to to "support a graph backend" means that it would be a layer on top of something like TitanDB

> Artificer would I assume be a layer on top of Titan which would be a layer on top of Cassandra.

Correct, sorry for not being clearer.  Artificer would sit on top of TitanDB and/or Cassandra and simply use it as its backing storage.

> There's a question about Artificer that I've wanted to ask. Does it support transactions?

It's not something we handle on our own -- we largely punt to the backing storage solution for those types of things.  Obviously, it's definitely possible with ModeShape and Infinispan (just a matter of configuring it).  Although, supporting JTA within our Java client is an interesting thought...