6 Replies Latest reply on Oct 14, 2014 1:50 PM by john.sanda

    Schema design

    john.sanda

      I have been continuing to think more on design and starting writing up some notes. Take a look at my design doc[1], the schema section in particular I provided an example of how we might declare tenant-level pre-computed aggregation and retention settings


      [1] https://docs.google.com/document/d/1Il9wHSKQD4QwVmgmZhrA7ghRejAt2Qka98_qoQFDfUU/edit?usp=sharing

        • 1. Re: Schema design
          mithomps

          You have probably already seen this, but here is another schema from: pyr/cyanite · GitHub - project that is a C* based backend replacement for graphite.

          It has tenants and rollup...

          CREATE KEYSPACE metric WITH replication = { 'class': 'SimpleStrategy', 'replication_factor': '1' };

          USE metric;

          CREATE TABLE metric ( period int,

          rollup int,

          tenant text,

          path text,

          time bigint,

          data list<double>,

          PRIMARY KEY ((tenant, period, rollup, path), time) )

          WITH bloom_filter_fp_chance=0.010000 AND caching='KEYS_ONLY' AND comment='' AND dclocal_read_repair_chance=0.000000 AND gc_grace_seconds=864000 AND index_interval=128 AND read_repair_chance=0.100000 AND replicate_on_write='true' AND populate_io_cache_on_flush='false' AND default_time_to_live=0 AND speculative_retry='NONE' AND memtable_flush_period_in_ms=0 AND compaction={'class': 'SizeTieredCompactionStrategy'} AND compression={'sstable_compression': 'LZ4Compressor'};


          • 2. Re: Schema design
            john.sanda

            I have seen it. In fact, you may recall that the RHQ core dev team discussed it some a while back when we first started discussing RHQ Metrics. Gotta love that it is in Clojure The path column is the metric identifier. One thing that I find interesting is that in index of paths is maintained. From what I can see so far, that index is used in querying/search for metrics from the HTTP interface. By default the index is stored in memory, but it also supports using ElasticSearch. I am not sure about all of the search capabilities that are available to clients, but it definitely offers a nice, flexible way to find metrics.

            • 3. Re: Schema design
              john.sanda

              My Clojure skills are admittedly rusty, but it looks like cyanite only supports a few aggregation functions - max, min, mean, and sum. It does support pre-computed aggregates in addition to downsampling.

              • 4. Re: Schema design
                mithomps

                I was only thinking that the schema might provide multi-tenant and rollup  ideas that may be useful. That's all.

                • 5. Re: Schema design
                  mithomps

                  What about units of measure (UoM) for the values. Different metrics will have various UoM.  I didn't see anything in the schema to capture that?

                  • 6. Re: Schema design
                    john.sanda

                    Mike Thompson wrote:

                     

                    What about units of measure (UoM) for the values. Different metrics will have various UoM.  I didn't see anything in the schema to capture that?

                    My design doc is a work in progress. Take a look at the notes and examples for the numeric_data table. It includes an attributes column now.