6 Replies Latest reply on Sep 5, 2013 4:18 PM by john.sanda

    "Failed to load graph data" after specifying date range

    josgood
    josgood Sep 3, 2013 7:04 PM

      RHQ Community,

       

      I've been routinely running into issues with RHQ 4.8 while viewing resource schedule graphs (Resource > Monitoring > Graphs). While these graphs typically render without fuss, I will often receive "Failed to load graph data" errors after specifying a date range. When this happens, the charts will fail to render. Additionally, when I attempt to retrieve the same bucketized metric data (with the same start/end dates) via the rest API, it fails and returns the following:

       

      {"message":"java.lang.IllegalArgumentException: highValue (87.0) is not greater than or equal to value (111.33333333333331)."}

       

      The numbers above change depending on the schedule I am accessing, but in every case value was greater than highValue. In general, these errors seem to be less likely of occurring among schedules with flat (mostly consistent) data. Additionally, these errors are more prevalent when specifying ranges containing dates greater than 7 days prior to the current date, so it may have to do with the way RHQ condenses metric data after 7 days.

       

      Anyway - I was wondering if this was an isolated issue particular to my installation, or if other people had encountered this as well.

       

      Gratzi,

      Josh

      • 308 Views
      • Tags:
        • 1. Re: "Failed to load graph data" after specifying date range
          john.sanda
          john.sanda Sep 3, 2013 11:09 PM (in response to josgood)

          Hi Josh,

           

          As you may be aware, the metrics storage backend re-implemented in RHQ 4.8. This could very well be a bug. I think the easiest way to determine what is happening is to look at the actual data points. Since RHQ does not provide APIs to directly access the data points (except for raw data), the easiest thing to do is directly query Cassandra. If you want to do that, here are the steps you need to perform:

           

          1. Log into RHQ
          2. Navigate to Administration --> Storage Nodes
          3. Select storage node row so that it is highlighted
          4. In the footer, click on the Operation button and then in the pop-up click on Enable Debug Mode
          5. Create a text file named raw_metrics.cql that contains the text, SELECT * FROM raw_metrics WHERE schedule_id = <schedule_id>;
          6. Create a text file named one_hour_metrics.cql that contains the text, SELECT * FROM one_hour_metrics WHERE schedule_id = <schedule_id>;
          7. In a terminal cd into <rhq-server-dir>rhq-storage/bin
          8. Execute ./cqlsh -u rhqadmin -p rhqadmin -k rhq -f raw_metrics.cql > raw_metrics.txt
          9. Execute ./cqlsh -u rhqadmin -p rhqadmin -k rhq -f one_hour_metrics.cql > one_hour_metrics.txt


          The above obviously assumes you are querying for one hour data.  If querying for older data, it is just a matter of changing the table name. If you have a lot of data, we can apply a date filter to limit the results. If you can share the data points, I would be more than happy to take a closer look and try to figure out what is happening. Lastly, what date ranges are you using in your queries?

            • Actions
          • 2. Re: "Failed to load graph data" after specifying date range
            mazz
            mazz Sep 4, 2013 2:29 PM (in response to john.sanda)

            I see the same problem (but only in my platform resource at the moment). I created a BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1004498

              • Actions
            • 3. Re: "Failed to load graph data" after specifying date range
              mazz
              mazz Sep 4, 2013 2:43 PM (in response to mazz)

              I can't get that data that John asked for - I don't know what the schedule ID is for the metric that is bad.

                • Actions
              • 4. Re: "Failed to load graph data" after specifying date range
                mazz
                mazz Sep 4, 2013 3:14 PM (in response to mazz)

                I attached a debugger, found my schedule ID and ran the CQL. It turns out, I have data from days ago still. I had -Ddbsetup my  RDBMS this mroning, but did not clean the storage node data. So my testing was bogus - I'm sure my data was not in sync between RDBMS and storage node. I closed the BZ as "not a bug".

                 

                So, make sure if you re-install RHQ and blow away your RDBMS data, you also blow away the storage node data as well.

                  • Actions
                • 5. Re: "Failed to load graph data" after specifying date range
                  josgood
                  josgood Sep 5, 2013 3:35 PM (in response to john.sanda)

                  John,

                   

                  I am unable to click the Operation button after selecting the storage node, and when I attempt to expand the node for more information, it throws an error. This is starting to sound like an issue with my storage installation, though data collection has seemed fairly functional thus far.

                   

                  I'll plan on posting again after a fresh install. Thanks for all your help,

                   

                  Josh

                    • Actions
                  • 6. Re: "Failed to load graph data" after specifying date range
                    john.sanda
                    john.sanda Sep 5, 2013 4:18 PM (in response to josgood)

                    Alternatively, you can do this,

                     

                    • Open <rhq-server-dir>/rhq-storage/conf/cassandra.yaml
                    • Find the start_rpc property and set it to true
                    • Restart the storage node

                     

                    After that change, you can use cqlsh and run the queries I posted earlier.

                      • Actions