6 Replies Latest reply on Sep 5, 2013 4:18 PM by John Sanda

    "Failed to load graph data" after specifying date range

    Josh Osgood Newbie

      RHQ Community,

       

      I've been routinely running into issues with RHQ 4.8 while viewing resource schedule graphs (Resource > Monitoring > Graphs). While these graphs typically render without fuss, I will often receive "Failed to load graph data" errors after specifying a date range. When this happens, the charts will fail to render. Additionally, when I attempt to retrieve the same bucketized metric data (with the same start/end dates) via the rest API, it fails and returns the following:

       

      {"message":"java.lang.IllegalArgumentException: highValue (87.0) is not greater than or equal to value (111.33333333333331)."}

       

      The numbers above change depending on the schedule I am accessing, but in every case value was greater than highValue. In general, these errors seem to be less likely of occurring among schedules with flat (mostly consistent) data. Additionally, these errors are more prevalent when specifying ranges containing dates greater than 7 days prior to the current date, so it may have to do with the way RHQ condenses metric data after 7 days.

       

      Anyway - I was wondering if this was an isolated issue particular to my installation, or if other people had encountered this as well.

       

      Gratzi,

      Josh

        • 1. Re: "Failed to load graph data" after specifying date range
          John Sanda Apprentice

          Hi Josh,

           

          As you may be aware, the metrics storage backend re-implemented in RHQ 4.8. This could very well be a bug. I think the easiest way to determine what is happening is to look at the actual data points. Since RHQ does not provide APIs to directly access the data points (except for raw data), the easiest thing to do is directly query Cassandra. If you want to do that, here are the steps you need to perform:

           

          1. Log into RHQ
          2. Navigate to Administration --> Storage Nodes
          3. Select storage node row so that it is highlighted
          4. In the footer, click on the Operation button and then in the pop-up click on Enable Debug Mode
          5. Create a text file named raw_metrics.cql that contains the text, SELECT * FROM raw_metrics WHERE schedule_id = <schedule_id>;
          6. Create a text file named one_hour_metrics.cql that contains the text, SELECT * FROM one_hour_metrics WHERE schedule_id = <schedule_id>;
          7. In a terminal cd into <rhq-server-dir>rhq-storage/bin
          8. Execute ./cqlsh -u rhqadmin -p rhqadmin -k rhq -f raw_metrics.cql > raw_metrics.txt
          9. Execute ./cqlsh -u rhqadmin -p rhqadmin -k rhq -f one_hour_metrics.cql > one_hour_metrics.txt


          The above obviously assumes you are querying for one hour data.  If querying for older data, it is just a matter of changing the table name. If you have a lot of data, we can apply a date filter to limit the results. If you can share the data points, I would be more than happy to take a closer look and try to figure out what is happening. Lastly, what date ranges are you using in your queries?

          • 2. Re: "Failed to load graph data" after specifying date range
            mazz Master

            I see the same problem (but only in my platform resource at the moment). I created a BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1004498

            • 3. Re: "Failed to load graph data" after specifying date range
              mazz Master

              I can't get that data that John asked for - I don't know what the schedule ID is for the metric that is bad.

              • 4. Re: "Failed to load graph data" after specifying date range
                mazz Master

                I attached a debugger, found my schedule ID and ran the CQL. It turns out, I have data from days ago still. I had -Ddbsetup my  RDBMS this mroning, but did not clean the storage node data. So my testing was bogus - I'm sure my data was not in sync between RDBMS and storage node. I closed the BZ as "not a bug".

                 

                So, make sure if you re-install RHQ and blow away your RDBMS data, you also blow away the storage node data as well.

                • 5. Re: "Failed to load graph data" after specifying date range
                  Josh Osgood Newbie

                  John,

                   

                  I am unable to click the Operation button after selecting the storage node, and when I attempt to expand the node for more information, it throws an error. This is starting to sound like an issue with my storage installation, though data collection has seemed fairly functional thus far.

                   

                  I'll plan on posting again after a fresh install. Thanks for all your help,

                   

                  Josh

                  • 6. Re: "Failed to load graph data" after specifying date range
                    John Sanda Apprentice

                    Alternatively, you can do this,

                     

                    • Open <rhq-server-dir>/rhq-storage/conf/cassandra.yaml
                    • Find the start_rpc property and set it to true
                    • Restart the storage node

                     

                    After that change, you can use cqlsh and run the queries I posted earlier.