Is it possible to flag a metric for non-compression?

[mark4]

I'm trying to find a way to store a metric for a longer period of time than the normal compression threshold, to allow for the possibility of detailed comparison over a longer stretch of time than a week. I'd like to be able to compare production-system metrics in a same-time-last-month sort of scenario, or to be able to compare baseline performance-system metrics that might be generated with infrequent performance runs over the course of a couple months.

Is there any way to keep metrics around without compression and pay the hit for db size?

Or when compression runs, is there an option to move that data into archive tables instead of deleting them?

Failing that, is there a way to snapshot (for lack of a better term) a set of metrics for baseline comparison at a later time?

I'd prefer to point one browser at the 'old' timespan, and a second browser at the 'new' timespan. I suppose a fallback might be to generate a series of per-metric graphs and saving the images off in an offline-report sort of idea.

Thanks.

[mazz]

No, you cannot configure whether a metric's data is compressed later in life.

You might consider archiving the measurement data by copying them to your own custom tables and using your own reporting queries to extract the reports you want. That sounds like it would be possible.

[mark4]

I like the idea of having another set of tables that are geared to longtime storage - populating them with on-delete triggers into another set of tables, or to a parallel schema. The main jopr schema stays compressed, but the archive schema stays un-compressed, or compressed on a longer cycle.

How about running a jopr instance in read-only mode on top of that archive data? No communication / monitoring of agents, no data purging. Just data viewing. Is that possible?

[mazz]

you can certainly run an Server pointing to a database with a copy of the tables/data without agents, but that server will still perform compression and data purging of old data (the server has jobs that periodically purge old alerts, events, etc...). I doubt it will let you do what you want.

You'll probably want your own periodic job running that extracts data from the live database and store it in your own data store somewhere. In the latest RHQ 3.0 beta release, we actually have a way for you to write your own server jobs (via the server-side plugins) - you could actually write some code, deploy it as a server plugin, and have it configured to run periodically to copy live data over to your own database tables. Server side plugins have full access to the EJB3 business objects and the live database (via hibernate and/or the EJB3 business objecs) so you can ask it for data and you then store the data elsewhere.

I actually wrote a sample server plugin that kinda does this (it has a job (schedulable in cron-like syntax) but it reports on the inventory, not the metric data - but its just a matter of calling the right EJB3 business methods to get the data you want.

For the source code, see http://git.fedorahosted.org/git/rhq/rhq.git?p=rhq/rhq.git;a=blob;f=etc/samples/simplereport-serverplugin/src/main/java/org/custom/SimpleReportsPluginComponent.java;hb=master

Here's the sample server plugin's descriptor (you can see it defines this scheduled job in there):

http://git.fedorahosted.org/git/rhq/rhq.git?p=rhq/rhq.git;a=blob;f=etc/samples/simplereport-serverplugin/src/main/resources/META-INF/rhq-serverplugin.xml;hb=master

Here's some docs on server side plugins:

http://rhq-project.org/display/RHQ/Server+Plugin+Development

Better docs on server side plugin development are on the way.

[pilhuhn]

Mark

would you be interested in providing that periodically copying over server side plugin that Mazz was pointing out?

Perhaps with some additional graphing based on JFreeChart or such to make the older data available

visually?

If so, ping us.

  Heiko

[mazz]

Yeah, this could lend itself to be the start of a  nice reporting tool. We can help point you in the right direction for APIs to call, how to write the server plugin, etc.

It would still put the onus on the deployer to provide access to another set of database tables (of course, the server side plugin could create the DDL and create the custom data model if it detects it doesn't exist - so it might even be less of a burden on the deployer if the server plugin dev puts in that extra work).

But, I think something like this is doable.

[mark4]

Thanks for the leads to the server-side reporting scheduler. That's definitely the backbone of periodic snapshot reports.

We were entertaining a "wouldn't it be nice if" discussion a week or so ago about the ability to give non-jopr users a little insight into "yesterday's operational metrics", by generating a simple html-with-graphs style report to post to the wiki every day, and rely on the wiki changed-pages-notification to publish "Here's a link to a page with yesterday's production status." notifications via email. A schedule-based reporting infrastructure could sure go a long way towards making this a reality.

Any idea if the rhq-3 server plugins will work in a jopr-2.3.1 environment? I guess I have some experimentation to do...

[mazz]

RHQ 3 server plugins will NOT work in Jopr 2.3.1 environments. Server side plugins are a new feature introduced in the new RHQ 3 line of development.

[mark4]

Of course, the next question is: When's the rhq-3 release expected?

I made this feeble attempt at a bugzilla query

https://bugzilla.redhat.com/report.cgi?x_axis_field=bug_status&y_axis_field=version&z_axis_field=&query_format=report-table&short_desc_type=allwordssubstr&short_desc=&classification=Other&product=RHQ+Project&longdesc_type=allwordssubstr&longdesc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&status_whiteboard_type=allwordssubstr&status_whiteboard=&keywords_type=allwords&keywords=&bug_status=NEW&bug_status=ASSIGNED&bug_status=MODIFIED&bug_status=ON_DEV&bug_status=ON_QA&bug_status=RELEASE_PENDING&emailtype1=substring&email1=&emailtype2=substring&email2=&bugidtype=include&bug_id=&votes=&chfieldfrom=&chfieldto=Now&chfieldvalue=&format=table&action=wrap&field0-0-0=noop&type0-0-0=noop&value0-0-0=

Is that at all representative of the remaining work? 13 or so bugs that are 'interesting' for the 3.0.0 release? Or is bugzilla only tracking real bugs, and none of the improvements you guys are making?

[pilhuhn]

I fear that bug count does not reflect reality, as we will do much more work until release than only fix those 13 bugs :-)

There is no official release data yet that I know of, but I think you may see something in the summer.

The recent community releases 3.0.0.B02 and B03 have the server side plugin container, so you can just start developing on this.

See http://rhq-project.org/display/RHQ/Release+Notes+3.0.0.B02 and http://rhq-project.org/display/RHQ/Release+Notes+3.0.0.B03

And of course there is the source :-)

And Mazz has written a server side plugin that actually does periodical tasks (see his posting above), so you can start by copy&pasting this.

  Heiko