Thanks for posting this Trevor - great title
I agree this would be a great demo to have. I see this as:
1) A switchyard app to manage the control logic (with suitable simulator to generate measures, react to control signals and periodically introduce 'problems')
2) BAM infrastructure to:
a) capture events based on the interactions between the services and record the metrics
b) gadget server to present the metrics in a dashboard
c) CEP rules to monitor the temporal conditions and raise alerts when necessary
d) distribute alerts via JMX and/or gadget on the dashboard
If you could provide the switchyard app (1), then Jeff Yu and I can work on the aspects in (2). Within the next couple of weeks we will be doing a milestone release which will have more functionality - the only thing that might be missing is a generic/configurable gadget for 2b, which is on the plan for the following milestone - but when Jeff returns from leave he may be able to create a specific gadget as an interim solution.
This would actually be a good example to distribute with the next milestone - not sure how that would fit with your timescales/workload?