prevent tests get overwritten

[akostadinov]

We've had recently a problem where some tests was run twice or more from different targets without being renamed with the jboss formatter. This leaded to the possibility of undetected failures if a test fails and on the next run passes. See this link for a reference:
http://jboss.com/index.html?module=bb&op=viewtopic&t=101760

It was decided to rename all tests but tests-standard-unit ones.

I'm thinking about how to prevent this in the future - when new test targets are added and the developers forget to use the custom formatter.

I can think of two options:
1. implement in the JBoss AS test suite itself
2. implement into our continuous integration system

First one is hard. I see it like counting test cases files specified by the patterns used in the build file and comparing to the number of generated XML report files or number reported in the final report. I think there will be plenty of problems to count the files. First we'll need a resourcecont ant task or a similar one and also some of the patterns might be invisible to where the task is called. ResourceCount target is available in ant 1.7 and we are using 1.6.5.

That's why I think it will be easier and enough for our needs to have the check implemented in our continuous builds. It should be one line shell script. If overwriting occurs developers will be notified.

Any other ideas/thoughts?

[ryan.campbell]

I'm fine with doing it in continuous builds for now. Ideally, this guarantee would be built into a custom test runner. IE, a notion of a test run identifier which would prevent the overwriting of results.

[akostadinov]

that will be incorporated near days in hudson JBoss AS 5.0 runs