jboss-head build testsuite action plan

[starksm64]

After a number of email on the need to lock down the jboss-head testsuite and introduce build breakage and testsuite failure consequences, there was no action plan. I'd like to get that finalized in this thread. No whining please, just posts that move toward the goal.

[starksm64]

The first change should be rolling back changes that break the build of the server or testsuite. What is required to do this with the current CC runs?

[ryan.campbell]

For the moment, let's take cruisecontrol out of the equation as I would prefer we do this as a generic hudson plugin so that we can apply it generically for any project that chooses to go this route.

That said, here is how it would roughly work (erring on the side of completeness):

1. CI system polls SVN every 15 minutes looking for changes
2. If changes are found, CI system polls SVN for additional 7 minutes to make sure the commits are complete.
3. CI system performs a clean build
4. If the build fails, rollback the commit by doing a reverse merge on the workspace using the last known good revision ID.
5. Commit this changeset to the repository.
6. Send out an email flaming the committers to the build.
7. GOTO 1

This is the algorithm as I understand it. Please confirm that I have it right before I tell you why I think its a bad idea.

[starksm64]

If we are rolling back changes that break the build, then yes, that seems like the algorithm. I would like the build to be triggered by an N minute quiescent period with a one minute resolution rather than 15 though.

[rrajesh]

This algorithm would work fine for build breakages. However one issue when we rollback the commits to a previous working version, we might lose the commits that were done after the build started; which would have ideally been included in the next build.
In that case, can we rollback to the previous working version and then apply the commits that were done after this build started?

For testsuite regressions,
* We need to get in place a notion of known failures, to actually track the regressions.
And, is rolling back the commits, a right option for testsuite regressions as well?

[starksm64]

"rrajesh" wrote:
This algorithm would work fine for build breakages. However one issue when we rollback the commits to a previous working version, we might lose the commits that were done after the build started; which would have ideally been included in the next build.
In that case, can we rollback to the previous working version and then apply the commits that were done after this build started?

In general the commits made after the build are invalid as they could rely on changes that need to be rolled back. The main issue is how are the rolled back changes identified and the responsible parties notified so that the breakage is fixed soon.

"rrajesh" wrote:

For testsuite regressions,
* We need to get in place a notion of known failures, to actually track the regressions.
And, is rolling back the commits, a right option for testsuite regressions as well?

Not all tests are equal, which is why I only wanted to see this validated against integration tests where a failure means some contract usage is broken.

[bill.burke]

Maybe a freeze is better than a rollback? Not even necessarily a real SVN freeze. What we do when we see a regression is send an email saying we're freezing, no commits. We look at Hudson, or wherever on who committed what and get them to fix their stuff.

Here are my thoughts that I posted on a different forum by mistake:

1. Freeze trunk
2. Generate JIRA tasks for each failing test in EJB3 and testsuite module (and any other module that has a testsuite in AS trunk.
3. Disable all failing tests
4. Unfreeze trunk.

5. Nightly build runs
6. If it is not successful, then freeze trunk until regressions are fixed
7. If successful tag and increment build number

The TCK should be a part of the regression process. If it regresses, we should freeze as well. I dont' know how feasible this is given TCK takes what, 10 hours to run?

I've seen this process work well at other companies (one of them a middleware company!). One thing that makes this process work is for developers to work in their own branch off of trunk. This allows them to commit and share work with others, without breaking or regressing trunk. Since we are on SVN now, this should be much much easier to do.

[starksm64]

Just freezing sounds like a better way to get started with this.

[bill.burke]

do we want to disable broken tests and create jira tasks for them?

[wolfc]

I don't think there is a need with Hudson. You know when, what has regressed.
It would also be a bit of a burden to create a JIRA for each test if I have >50% regression.