Regression consequences and improved process

[bill.burke]

We had some discussion about this on jboss-dev and the-core. Head/trunk has not had a full passing testsuite in years. This has resulted many times in people committing stuff that regresses other projects without knowing it. Much of this is because of the lack of stable (or any) SPIs for projects loaded into JBoss AS through thirdparty (i.e. JBoss WS), but IMO, we cannot wait for these things to stabilize.

I suggest the following actions:

1. Freeze trunk
2. Generate JIRA tasks for each failing test in EJB3 and testsuite module (and any other module that has a testsuite in AS trunk.
3. Disable all failing tests
4. Unfreeze trunk.

5. Nightly build runs
6. If it is not successful, then freeze trunk until regressions are fixed
7. If successful tag and increment build number

The TCK should be a part of the regression process. If it regresses, we should freeze as well. I dont' know how feasible this is given TCK takes what, 10 hours to run?

I've seen this process work well at other companies (one of them a middleware company!). One thing that makes this process work is for developers to work in their own branch off of trunk. This allows them to commit and share work with others, without breaking or regressing trunk. Since we are on SVN now, this should be much much easier to do.

Thanks.

[starksm64]

This is related to:
http://www.jboss.com/index.html?module=bb&op=viewtopic&t=113287

Ok, but the current testsuite runs just as long as the tck. I still think we need to focus the regression testsuite on integration and stable api tests and move the implementation tests out to the modules. For example, several of the broken tests rely on deployer implementation details and should be moved into the deployer projects.

[bill.burke]

Whoops, went to the wrong forum. Sorry Scott. i'll repost there.