1. Continuous integration

Continuous integration is an approach to quality assurance in which a software product is checked out of its repository, built and possibly tested, on a reasonably continuous basis. The frequency of this cycle can be based on the number of changes to the source repository for the product, some desired minimal frequency (e.g 3 hourly, nightly) or both. The aim is to identify product build errors sooner rather than later, and thus ease the product release process.

Cruisecontrol is used in JBoss QA to implement continuous integration in an automated fashion. Cruisecontrol is currently set up to automatically perform product builds, testsuite builds, and testsuite runs. The list of products which are subject to continuous integration are available at the cruisecontrol status page, located at http://cruisecontrol.jboss.com.

In addition to the scheduling of product builds, testsuite builds and testsuite runs, a continuous integration scheme requires the manual identification and repair of failed product builds, testsuite builds and testsuite runs. It is the shared responsability of the product developers and the JBoss QA group to monitor the builds and runs initiated by cruisecontrol and deal with any build/run failures.

The remainder of this wiki outlines the process for manual identification and repair of product builds.

2. Monitoring cruisecontrol builds

One way to determine the current status of product builds is to check the cruisecontrol status page. This page lists the current status of the product build, testuite build, or testsuite run for each product subject to continuous integration testing. It also shows the recent history of such builds.

Alternatively, you can subscribe to the mailing list qa@jboss.com and be notified of build results automatically. Cruisecontrol sends out e-mail notifications to the mailing list qa@jboss.com for every build/run it carries out. The email content provides a link to a web page presenting the results of the build or run within cruisecontrol. The e-mail subject describes the type of the build as well as its overall result. Possible e-mail subject lines include:

• jbpm.3-build failed

• jboss-4.0.-testsuite-1.6-build failed

• jboss-portal-2.4-testsuite-build completed with errors

The first subject line describes a case in which the lastest version of the product jBPM.3 has been checked out of its source repository and the build of the product has failed. The second subject line describes a case in which the latest version of the jboss-4.0 testsuite (which tests the jboss-4.0 application server product) has been checked out of its source repository and the build of the testsuite has failed. In the last case, the testsuite of the jboss-portal product, version 2.4, has been checked out, successfully built, and the execution of the testsuite completed with some testcase errors.

Not all failures are equal. When considering failures for a given product:

• a failed product build is of a higher priority than a failed testsuite build

• a failed testuite build is of higher priority than a testuite which builds successfully but for which certain test cases fail 

These relative priorities should be taken into account when dealing with multiple build/run failures.

You should aim to monitor the cruisecontrol builds in such a way that you are aware of problem builds on a daily basis.

3. Dealing with cruisecontrol build failures

Failed product builds, testsuite builds and testsuite runs need to addressed, generally by:

performing root cause analysis to identify the source of the failure and

(ii) making changes to the source code of the product and or testsuite in order to clear the problem.

Responsability for carrying out root cause analysis and repair is not always easy to determine, and some guidelines are given in the following section. However, in general, once a build/run failure has been identified (either by checking the cruisecontrol status page, or the qa@jboss.com mailing list), you should allow the person or group responsible for performing root cause analysis and clearing the failure at least 24 hours to correct the error. After 24 hours, if there has still been no improvement in the situation, send an e-mail to jboss-development mailing list, copying the person most likely to have influenced the error through recent commits to the source code repository. The email should contain a description of the error. This is simply to bring the error to the attention of those involved, in case they missed it during the 24 hour period.

4. Determining who is responsible for doing root cause analysis and clearing the error

Again, this is not an exact science; however, the following guidelines should help you decide:

failed product build

Notify the developer by e-mail and copy jboss-development mailing list. You can get a idea of which developer to contact based on who committed the last changes to the source repository for the product which have influenced the error (e.g. if a file is mentioned in the error log, and a change to that was recently committed, it's possible that that developer's change was involved). JBoss QA are rarely involved in root cause analysis and clearing of these types of errors.

Possible reasons for product build failures include:

• incorrect or missing product artifacts (source code and other files, dependent jars)

• compiler errors

• errors in the product build file (build.xml) itself

failed testsuite build

Try to narrow down the probable cause of the error, as responsibility for root cause analysis and clearing of testsuite errors may fall on either the JBoss QA group or developers, depending on the error. If resposibility for root case analysis and repair lie with a developer, then advise through e-mail as before. If root case analysis and repair are the response of the QA group, then set up a JIRA task for the problem.    

Possible reasons for testsuite build failures:

• incorrect or missing testsuite artifacts (test case source code and other files, dependent jars)

• compiler errors in test case source code

• errors in the testsuite build file (build.xml) itself

successful testsuite build but failed testsuite run

Try to narrow down the probable cause of the error, and operate as in the case of a testsuite build failure. In particular, when many test cases have failed, look for a common cause for the problem. Also remember that test cases can fail intermittently, meaning that the may fail during cruisecntrol execution, but pass when run in isolation.

Possible reasons for testsuite run failures:

• incorrect or missing configuration or test case artifacts (configuration files, dependent jars)

• failed startups of test case servers

• incomplete deployments of applications within servers

• errors in the testsuite build file (build.xml) itself or the jbossbuild framework

• other possibilities

Referenced by: