Why current situation is not enough?

jsf-unit tests does not completely reflect real environments (application servers, real browsers).

There are also expensive to develop and maintain because of usage of HtmlUnit, which have limited debuggability.

Metamer tests are expensive to be written and even more expensive to be run. We have too many functional tests with inadequate impact on quality.

Therefore we don't have enough tests to control quality of framework in real environment which could support agile technique of development, supporting confidence in making radical changes.

Proposal for new test structure:

Component testing should be covered on several levels:

1. JavaScript widget unit tests
2. Java unit tests
3. renderer tests
4. functional (feature) tests
5. integration tests
6. regression tests
7. sample tests

JavaScript widget unit tests should be written in QUnit and run automatically using Selenium runner (WIP). Those tests verifies JavaScript interface of a widget (construction, input, output, destroyment, DOM events)

Java unit tests should verify any logic which is stand-alone and does not need any integration with servlet or JSF container or any other part. Should not require mocking at all.

Renderer tests are focused on verification of rendering logic as a whole. Component should be rendered to the page in its basic state and HTML output should be verified against pre-defined XML pattern (XMLUnit). This should be done for each of the rendering types for given component (e.g. Tooltip does have several rendering strategies for different layours - div, table and list).

Functional (feature) tests should test any particular feature of the component, in various configurations. These tests are written in Graphene. The number of these tests should be limited and as much as testing should be delegated to JavaScript unit tests.

Integration tests are written in Arquillian, any JSF interaction is then tested with Warp JSF extension. These tests should verify every interaction where components influences state of the FacesContext or triggers events or where life-cycle behavior is specific for given component. Integration tests also covers aspects where components integrates with each other.

Regression tests verifies important aspects which can’t be easily added to any other group and typically are created for reproducing and verifying found issues. They are typically written with Graphene and Warp.

Sample tests covers logic of sample applications, quickstarts, etc. They are written using Graphene, Warp and other Arquillian extensions. For testing component logic, Graphene Component Objects pattern is used.

Running:

Tests are written using Arquillian and Drone, thus it is possible to achieve high-level of portability.

This has some drawbacks:

• test execution in real containers is slow
• test execution in real browsers is slow
• test execution in real browsers is restricted to environments with display

In order to allow agile development (fast continuous integration, commit/pull request verification), but do not lose possibility to run tests in real environment, mocked containers and browsers might be used.

However it is also connected with some risks:

• portability and debugging on mocked browsers is paintful

Possibilities to improve execution time in continuous integration:

• alternative mocked browsers (GhostDriver)
• mocked servlet environment (jsf-test staging servlet feature)

Development & Maintanance:

Developers are responsible for writing basic set of tests for each component and each type of tests.

Quallity Engineers are responsible for running those tests on periodical basis (continuous integration), keeping portability (across browsers and containers). Quality Engineers are encouraged to extend basic tests, analyse them and incorporate improvements as required.