GSoC '12: Automated Visual Verification (RushEye)

[lfryc]

This is the thread for discussing Automated Visual Verification concept for GSoC 2012 project.

The concept builts on implementation started as RushEye project as Arquillian subproject.

References

• Visual Testing of Browser Screen Captures (thesis, PDF)
• The Automated Visual Testing using RushEye (PDF)

The Image Comparison Tool (Current State)

The current RushEye Core contains

• image comparison alghoritm
• with suite defined in XML
• with statistics/result output to XML

The goal of the GSoC project Automated Visual Verification are implementing missing pieces:

• integration with test suites
• RushEye Manager (review tool)

Integration with test suites (Proposal)

As the thesis [1] shown, turning Selenium test suite to image generation tool is really simple,

still, integration with current tools should be developed to allow developers reuse the tool as easily as possible.

The proposed integration point is Arquillian Graphene (built on top of Selenium 2 / WebDriver).

Graphene offers interceptors which allows you to hook into tool's interactions with the tested page,

and after each interaction, you can capture screenshot (TakesScreenshot interface in WebDriver API).

The only what would user need to do is turning on RushEye integration:

<arquillian>
    <extension name="rusheye">
          <property name="enabled">true</property>
    </extension>
</arquillian>

Additionally, annotation-based approach can be used to select all tests/cases which should be used for image generation:

@RunWith(Arquillian.class)
@RushEye
public TestCase {}

or

@Test
@RushEye
public void testMethod() {}

RushEye Manager (Proposal)

RushEye manager is fundamental tool to allow the project be successful.

The essential goal of the project is avoiding the manual testing by automating visual comparison.

Still, lot of changes in the tested applications are expected each day and without any tool, the review process of generated images would be time-consuming.

The value-add of the RushEye project is integration of the review tool (Manager) into the process:

Description of use cases:

1. continuous build generates results from predefined configuration and images
2. continuous integration system shows number of failures
3. Manager downloads configuration and results
4. Manager downloads generated images for failed tests
5. Manager allows following actions to user:
   • review images (including difference image)
   • accept changed image (rejecting the old one)
   • reject image (real failure)
   • upload changed suite to the continuous integration
   • generate report of review (the list of real failures)

[udesh]

By referring to your discussion I came up with an initial stage of the proposal for the project. Please let me know your views regarding the proposal.

http://www.google-melange.com/gsoc/proposal/review/google/gsoc2012/udesh/39003

I may have time till 6th April to edit this.