I am here to kindly ask you guys for a feedback for my recent developments of visual testing tool.
I would like to achieve following:
Create a tool and enhance a process for visual testing of web application. In other words, protect your web application from CSS regressions, and other bugs, which can not be checked by unit/integration/functional or other tests, but need to be tested manually when testing with traditional tools. By automating this, rapidly decrease demand for human resources put into the testing.
There are already some solutions, which take screenshots and then compare them. I miss in that solutions some things, and therefore I started to develop another tool (with a stress to not a reinvent a wheel).
My approach would involve:
- reusing of functional tests for visual testing (because those tests click here and there over the page, and put the application into various states)
- make during such testing screenshots
- store the screenshots somewhere = pattern screenshots
- compare new screenshots = sample screenshots with the pattern screenshots
- see reasonable, useful, concise output of such comparison, and to be able to take appropriate action according to it
In this video I tried to summarize what I have done so far. Basically it runs Arquillian Graphene functional tests, make screenshots during that testing with Graphene screenshooter, compare screenshots with Arquillan Rusheye and make some outputs from such comparison.
That is for background. Now I would like to ask you guys for you opinion on how such a meaningful and useful report should look like. What would be most effective way for you to work with such tool.
Currently the process would start with the tasks presented in the video. Then the results will be uploaded to a web application.
I can see these functional/non functional requirements for such an application:
- application will provide REST endpoints, by which an user will be able to upload screenshots, and respective meta data
- logically the results will be separated into lets say jobs (CI jobs), and each will contain multiple runs of the test suite
- when reviewing results, one will be able to mark a new screenshot as a correct one, or otherwise
- authentication and authorization will be deployed as well
I also prepared GUI mockups of such an application, which you can find attached. You can find there:
- Front page - list of recent screenshots uploads
- Page with particular job - executions of visual testing
- A particular upload - always pattern screenshot, diff and the new one
Do you miss something ? What are your requirements on such web app to review the results of such visual testing ? Any ideas are more than welcomed.