ISSUE #51 - March 18, 2018
Continuing coverage of web UI testing, we focus on visual testing / validation: what it is (just pixel comparison of screenshots, really, but with a lot of gotchas), how to do it, choice of tools and service providers. This became a long issue as I kept digging into the topic. There appear to be two levels of visual testing, with one set of tools geared more toward end-to-end tests that run in staging environment (Applitools Eyes, Percy, Backtrac), and another targeting developers at unit test level (React Storybook, Jest image snapshots, etc).
Distinction between functional testing of the UI and visual testing:
Why do visual testing, by Applitools but not specific to their product. It’s the basic idea of recording screenshots from functional tests, doing automated image comparison with last round’s screenshots, and serving up the changed ones to a human for validation.
What’s hard about diffing screenshots for visual regression testing? Post from another service provider who promises to do it better (Backtrac).
The third popular provider, Percy, writes about using their own product to validate a large scale restyling of their site.
More detail on what it looks like to test with Applitools Eyes. This is written by their evangelist, using a Calculator app as an example:
This satisfied user of Applitools built a dynamic baseline by comparing staging and production versions of the app pages via Selenium tests. Their approach aims to solve the problem of needing to thumbs up/down on lots of changes to manually re-record the baseline.
How developers can incorporate visual testing into Test Driven Development, with React Storybook. Tests run automated, output gets checked manually. The post doesn’t cover how this would integrate into CI, perhaps with a manual build stage?..
Nice description of why you shouldn’t compare complete screenshots of your website/app pages from someone who did it that way at first, then moved to component-level isolated screenshots, advocates BackstopJS.
Good analysis of what snapshot tests are good for, at unit test level (using Jest image snapshots):
Reflection on how visual testing fits into the work process of a development team, including code review.
Lists of (mostly free open source) visual validation tools:
If you received this email directly then you’re already signed up, thanks! Else if this newsletter issue was forwarded to you and you’d like to get one weekly, then you can subscribe at http://testersdigest.mehras.net
If you come across content worth sharing, please send me a link at firstname.lastname@example.org