Visual regression testing: update workflow

[LarsDenBakker]

When two screenshots do not match, it would be great if there is some workflow for viewing and approving changes.

Are there any existing tools that handle this? Otherwise we could make something, it can be a simple HTML page which iterates through the images which are found on disk.

[web-padawan]

One option to consider is Argos CI: https://github.com/argos-ci/argos

[Westbrook]

I've been playing with this a little bit: https://53319-158297197-gh.circle-artifacts.com/0/test/visual/review/index.html It's built from the current output of https://github.com/modernweb-dev/web/tree/master/packages/test-runner-visual-regression as used in one of the tests passes we make in CI for SWC

[MathieuPuech]

A workflow I currently use:

• A CI job only for pull requests is updating the baseline screenshots. If there is new baseline screenshots, this job commit on the branch the new screenshots
• The maintainer can see on the PR the screenshots updated and approve or not the changes.

It works only if you can have CI on PR of course and in my workflow, the baseline screenshots are committed with the sources.

[WickyNilliams]

Coming from jest/jest-image-snapshot, in case of a mismatch, an image is written to disk that shows next to one another the original image, the new image, and a diff of the two. The file path gets output in the console, so it's very quick to open and visually check. If you then accept the new baseline, it clears up the relevant diff image. I find this workflow very productive and quick.

Edit: Oh I see that a diff image is output. but it is not cleaned up once you update the baseline.

[WickyNilliams]

Is the workflow here like this?

1. start test runner in watch mode
2. visual diff fails
3. stop test runner
4. restart test runner with --update-visual-baseline flag
5. delete any diff files
6. stop test runner
7. start test runner without --update-visual-baseline flag (so that you don't accidentally/incorrectly overwrite any other existing snapshots as you develop)

This feels quite cumbersome, since you need to stop and switch between "modes" (update or fail)?

Coming from jest the workflow is more like:

1. start jest in watch mode
2. visual diff fails
3. press u to accept new baseline

[Westbrook]

That’s an interesting workflow that we’ve not looked at supporting yet. How do you manage creating local screenshots and sharing a project with others on different OS/browser versions?

To dates, I’ve used local testing as a double check here and there but relied on CI testing predominately to support my visual regression testing.

[WickyNilliams]

I've mostly been working in small teams where we all have mac OS (not by design, rather by chance). For CI we've used a mac image for consistent results. In a more technologically diverse team I'd consider docker.

I really value working with local files since the workflow is simpler and quicker.