Allow humans to verify screenshots that differ in irrelevant ways
Asserting screenshots has the inherent problem that while some aspects of graphical design are irrelevant or unpredictable for technical reasons (such as font rendering when it comes to comparing results to the pixel), it is impractical and generally impossible to programmatically express what's allowed or not. Therefore, it should be possible to fall back to human judgement where appropriate.
Providing a set of expected images for each screenshot assertions will probably not scale. Another possibility is to keep reference screenshots outside the code base, in some cache on the machine that took them in the first place.
- This would require some kind of GUI tool that allows a developer to easily verify any screenshots that were dumped during a test run and are not yet in the cache.
- Also, a way would need to be found to perform this human check for tests run by Jenkins - some kind of web interface, FTP access, e-mail, whatever.
- To decrease the amount human interaction necessary, screenshots could be accepted automatically if they are very close to some expected image that does reside within the code base (e.g. in cases that don't involve much text).
- One problem with any solution involving a cache will be cache invalidation which is needed in ensure that tests don't keep passing by accident.
- Make a plan for a minimal solution (4h)