Project

General

Profile

Bug #12317

Allow humans to verify screenshots that differ in irrelevant ways

Added by Thomas Lotze over 7 years ago. Updated about 4 years ago.

Status:
Rejected
Priority:
Normal (P3)
Assignee:
-
Category:
Backend Webdriver
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
4.25 h
Keywords:

Description

Asserting screenshots has the inherent problem that while some aspects of graphical design are irrelevant or unpredictable for technical reasons (such as font rendering when it comes to comparing results to the pixel), it is impractical and generally impossible to programmatically express what's allowed or not. Therefore, it should be possible to fall back to human judgement where appropriate.

Providing a set of expected images for each screenshot assertions will probably not scale. Another possibility is to keep reference screenshots outside the code base, in some cache on the machine that took them in the first place.

  • This would require some kind of GUI tool that allows a developer to easily verify any screenshots that were dumped during a test run and are not yet in the cache.
  • Also, a way would need to be found to perform this human check for tests run by Jenkins - some kind of web interface, FTP access, e-mail, whatever.
  • To decrease the amount human interaction necessary, screenshots could be accepted automatically if they are very close to some expected image that does reside within the code base (e.g. in cases that don't involve much text).
  • One problem with any solution involving a cache will be cache invalidation which is needed in ensure that tests don't keep passing by accident.

TODO:

  • Make a plan for a minimal solution (4h)

Related issues

Related to gocept.selenium - Bug #12338: assertScreenshot und TexteRejected

History

#1 Updated by Sebastian Wehrmann over 7 years ago

  • Category set to Backend Webdriver
  • Status changed from New to Confirmed

Gegenentwurf zu #12338

#2 Updated by Sebastian Wehrmann over 7 years ago

  • Description updated (diff)
  • Estimated time set to 4.25 h

#3 Updated by Michael Howitz about 4 years ago

  • Status changed from Confirmed to Rejected

When tests are run using py.test the screenshots are accessible via Jenkins.
Current solution for the fonts problem: Run the assert screenshot tests only on Jenkins.

Also available in: Atom PDF