Iris may be used to create various forms of graphical output; to ensure the output is consistent, there are automated tests to check against known acceptable graphical output. See Running the Tests for more information.
At present graphical tests are used in the following areas of Iris:
Gallery plots contained in
Iris uses many dependencies that provide functionality, an example that applies here is matplotlib. For more information on the dependences, see Installing Iris. When there are updates to the matplotlib or a dependency of matplotlib, this may result in a change in the rendered graphical output. This means that there may be no changes to Iris, but due to an updated dependency any automated tests that compare a graphical output to a known acceptable output may fail. The failure may also not be visually perceived as it may be a simple pixel shift.
The Iris Cirrus-CI matrix defines multiple test runs that use different versions of Python to ensure Iris is working as expected.
To make this manageable, the
routine tests against multiple alternative acceptable results. It does
this using an image hash comparison technique which avoids storing
reference images in the Iris repository itself.
This consists of:
iris.tests.IrisTest_nometa.check_graphicfunction uses a perceptual image hash of the outputs (see https://github.com/JohannesBuchner/imagehash) as the basis for checking test results.
The hashes of known acceptable results for each test are stored in a lookup dictionary, saved to the repo file
An actual reference image for each hash value is stored in a separate public repository https://github.com/SciTools/test-iris-imagehash.
The reference images allow human-eye assessment of whether a new output is judged to be close enough to the older ones, or not.
The utility script
iris/tests/idiff.pyautomates checking, enabling the developer to easily compare proposed new acceptable result images against the existing accepted reference images, for each failing test.
Reviewing Failing Tests
When you find that a graphics test in the Iris testing suite has failed, following changes in Iris or the run dependencies, this is the process you should follow:
Create a new, empty directory to store temporary image results, at the path
lib/iris/tests/result_image_comparisonin your Iris repository checkout.
In your Iris repo root directory, run the relevant (failing) tests directly as python scripts, or by using a command such as:
python -m unittest discover paths/to/test/files
iris/lib/iris/testsfolder, run the command:
This will open a window for you to visually inspect side-by-side old, new and difference images for each failed graphics test. Hit a button to either accept, reject or skip each new result.
If the change is accepted:
the imagehash value of the new result image is added into the relevant set of ‘valid result hashes’ in the image result database file,
the relevant output file in
tests/result_image_comparisonis renamed according to the image hash value, as
<hash>.png. A copy of this new PNG file must then be added into the reference image repository at https://github.com/SciTools/test-iris-imagehash (See below).
If a change is skipped:
no further changes are made in the repo.
when you run
iris/tests/idiff.pyagain, the skipped choice will be presented again.
If a change is rejected:
the output image is deleted from
when you run
iris/tests/idiff.pyagain, the skipped choice will not appear, unless the relevant failing test is re-run.
Now re-run the tests. The new result should now be recognised and the relevant test should pass. However, some tests can perform multiple graphics checks within a single test case function. In those cases, any failing check will prevent the following ones from being run, so a test re-run may encounter further (new) graphical test failures. If that happens, simply repeat the check-and-accept process until all tests pass.
Add Your Changes to Iris
To add your changes to Iris, you need to make two pull requests (PR).
The first PR is made in the
test-iris-imagehashrepository, at https://github.com/SciTools/test-iris-imagehash.
First, add all the newly-generated referenced PNG files into the
images/v4directory. In your Iris repo, these files are to be found in the temporary results folder
Then, to update the file which lists available images,
v4_files_listing.txt, run from the project root directory:
Create a PR proposing these changes, in the usual way.
The second PR is created in the Iris repository, and should only include the change to the image results database,
tests/results/imagerepo.json. The description box of this pull request should contain a reference to the matching one in
result_image_comparison folder is covered by a project
.gitignore setting, so those files will not show up in a
git status check.
The Iris pull-request will not test successfully in Cirrus-CI until the
test-iris-imagehash pull request has been merged. This is because there
is an Iris test which ensures the existence of the reference images (uris)
for all the targets in the image results database. It will also fail
if you forgot to run
recreate_v4_files_listing.py to update the
image-listing file in