Adding or Updating Graphics Tests

Note

If a large number of images tests are failing due to an update to the libraries used for image hashing, follow the instructions on Refreshing the Stored Hashes.

Generating New Results

When you find that a graphics test in the Iris testing suite has failed, following changes in Iris or the run dependencies, this is the process you should follow:

1. Create a new, empty directory to store temporary image results, at the path lib/iris/tests/result_image_comparison in your Iris repository checkout.

2. Run the relevant (failing) tests directly as python scripts, or using pytest.

The results of the failing image tests will now be available in lib/iris/tests/result_image_comparison.

Note

The result_image_comparison folder is covered by a project .gitignore setting, so those files will not show up in a git status check.

Reviewing Failing Tests

1. Run iris/lib/iris/tests/graphics/idiff.py with python, e.g.:

   python idiff.py
   

   This will open a window for you to visually inspect side-by-side old, new and difference images for each failed graphics test. Hit a button to either accept, reject or skip each new result.

   If the change is accepted:

   • the imagehash value of the new result image is added into the relevant set of ‘valid result hashes’ in the image result database file, tests/results/imagerepo.json

   • the relevant output file in tests/result_image_comparison is renamed according to the test name. A copy of this new PNG file must then be added into the iris-test-data repository, at SciTools/iris-test-data (See below).

   If a change is skipped:

   • no further changes are made in the repo.

   • when you run iris/tests/idiff.py again, the skipped choice will be presented again.

   If a change is rejected:

   • the output image is deleted from result_image_comparison.

   • when you run iris/tests/idiff.py again, the skipped choice will not appear, unless the relevant failing test is re-run.

2. Now re-run the tests. The new result should now be recognised and the relevant test should pass. However, some tests can perform multiple graphics checks within a single test case function. In those cases, any failing check will prevent the following ones from being run, so a test re-run may encounter further (new) graphical test failures. If that happens, simply repeat the check-and-accept process until all tests pass.

3. You’re now ready to Add Your Changes to Iris

Adding a New Image Test

If you attempt to run idiff.py when there are new graphical tests for which no baseline yet exists, you will get a warning that idiff.py is Ignoring unregistered test result.... In this case,

1. rename the relevant images from iris/tests/result_image_comparison by

   • removing the result- prefix

   • fully qualifying the test name if it isn’t already (i.e. it should start iris.tests...``or ``gallery_tests...)

2. run the tests in the mode that lets them create missing data (see Saving results). This will update imagerepo.json with the new test name and image hash.

3. and then add them to the Iris test data as covered in Add Your Changes to Iris.

Refreshing the Stored Hashes

From time to time, a new version of the image hashing library will cause all image hashes to change. The image hashes stored in tests/results/imagerepo.json can be refreshed using the baseline images stored in the iris-test-data repository (at SciTools/iris-test-data) using the script tests/graphics/recreate_imagerepo.py. Use the --help argument for the command line arguments.

Add Your Changes to Iris

To add your changes to Iris, you need to make two pull requests (PR).

1. The first PR is made in the iris-test-data repository, at SciTools/iris-test-data.

   • Add all the newly-generated referenced PNG files into the test_data/images directory. In your Iris repo, these files are to be found in the temporary results folder iris/tests/result_image_comparison.

   • Create a PR proposing these changes, in the usual way.

2. The second PR is the one that makes the changes you intend to the Iris repository. The description box of this pull request should contain a reference to the matching one in iris-test-data.

   • This PR should include updating the version of the test data in .github/workflows/ci-tests.yml and .github/workflows/ci-docs-tests.yml to the new version created by the merging of your iris-test-data PR.