Continuous Integration (CI) Testing

The Iris GitHub repository is configured to run checks against all its branches automatically whenever a pull-request is created, updated or merged. The checks performed are:

Cirrus-CI

Iris unit and integration tests are an essential mechanism to ensure that the Iris code base is working as expected. Running the Tests may be performed manually by a developer locally. However Iris is configured to use the cirrus-ci service for automated Continuous Integration (CI) testing.

The cirrus-ci configuration file .cirrus.yml in the root of the Iris repository defines the tasks to be performed by cirrus-ci. For further details refer to the Cirrus-CI Documentation. The tasks performed during CI include:

  • linting the code base and ensuring it adheres to the black format

  • running the system, integration and unit tests for Iris

  • ensuring the documentation gallery builds successfully

  • performing all doc-tests within the code base

  • checking all URL references within the code base and documentation are valid

The above cirrus-ci tasks are run automatically against all Iris branches on GitHub whenever a pull-request is submitted, updated or merged. See the Cirrus-CI Dashboard for details of recent past and active Iris jobs.

Cirrus CI Test environment

The test environment on the Cirrus-CI service is determined from the requirement files in requirements/ci/py**.yml. These are conda environment files that list the entire set of build, test and run requirements for Iris.

For reproducible test results, these environments are resolved for all their dependencies and stored as lock files in requirements/ci/nox.lock. The test environments will not resolve the dependencies each time, instead they will use the lock file to reproduce the same exact environment each time.

If you have updated the requirement yaml files with new dependencies, you will need to generate new lock files. To do this, run the command:

python tools/update_lockfiles.py -o requirements/ci/nox.lock requirements/ci/py*.yml

or simply:

make lockfiles

and add the changed lockfiles to your pull request.

New lockfiles are generated automatically each week to ensure that Iris continues to be tested against the latest available version of its dependencies. Each week the yaml files in requirements/ci are resolved by a GitHub Action. If the resolved environment has changed, a pull request is created with the new lock files. The CI test suite will run on this pull request and fixes for failed tests can be pushed to the auto-update-lockfiles branch to be included in the PR. Once a developer has pushed to this branch, the auto-update process will not run again until the PR is merged, to prevent overwriting developer commits. The auto-updater can still be invoked manually in this situation by going to the GitHub Actions page for the workflow, and manually running using the “Run Workflow” button. By default, this will also not override developer commits. To force an update, you must confirm “yes” in the “Run Worflow” prompt.

Skipping Cirrus-CI Tasks

As a developer you may wish to not run all the CI tasks when you are actively developing e.g., you are writing documentation and there is no need for linting, or long running compute intensive testing tasks to be executed.

As a convenience, it is possible to easily skip one or more tasks by setting the appropriate environment variable within the .cirrus.yml file to a non-empty string:

  • SKIP_LINT_TASK to skip flake8 linting and black formatting

  • SKIP_TEST_MINIMAL_TASK to skip restricted unit and integration testing

  • SKIP_TEST_FULL_TASK to skip full unit and integration testing

  • SKIP_GALLERY_TASK to skip building the documentation gallery

  • SKIP_DOCTEST_TASK to skip running the documentation doc-tests

  • SKIP_LINKCHECK_TASK to skip checking for broken documentation URL references

  • SKIP_ALL_TEST_TASKS which is equivalent to setting SKIP_TEST_MINIMAL_TASK and SKIP_TEST_FULL_TASK

  • SKIP_ALL_DOC_TASKS which is equivalent to setting SKIP_GALLERY_TASK, SKIP_DOCTEST_TASK, and SKIP_LINKCHECK_TASK

e.g., to skip the linting task, the following are all equivalent:

SKIP_LINT_TASK: "1"
SKIP_LINT_TASK: "true"
SKIP_LINT_TASK: "false"
SKIP_LINT_TASK: "skip"
SKIP_LINT_TASK: "unicorn"

GitHub Checklist

An example snapshot from a successful GitHub pull-request shows all tests passing:

../_images/ci_checks.png

If any CI tasks fail, then the pull-request is unlikely to be merged to the Iris target branch by a core developer.

SciTools CLA Checker

A bot which checks that the GitHub author of the pull-request has signed the SciTools Contributor’s License Agreement (CLA). For more information on this please see https://scitools.org.uk/organisation.html#governance.

pre-commit CI

A CI service for the pre-commit framework that checks and auto fixes all pull-requests given the Iris GitHub repository .pre-commit-config.yaml.

See the pre-commit.ci dashboard for details of recent past and active Iris jobs.