Continuous Integration (CI) Testing
Note
Iris is currently supported and tested against Python 3.8 running on Linux. We do not currently actively test on other platforms such as Windows or macOS.
The Iris GitHub repository is configured to run checks against all its branches automatically whenever a pull-request is created, updated or merged. The checks performed are:
Cirrus-CI
Iris unit and integration tests are an essential mechanism to ensure that the Iris code base is working as expected. Running the Tests may be performed manually by a developer locally. However Iris is configured to use the cirrus-ci service for automated Continuous Integration (CI) testing.
The cirrus-ci configuration file .cirrus.yml in the root of the Iris repository defines the tasks to be performed by cirrus-ci. For further details refer to the Cirrus-CI Documentation. The tasks performed during CI include:
linting the code base and ensuring it adheres to the black format
running the system, integration and unit tests for Iris
ensuring the documentation gallery builds successfully
performing all doc-tests within the code base
checking all URL references within the code base and documentation are valid
The above cirrus-ci tasks are run automatically against all Iris branches on GitHub whenever a pull-request is submitted, updated or merged. See the Cirrus-CI Dashboard for details of recent past and active Iris jobs.
Cirrus CI Test environment
The test environment on the Cirrus-CI service is determined from the requirement files
in requirements/ci/py**.yml
. These are conda environment files that list the entire
set of build, test and run requirements for Iris.
For reproducible test results, these environments are resolved for all their dependencies
and stored as lock files in requirements/ci/nox.lock
. The test environments will not
resolve the dependencies each time, instead they will use the lock file to reproduce the
same exact environment each time.
If you have updated the requirement yaml files with new dependencies, you will need to generate new lock files. To do this, run the command:
python tools/update_lockfiles.py -o requirements/ci/nox.lock requirements/ci/py*.yml
or simply:
make lockfiles
and add the changed lockfiles to your pull request.
New lockfiles are generated automatically each week to ensure that Iris continues to be
tested against the latest available version of its dependencies.
Each week the yaml files in requirements/ci
are resolved by a GitHub Action.
If the resolved environment has changed, a pull request is created with the new lock files.
The CI test suite will run on this pull request and fixes for failed tests can be pushed to
the auto-update-lockfiles
branch to be included in the PR.
Once a developer has pushed to this branch, the auto-update process will not run again until
the PR is merged, to prevent overwriting developer commits.
The auto-updater can still be invoked manually in this situation by going to the GitHub Actions
page for the workflow, and manually running using the “Run Workflow” button.
By default, this will also not override developer commits. To force an update, you must
confirm “yes” in the “Run Worflow” prompt.
Skipping Cirrus-CI Tasks
As a developer you may wish to not run all the CI tasks when you are actively developing e.g., you are writing documentation and there is no need for linting, or long running compute intensive testing tasks to be executed.
As a convenience, it is possible to easily skip one or more tasks by setting the appropriate environment variable within the .cirrus.yml file to a non-empty string:
SKIP_TEST_MINIMAL_TASK
to skip restricted unit and integration testingSKIP_TEST_FULL_TASK
to skip full unit and integration testingSKIP_GALLERY_TASK
to skip building the documentation gallerySKIP_DOCTEST_TASK
to skip running the documentation doc-testsSKIP_LINKCHECK_TASK
to skip checking for broken documentation URL referencesSKIP_ALL_TEST_TASKS
which is equivalent to settingSKIP_TEST_MINIMAL_TASK
andSKIP_TEST_FULL_TASK
SKIP_ALL_DOC_TASKS
which is equivalent to settingSKIP_GALLERY_TASK
,SKIP_DOCTEST_TASK
, andSKIP_LINKCHECK_TASK
e.g., to skip the linting task, the following are all equivalent:
SKIP_LINT_TASK: "1"
SKIP_LINT_TASK: "true"
SKIP_LINT_TASK: "false"
SKIP_LINT_TASK: "skip"
SKIP_LINT_TASK: "unicorn"
GitHub Checklist
An example snapshot from a successful GitHub pull-request shows all tests passing:
If any CI tasks fail, then the pull-request is unlikely to be merged to the Iris target branch by a core developer.
SciTools CLA Checker
A bot which checks that the GitHub author of the pull-request has signed the SciTools Contributor’s License Agreement (CLA). For more information on this please see https://scitools.org.uk/organisation.html#governance.
pre-commit CI
A CI service for the pre-commit framework that checks and auto fixes all pull-requests given the Iris GitHub repository .pre-commit-config.yaml.
See the pre-commit.ci dashboard for details of recent past and active Iris jobs.