Writing Tests#

Note

If you’re converting UnitTest tests to PyTest, check out Converting From unittest to pytest.

Test Categories#

There are two main categories of tests within Iris:

  • unit tests

  • integration tests

Ideally, all code changes should be accompanied by one or more unit tests, and by zero or more integration tests.

Code changes should be accompanied by enough unit tests to give a high degree of confidence that the change works as expected. In addition, the unit tests can help describe the intent behind a change.

The docstring for each test module must state the unit under test. For example:

"""Unit tests for the `iris.experimental.raster.export_geotiff` function."""

When testing a class, all the tests must reside in the module:

lib/iris/tests/unit/<fully/qualified/module>/test_<ClassName>.py

When testing a function, all the tests must reside in the module:

lib/iris/tests/unit/<fully/qualified/module>/test_<function_name>.py

Some code changes may require tests which exercise several units in order to demonstrate an important consequence of their interaction which may not be apparent when considering the units in isolation. These tests must be placed in the lib/iris/tests/integration folder.

With integration tests, folders and files must be created as required to help developers locate relevant tests. It is recommended they are named according to the capabilities under test, e.g. metadata/test_pp_preservation.py, and not named according to the module(s) under test.

If in any doubt about what tests to add or how to write them please feel free to submit a pull-request in any state and ask for assistance.

PyTest Style Guide#

Note

If you’re converting UnitTest tests to PyTest, check out Converting From unittest to pytest.

This style guide should be approached pragmatically. Many of the guidelines laid out below will not be practical in every scenario, and as such should not be considered firm rules.

At time of writing, some existing tests have already been written in PyTest, so might not be abiding by these guidelines.

conftest.py#

There should be a conftest.py file in the root/unit and root/integration folders. Additional lower level conftests can be added if it is agreed there is a need.

Fixtures#

As far as is possible, the actual test function should do little else but the actual assertion. Separating off preparation into fixtures may make the code harder to follow, so compromises are acceptable. For example, setting up a test Cube should be a fixture, whereas creating a simple string (expected = "foo"), or a single use setup, should not be a fixture.

New fixtures should always be considered for conftest when added. If it is decided that they are not suitably reusable, they can be placed within the local test file.

Parameterisation#

Though it is a useful tool, we should not be complicating tests to work around parameters; they should only be used when it is simple and apparent to implement.

Where you are parameterising multiple tests with the same parameters, it is usually prudent to use the parameterisation within fixtures. When doing this, ensure within the tests that it is apparent that they are being parameterised, either within the fixture name or with comments.

All parameterisation benefits from ids, and so should be used where possible.

Mocks#

Any mocking should be done with pytest.mock, and monkeypatching where suitable.

Note

If you think we’re missing anything important here, please consider creating an issue or discussion and share your ideas with the team!

Classes#

How and when to group tests within classes can be based on personal opinion, we do not deem consistency on this a vital concern.

Naming Test Classes and Functions#

When testing classes and their methods, each tested method within a test module may have corresponding test classes, for example:

  • Test_<name of public method>

  • Test_<name of public method>__<aspect of method>

Within these test classes, the test methods must be named according to the aspect of the tested method which they address.

Examples:

All unit tests for iris.cube.Cube reside in:

lib/iris/tests/unit/cube/test_Cube.py

Within that file the tests might look something like:

# A single test for the Cube.xml() method.
def test_xml_some_general_stuff(self):
    ...


# A single test for the Cube.xml() method, focussing on the behaviour of
# the checksums.
def test_xml_checksum_ignores_masked_values(self):
    ...


# Tests for the Cube.add_dim_coord() method.
class Test_add_dim_coord:
    def test_normal_usage(self):
        ...

    def test_coord_already_present(self):
        ...

When testing functions, within the test module there may be test classes, for example:

  • Test

  • TestAspectOfFunction

Within those test classes, the test methods must be named according to the aspect of the tested function which they address.

Examples:

All unit tests for iris.experimental.raster.export_geotiff() must reside in:

lib/iris/tests/unit/experimental/raster/test_export_geotiff.py

Within that file the tests might look something like:

# Tests focussing on the handling of different data types.
class TestDtypeAndValues:
    def test_int16(self):
        ...

    def test_int16_big_endian(self):
        ...


# Tests focussing on the handling of different projections.
def test_no_ellipsoid(self):
    ...

There is no fixed naming scheme for integration tests.

Testing tools#

Note

iris.tests.IrisTest has been deprecated, and replaced with the iris.tests._shared_utils module.

Iris has various internal convenience functions and utilities available to support writing tests. Using these makes tests quicker and easier to write, and also consistent with the rest of Iris (which makes it easier to work with the code). Most of these conveniences are accessed through the iris.tests._shared_utils module.

Tip

All functions listed on this page are defined within iris.tests._shared_utils. They can be accessed within a test using _shared_utils.example_function.

Custom assertions#

iris.tests._shared_utils supports a variety of custom pytest-style assertions, such as assert_array_equal(), and assert_array_almost_equal().

Saving results#

Some tests compare the generated output to the expected result contained in a file. Custom assertions for this include assert_CML_approx_data() assert_CDL() assert_CML() and assert_text_file(). See docstrings for more information.

Note

Sometimes code changes alter the results expected from a test containing the above methods. These can be updated by removing the existing result files and then running the file containing the test with a --create-missing command line argument, or setting the IRIS_TEST_CREATE_MISSING environment variable to anything non-zero. This will create the files rather than erroring, allowing you to commit the updated results.

Capturing exceptions and logging#

_shared_utils includes several context managers that can be used to make test code tidier and easier to read. These include assert_no_warnings_regexp() and assert_logs().

Graphic tests#

As a package capable of generating graphical outputs, Iris has utilities for creating and updating graphical tests - see Adding or Updating Graphics Tests for more information.