iris.loading#

Iris general file loading mechanism.

iris.loading.LOAD_PROBLEMS = <iris.loading.LoadProblems object>#

The global run-time instance of LoadProblems.

See LoadProblems for more details.

class iris.loading.LoadProblems[source]#

Bases: _local

A collection of objects that could not be loaded correctly.

Structured as a list - accessed via LoadProblems.problems - of LoadProblems.Problem instances; see LoadProblems.Problem for more details of the recorded content.

Provided to increase transparency (problem objects are not simply discarded), and to make it possible to fix loading problems without leaving the Iris API.

Expected usage is via the global LOAD_PROBLEMS instance; see the example below.

Examples

(expand to see setup)

This section is not necessary for understanding the examples.

>>> from pathlib import Path
>>> from pprint import pprint
>>> import sys
>>> import warnings

>>> import cf_units
>>> import iris
>>> import iris.common
>>> import iris.coords
>>> from iris.fileformats._nc_load_rules import helpers
>>> import iris.loading
>>> from iris import std_names

>>> # Ensure doctests actually see Warnings that are raised, and that
>>> #  they have a relative path (so a test pass is not machine-dependent).
>>> showwarning_original = warnings.showwarning
>>> warnings.filterwarnings("default")
>>> IRIS_FILE = Path(iris.__file__)
>>> def custom_warn(message, category, filename, lineno, file=None, line=None):
...     filepath = Path(filename)
...     filename = str(filepath.relative_to(IRIS_FILE.parents[1]))
...     sys.stdout.write(warnings.formatwarning(message, category, filename, lineno))
>>> warnings.showwarning = custom_warn

>>> build_dimension_coordinate_original = helpers._build_dimension_coordinate

>>> def raise_example_error_dim(filename, cf_coord_var, coord_name, coord_system):
...     if cf_coord_var.cf_name == "time":
...         raise ValueError("Example dimension coordinate error")
...     else:
...         return build_dimension_coordinate_original(
...             filename, cf_coord_var, coord_name, coord_system
...         )

>>> helpers._build_dimension_coordinate = raise_example_error_dim
>>> air_temperature = std_names.STD_NAMES.pop("air_temperature")
>>> iris.FUTURE.date_microseconds = True

For this example we have ‘booby-trapped’ the Iris loading process to force errors to occur. When we load our first cube, we see the warning that LOAD_PROBLEMS has been added to:

>>> cube_a1b = iris.load_cube(iris.sample_data_path("A1B_north_america.nc"))
iris/...IrisLoadWarning: Not all file objects were parsed correctly. See iris.loading.LOAD_PROBLEMS for details.
  warnings.warn(message, category=IrisLoadWarning)

Remember that Python by default suppresses duplicate warnings, so a second load action does not raise another:

>>> cube_e1 = iris.load_cube(iris.sample_data_path("E1_north_america.nc"))

Examining the contents of LOAD_PROBLEMS we can see that both files experienced some problems:

>>> problems_by_file = iris.loading.LOAD_PROBLEMS.problems_by_file
>>> print([Path(filename).name for filename in problems_by_file.keys()])
['A1B_north_america.nc', 'E1_north_america.nc']

Printing the A1B cube shows that the time dimension coordinate is missing:

>>> print(cube_a1b.summary(shorten=True))
air_temperature / (K)               (-- : 240; latitude: 37; longitude: 49)

A more detailed summary is available by printing LOAD_PROBLEMS:

>>> print(iris.loading.LOAD_PROBLEMS)
<iris.loading.LoadProblems object at ...>:
  .../A1B_north_america.nc: "'air_temperature' is not a valid standard_name", {'standard_name': 'air_temperature'}
  .../A1B_north_america.nc: "Example dimension coordinate error", unknown / (unknown)                 (-- : 240)
  .../E1_north_america.nc: "'air_temperature' is not a valid standard_name", {'standard_name': 'air_temperature'}
  .../E1_north_america.nc: "Example dimension coordinate error", unknown / (unknown)                 (-- : 240)

Below demonstrates how to explore the captured stack traces in detail:

>>> (a1b_full_name,) = [
...     filename for filename in problems_by_file.keys()
...     if Path(filename).name == "A1B_north_america.nc"
... ]
>>> A1B = problems_by_file[a1b_full_name]
>>> for problem in A1B:
...     print(problem.stack_trace.exc_type_str)
ValueError
ValueError

>>> last_problem = A1B[-1]
>>> print("".join(last_problem.stack_trace.format()))
Traceback (most recent call last):
  File ..., in _add_or_capture
    built = build_func()
  File ..., in raise_example_error_dim
ValueError: Example dimension coordinate error

LOAD_PROBLEMS also captures the ‘raw’ information in the object that could not be loaded - the time dimension coordinate. This is captured as a Cube:

>>> print(last_problem.loaded)
unknown / (unknown)                 (-- : 240)
    Attributes:...
        IRIS_RAW                    {'axis': 'T', ...}

Using last_problem.loaded, we can manually reconstruct the missing dimension coordinate:

>>> attributes = last_problem.loaded.attributes[
...     iris.common.LimitedAttributeDict.IRIS_RAW
... ]
>>> pprint(attributes)
{'axis': 'T',
 'bounds': 'time_bnds',
 'calendar': '360_day',
 'standard_name': 'time',
 'units': 'hours since 1970-01-01 00:00:00',
 'var_name': 'time'}

>>> units = cf_units.Unit(attributes["units"], calendar=attributes["calendar"])
>>> dim_coord = iris.coords.DimCoord(
...     points=last_problem.loaded.data,
...     standard_name=attributes["standard_name"],
...     units=units,
... )
>>> cube_a1b.add_dim_coord(dim_coord, 0)
>>> print(cube_a1b.summary(shorten=True))
air_temperature / (K)               (time: 240; latitude: 37; longitude: 49)

Note that we were unable to reconstruct the missing bounds - time_bnds - demonstrating that this error handling is a ‘best effort’ and not perfect. We hope to continually improve it over time.

(expand to see cleanup)

This section is not necessary for understanding the examples.

>>> warnings.showwarning = showwarning_original
>>> warnings.filterwarnings("ignore")
>>> helpers._build_dimension_coordinate = build_dimension_coordinate_original
>>> std_names.STD_NAMES["air_temperature"] = air_temperature
class Problem(filename, loaded, stack_trace)[source]#

Bases: object

A single object that could not be loaded correctly.

Parameters:
filename: str#

The file path/URL that contained the problem object.

loaded: CFVariableMixin | dict[str, Any] | None#

The object that experienced loading problems.

Four possible types:

  • Cube: if problems occurred while building a CFVariableMixin - currently the only handled case is DimCoord - then the information will be stored in a ‘bare bones’ Cube containing only the data array and the attributes. The attributes are un-parsed (they can still contain _FillValue etcetera), and are stored under a special key in the Cube attributes dictionary: IRIS_RAW.

  • dict: if problems occurred while building objects from NetCDF attributes - currently the only handled cases are standard_name, long_name, var_name. The dictionary key is the key of the attribute, and the value is the raw attribute returned by the netCDF4 library.

  • Built objects, such as DimCoord: if the object was built successfully, but could not be added to the Cube being loaded.

  • None: if a loading error occurred, but problems occurred while trying to store the problem object.

stack_trace: TracebackException#

The traceback exception that was raised during loading.

This instance contains rich information to support user-specific workflows, e.g:

  • "".join(stack_trace.format()): the full stack trace as a string - the same way this would be seen at the command line.

  • stack_trace.exc_type_str: the exception type e.g. ValueError.

property problems: list[Problem]#

All recorded LoadProblems.Problem instances.

property problems_by_file: dict[str, list[Problem]]#

All recorded LoadProblems.Problem instances, organised by filename.

Returns:

A dictionary with filenames as keys and lists of LoadProblems.Problem instances as values.

Return type:

dict[str, list[LoadProblems.Problem]]

record(filename, loaded, exception)[source]#

Record a problem object that could not be loaded correctly.

The arguments passed will be used to create a LoadProblems.Problem instance - see that docstring for more details.

Parameters:

  • filename (str) – The file path/URL that contained the problem object.

  • loaded (CFVariableMixin | dict[str, Any] | None) – The object that experienced loading problems. See LoadProblems.Problem.loaded for details on possible values.

  • exception (Exception) – The traceback exception that was raised during loading.

Returns:

The recorded load problem.

Return type:

LoadProblems.Problem

reset(filename=None)[source]#

Remove all recorded LoadProblems.Problem instances.

Parameters:

filename (str, optional) – If provided, only remove problems for this filename.

Return type:

None

iris.loading.load(uris, constraints=None, callback=None)[source]#

Load any number of Cubes for each constraint.

For a full description of the arguments, please see the module documentation for iris.

Parameters:

  • uris (str or pathlib.PurePath) – One or more filenames/URIs, as a string or pathlib.PurePath. If supplying a URL, only OPeNDAP Data Sources are supported.

  • constraints (optional) – One or more constraints.

  • callback (optional) – A modifier/filter function.

Returns:

An iris.cube.CubeList. Note that there is no inherent order to this iris.cube.CubeList and it should be treated as if it were random.

Return type:

iris.cube.CubeList

iris.loading.load_cube(uris, constraint=None, callback=None)[source]#

Load a single cube.

For a full description of the arguments, please see the module documentation for iris.

Parameters:

  • uris – One or more filenames/URIs, as a string or pathlib.PurePath. If supplying a URL, only OPeNDAP Data Sources are supported.

  • constraints (optional) – A constraint.

  • callback (optional) – A modifier/filter function.

Return type:

iris.cube.Cube

iris.loading.load_cubes(uris, constraints=None, callback=None)[source]#

Load exactly one Cube for each constraint.

For a full description of the arguments, please see the module documentation for iris.

Parameters:

  • uris – One or more filenames/URIs, as a string or pathlib.PurePath. If supplying a URL, only OPeNDAP Data Sources are supported.

  • constraints (optional) – One or more constraints.

  • callback (optional) – A modifier/filter function.

Returns:

An iris.cube.CubeList. Note that there is no inherent order to this iris.cube.CubeList and it should be treated as if it were random.

Return type:

iris.cube.CubeList

iris.loading.load_raw(uris, constraints=None, callback=None)[source]#

Load non-merged cubes.

This function is provided for those occasions where the automatic combination of cubes into higher-dimensional cubes is undesirable. However, it is intended as a tool of last resort! If you experience a problem with the automatic combination process then please raise an issue with the Iris developers.

For a full description of the arguments, please see the module documentation for iris.

Parameters:

  • uris – One or more filenames/URIs, as a string or pathlib.PurePath. If supplying a URL, only OPeNDAP Data Sources are supported.

  • constraints (optional) – One or more constraints.

  • callback (optional) – A modifier/filter function.

Return type:

iris.cube.CubeList