Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to cell_method parsing #6083

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

ukmo-ccbunney
Copy link

@ukmo-ccbunney ukmo-ccbunney commented Jul 24, 2024

🚀 Pull Request

Description

Iris fails to load cubes from a netCDF file if the cell_method attribute is malformed and throws this slightly unhelpful error:

File /opt/scitools/conda/deployments/default-2024_02_20/lib/python3.11/site-packages/iris/fileformats/_nc_load_rules/helpers.py:267, in _split_cell_methods(nc_cell_methods)
    265 for ii in range(len(name_start_inds) - 1):
    266     method_indices.append((name_start_inds[ii], name_start_inds[ii + 1]))
--> 267 method_indices.append((name_start_inds[-1], len(nc_cell_methods)))
    269 # Index the string and match against each substring
    270 nc_cell_methods_matches = []

IndexError: list index out of range

For the netCDF files in my particular case, inspection of the stack trace reveals it was failing to correctly parse the cell_method attribute which was missing a space between the colon separator and time method. I.e.

cell_methods = "time:point"

rather than:

cell_methods = "time: point"

This PR makes two changes to the cell method parsing in fileformats/_nc_load_rules/helpers.py:

  1. In _split_cell_methods: If the value of cell_methods does not match the expected name: pattern, a Warning is issued and the function returns an empty list so cube loading can continue. This is consistent with the behaviour further down the function if the name: value pattern cannot be matched.

  2. Updates _CM_PARSE_NAME and _CM_PARSE regexes to make the pattern matching a little more lenient by making the space after the colon separator between the name and method pair optional. The CF Convention states that the pairs are "blank-separated list of name: method pairs". It does show a space after the colon in the examples it gives, but the CF Compliance checker is happy with no space after the colon.

Unit tests have been updated to test that:

  1. Malformed cell_method raises a warning; e.g. name but no method, such as cell_method='time'
  2. Space after colon between name and method is optional; e.g. cell_method='time:mean'

Tested on Iris version 3.10.0.dev96.

Add any of the below labels to trigger actions on this PR:

@CLAassistant
Copy link

CLAassistant commented Jul 24, 2024

CLA assistant check
All committers have signed the CLA.

ukmo-ccbunney and others added 4 commits July 29, 2024 17:22
@ukmo-ccbunney ukmo-ccbunney marked this pull request as ready for review July 29, 2024 16:23
@ukmo-ccbunney
Copy link
Author

Modified slightly to address some failing unit tests.
All tests passing locally for me now.

@bjlittle bjlittle removed their assignment Sep 2, 2024
@trexfeathers
Copy link
Contributor

trexfeathers commented Sep 11, 2024

From @SciTools/peloton:

Related to #5165.

At the very least there needs to be an informative warning, otherwise what's the point in Iris being strict about CF.

Also, if the CF checker is OK with this, maybe we should be OK with it also? @pp-mo would love your opinion on the CF checker's place in these decisions.

It could be in this case that CF actually doesn't care about the space (why should it), but just that all the examples have one because it's a bit easier to read.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

Loading cube fails if cell_methods is not valid
4 participants