Inconsistent units associated with txxETCCDI variable

[corviday]

Some txxETCCDI files have degrees_c as their unit; others degC. The /data query quite sensibly refuses to construct a timeseries from files with different units, and returns a 500 error.

Solutions might be:

• use a list of "synonymous" units (degC / degrees_c for temperature and mm / kg d-1 m-2 for precipitation)
• standardize the units in existing files and reindex them, if it's only a few

I haven't investigated thoroughly, and it's possible that this error is actually just a side effect of the /data query's present erroneous attempt to construct a timeseries from both nomimal-time and multi-year-mean datasets, in which case this bug will go away when PR 82 is merged.

EDIT: PR 82 has been merged, but did not solve this bug.

[jameshiebert]

A synonymous list would be quick and dirty and would probably do the job since we actually have very few of these.

A more comprehensive solution could be using something like pint to test units for comparability (and possibly convertibility). I've done this in crmprtd and while it works OK, it's not without its pitfalls. That and mm and kg d-1 m-2 are technically not the same unit (and thus will never be recognized as comparable by the physical unit libraries), even though the express the same quantity in a climate science context.

[corviday]

Got a similar error message for a tn10p dataset:
does not have consistent units {'%', 'days'}

The official definition of tn10p says it's a percentage, so this is more a case of "wrong units" than "multiple names for the same unit".
TN10p, Percentage of days when TN < 10th percentile

[corviday]

SELECT variable_aliases.variable_units, variable_aliases.variable_standard_name
FROM ce_meta.variable_aliases
GROUP BY variable_aliases.variable_units, variable_aliases.variable_standard_name
ORDER BY variable_aliases.variable_standard_name;

units	variable_standard_name
K	air_temperature
degC	air_temperature
degrees_C	dtrETCCDI
degC	dtrETCCDI
kg d-1 m-2	precipitation_flux
kg m-2 d-1	precipitation_flux
days	tn10pETCCDI
%	tn10pETCCDI
days	tn90pETCCDI
%	tn90pETCCDI
degC	tnnETCCDI
degrees_C	tnnETCCDI
degC	tnxETCCDI
degrees_C	tnxETCCDI
days	tx10pETCCDI
%	tx10pETCCDI
days	tx90pETCCDI
%	tx90pETCCDI
degrees_C	txnETCCDI
degC	txnETCCDI
degC	txxETCCDI
degrees_C	txxETCCDI

Looks like there are four possible unit inconsistencies in the current CE data:

• degC <-> degrees_C - synonyms, handleable with a synonym list or pint
• kg d-1 m-2 <-> kg m-2 d-1 - synonyms, handleable with a synonym list or pint
• K <-> degC - not synonyms, but a trivial conversion with pint or similar
• % <-> days - present in several climdices, not a trivial conversion, would require direct intervention

[corviday]

For tn10p, tn90p, tx10p, and tx90p, only the data located in the bc_subset directory has the incorrect unit "days"; the rest of the data for these variables has the correct unit "%".

We're not using the bc_subset data because it duplicates all-Canada data we have (and confuses PCEX). So these variables should be considered unaffected by the issue.

[corviday]

This issue was fixed for all files in active PCEX useage by normalizing data when climatologies were generated. There are still some contradictory files, but they're legacy PDP files, or files in the redundant and screwy bc_subset.

All PCEX datasets now agree on which variables have which units. (Though some of them still have incorrect units, but the incorrect unit appears in all datasets):

• prsn data should be in cm, not mm climate-explorer-data-prep#71
• Wrong units in datafiles climate-explorer-data-prep#51

[corviday]

From the logs:

2019-10-13 20:19:56 [2085] [INFO] 172.18.0.1 - - [13/Oct/2019:20:19:56 +0000] "GET /api/data?ensemble_name=ce_files&model=CanESM2&variable=dtrETCCDI&emission=historical,+rcp85&timescale=yearly&time=0&area= HTTP/1.1" 500 291 "https://services.pacificclimate.org/pcex/app/" "Mozilla/5
.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36"
2019-10-13 20:19:57 [2073] [ERROR] Exception on /api/data [GET]
Traceback (most recent call last):
  File "/app/ce/api/data.py", line 142, in data
    run_result = result[data_file_variable.file.run.name]
KeyError: 'r1i1p1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/dist-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/app/ce/views.py", line 11, in api_request
    return ce.api.call(db.session, *args, **kwargs)
  File "/app/ce/api/__init__.py", line 75, in call
    rv = func(session, **args)
  File "/app/ce/api/data.py", line 147, in data
    data_file_variable.file.run, variable),
  File "/app/ce/api/util.py", line 31, in get_units_from_run_object
    raise Exception("File list {} does not have consistent units {}".format(run.files, units))
Exception: File list [<modelmeta.v2.DataFile object at 0x7f59c0f395f8>, <modelmeta.v2.DataFile object at 0x7f59c0f39668>, <modelmeta.v2.DataFile object at 0x7f59c0f39780>, <modelmeta.v2.DataFile object at 0x7f59c0f39898>, <modelmeta.v2.DataFile object at 0x7f59c0f399b0>, <modelmeta.v2.DataFile object at 0x7f59c0f39ac8>, <modelmeta.v2.DataFile object at 0x7f59c0f39be0>, <modelmeta.v2.DataFile object at 0x7f59c0f39cf8>, <modelmeta.v2.DataFile object at 0x7f59c0f39e10>, <modelmeta.v2.DataFile object at 0x7f59c0f39f28>, <modelmeta.v2.DataFile object at 0x7f59c0f42080>, <modelmeta.v2.DataFile object at 0x7f59c0f42198>, <modelmeta.v2.DataFile object at 0x7f59c0f422b0>, <modelmeta.v2.DataFile object at 0x7f59c0f424e0>, <modelmeta.v2.DataFile object at 0x7f59c0f42668>, <modelmeta.v2.DataFile object at 0x7f59c0f427f0>, <modelmeta.v2.DataFile object at 0x7f59c0f42978>, <modelmeta.v2.DataFile object at 0x7f59c0f42b00>, <modelmeta.v2.DataFile object at 0x7f59c0f42c88>......... File object at 0x7f59c1245748>, <modelmeta.v2.DataFile object at 0x7f59c1245eb8>, <modelmeta.v2.DataFile object at 0x7f59c1245358>, <modelmeta.v2.DataFile object at 0x7f59c1245860>]
does not have consistent units {'degC', 'degrees_C'}

[corviday]

I've split this issue into two related issues.

• This data preparation issue for fixing the dtrETCCDI data.

• This backend issue for upgrading the backend to handle unit conversions, which would be useful for the frontend and users.