Standardised col_name check #133
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Brief overview of changes
I've expanded the ethnicity headers check to allow checks of any standardised col_name that comes under harmonisation rules. This runs off a look-up table of common aliases or sub-strings used in publications that we can check on and their related standard col_names. The test does a scan for these aliases / sub-strings and throws an error if it finds them as non-standard col_names.
Why are these changes being made?
End users currently have a confusing collection of different col_names across different data files. We'd like to improve consistency in the data catalogue.
Detailed description of changes
The initial look-up table is provided in data/harmonised_col_names.csv. This can be added to as more themes fall under the standardised bracket. For now we have:
The
ethnicity_headers()function has been made more generic and renamed asstandard_filter_headers().Lots of the test data failed the more generalised test, so I've gone through and updated all the data files that were now failing to meet the latest standards.
Additional information for reviewers
I've not added any tests for standardised filter items as part of this test as it was already breaking lots of the unit tests as it was, so has become quite a substantial update despite it being one small test that's been updated. Feel like this is about right for a single PR and I can add further tests on a second PR.
Issue ticket number/s and link