diff --git a/docs/Sprint-Review/sprint-101-summary.md b/docs/Sprint-Review/sprint-101-summary.md new file mode 100644 index 0000000000..1071e62980 --- /dev/null +++ b/docs/Sprint-Review/sprint-101-summary.md @@ -0,0 +1,87 @@ +# sprint-101-summary + +6/5/2024 - 6/18/2024 + +**Dev:** + +_**Prioritized DAC and Notifications Work**_ + +* As sys admin, I want to be able to reparse datafile sets #2978 +* As a software engineer, I want to be able to test django-admin-508 #3008 +* As tech lead, I need the STT filter for search\_indexes to be updated #2950 +* As a data analyst I want to be notified of approaching data deadlines #2473 +* add `SENDGRID_API_KEY` to deploy.backend.sh #2677 +* Implement (small) data lifecycle (backup/archive ES) #3004 +* As a developer I want to test django-508 repo #2980\ + + +**DevOps:** + +_**Successful deployments across environments and pipeline stability investments**_ + +* Application health monitoring #831 + +**Design:** + +_**Close out error guide work, coordinate with dev on a plan for Cat 3 problems introduced by Cat 2 work, support spec-writing for upcoming work, and continued error audit dev ticket refinement.**_ + +* Error Report Guide #2847 is going through final edits + * Walk-on Dear Colleague letter link update to this PR (or spin up a separate ticket if deployment of the letter to OFA's website doesn't align to this) +* Deliver spec for #3014 (Blanked-out values in Submission History) +* \#3021 Updated KC Release Notes & Update Indicator FAQ - stretch goal for this sprint +* Write follow-on / spec tickets from #2909 findings - stretch/ongoing lift +* Category 3 error messages clean-up #2792 - stretch/ongoing lift + +## Tickets + +### Completed/Merged + +* [#2980 As a developer I want to test django-508 repo](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2980) +* [#2892 Correct misleading error message for unaligned reporting year/q against header year/q](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2892) +* [#2909 \[Research Spike\] OOtB OFA Kibana Experience & DIGIT Data Access](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2909) +* [#2991 As tech lead, I need the sftp file transfer feature to be deprecated](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2991) +* [#2847 \[Design Deliverable\] Error Report Knowledge Center Explainer](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2847) +* [#3024 2897 follow-on for a11y-related enhancement ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3024) +* [#2897 As a data analyst I want finalized language and guidance resources in Submission History & Error Reports ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2897) + +### Submitted (QASP Review, OCIO Review) + +* [#2133 \[Dev\] Enhancement for Request Access form (Tribe discoverability) ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2133) +* [#3023 as STT approved user, I need my IP address whitelisted so i can access TDP](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3023) +* [#3000 \[Design Deliverable\] TDP Poster for summer 2024 conferences](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3000) +* [#2795 As tech lead, I need TDP to detect duplicate records within a file and not store them in the db. ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2795) +* [#2693 \[Error Audit\] Category 2 error messages clean-up ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2693) +* [#2801 Friendly name cleanup ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2801) +* [#2883 Pre-Made Reporting Dashboards on Kibana](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2883) +* [#3021 \[Design Deliverable\] Updated KC Release Notes & Update Indicator FAQ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3021) +* [#2954 Extend SESSION\_COOKIE\_AGE](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2954) + +### Ready to Merge + +* + +### Closed (Not Merged) + +* [#2491 Create root-level docker-compose configuration file(s)](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2491) +* [#1690 As a system admin, I need a way to be redirected to frontend from DAC](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1690) +* [#2351 As a user I want to be notified when the files are being scanned or uploaded when I push upload button](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2351) +* [#2591 Allow `manage.py` commands to be run by circleci](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2591) + +### Moved to Next Sprint + +**In Progress** + +* [#3004 Implement (small) data lifecycle (backup/archive ES)](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3004) +* [#831 \[Spike\] As a Tech Lead, I want to get alerts when there is a backend or frontend error that affects an STT user ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/831) +* [#2978 As sys admin, I want to be able to reparse datafile sets](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2978) + +#### Blocked + +* + +**Raft Review** + +* [#2950 As tech lead, I need the STT filter for search\_indexes to be updated ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2950) +* [#3008 As a software engineer, I want to be able to test django-admin-508](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3008) +* [#3016 Spike - Cat2 Validator Improvement](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3016) +* [#2473 As a data analyst I want to be notified of approaching data deadlines](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2473) diff --git a/tdrs-backend/tdpservice/data_files/test/test_api.py b/tdrs-backend/tdpservice/data_files/test/test_api.py index 9ae1a408ef..55dba626e4 100644 --- a/tdrs-backend/tdpservice/data_files/test/test_api.py +++ b/tdrs-backend/tdpservice/data_files/test/test_api.py @@ -100,10 +100,10 @@ def assert_error_report_tanf_file_content_matches_with_friendly_names(response): assert ws.cell(row=1, column=1).value == "Please refer to the most recent versions of the coding " \ + "instructions (linked below) when looking up items and allowable values during the data revision process" - assert ws.cell(row=8, column=COL_ERROR_MESSAGE).value == ("if Cash Amount :873 validator1 passed then Cash and " - "Cash Equivalents: Number of Months T1 Item -1 (Cash " - "and Cash Equivalents: Number of Months): 0 is not " - "larger than 0.") + assert ws.cell(row=8, column=COL_ERROR_MESSAGE).value == ( + "if Cash Amount :873 validator1 passed then Item 21B " + "(Cash and Cash Equivalents: Number of Months) 0 is not larger than 0." + ) @staticmethod def assert_error_report_ssp_file_content_matches_with_friendly_names(response): @@ -134,9 +134,10 @@ def assert_error_report_file_content_matches_without_friendly_names(response): assert ws.cell(row=1, column=1).value == "Please refer to the most recent versions of the coding " \ + "instructions (linked below) when looking up items and allowable values during the data revision process" - assert ws.cell(row=8, column=COL_ERROR_MESSAGE).value == ("if CASH_AMOUNT :873 validator1 passed then " - "NBR_MONTHS T1 Item -1 (NBR_MONTHS): 0 is not " - "larger than 0.") + assert ws.cell(row=8, column=COL_ERROR_MESSAGE).value == ( + "if CASH_AMOUNT :873 validator1 passed then Item 21B " + "(Cash and Cash Equivalents: Number of Months) 0 is not larger than 0." + ) @staticmethod def assert_data_file_exists(data_file_data, version, user): diff --git a/tdrs-backend/tdpservice/parsers/row_schema.py b/tdrs-backend/tdpservice/parsers/row_schema.py index a1f223ead6..7dd01556fe 100644 --- a/tdrs-backend/tdpservice/parsers/row_schema.py +++ b/tdrs-backend/tdpservice/parsers/row_schema.py @@ -1,7 +1,7 @@ """Row schema for datafile.""" from .models import ParserErrorCategoryChoices from .fields import Field, TransformField -from .validators import value_is_empty, format_error_context +from .validators import value_is_empty, format_error_context, ValidationErrorArgs import logging logger = logging.getLogger(__name__) @@ -154,12 +154,20 @@ def run_field_validators(self, instance, generate_error): ) elif field.required: is_valid = False + eargs = ValidationErrorArgs( + value=value, + row_schema=self, + friendly_name=field.friendly_name, + item_num=field.item, + error_context_format='prefix' + ) + errors.append( generate_error( schema=self, error_category=ParserErrorCategoryChoices.FIELD_VALUE, error_message=( - f"{format_error_context(self, field.friendly_name, field.item)}: " + f"{format_error_context(eargs)} " "field is required but a value was not provided." ), record=instance, diff --git a/tdrs-backend/tdpservice/parsers/test/test_parse.py b/tdrs-backend/tdpservice/parsers/test/test_parse.py index 62cdf0f0be..43a752a2b6 100644 --- a/tdrs-backend/tdpservice/parsers/test/test_parse.py +++ b/tdrs-backend/tdpservice/parsers/test/test_parse.py @@ -1673,10 +1673,12 @@ def test_parse_m2_cat2_invalid_37_38_39_file(m2_cat2_invalid_37_38_39_file, dfs) assert parser_errors.count() == 3 - error_msgs = {"M2 Item 37 (Educational Level): 00 is not in range [1, 16]. or M2 Item 37 (Educational Level): " + - "00 is not in range [98, 99].", - "M2 Item 38 (Citizenship/Immigration Status): 0 is not in [1, 2, 3, 9].", - "M2 Item 39 (Cooperated with Child Support): 0 is not in [1, 2, 9]."} + error_msgs = { + "Item 37 (Educational Level) 00 is not in range [1, 16]. or " + "Item 37 (Educational Level) 00 is not in range [98, 99].", + "M2 Item 38 (Citizenship/Immigration Status): 0 is not in [1, 2, 3, 9].", + "M2 Item 39 (Cooperated with Child Support): 0 is not in [1, 2, 9]." + } for e in parser_errors: assert e.error_message in error_msgs @@ -1697,10 +1699,10 @@ def test_parse_m3_cat2_invalid_68_69_file(m3_cat2_invalid_68_69_file, dfs): assert parser_errors.count() == 4 - error_msgs = {"M3 Item 68 (Educational Level): 00 is not in range [1, 16]. or M3 Item 68 (Educational Level): " + + error_msgs = {"Item 68 (Educational Level) 00 is not in range [1, 16]. or Item 68 (Educational Level) " + "00 is not in range [98, 99].", "M3 Item 69 (Citizenship/Immigration Status): 0 is not in [1, 2, 3, 9].", - "M3 Item 68 (Educational Level): 00 is not in range [1, 16]. or M3 Item 68 (Educational Level): " + + "Item 68 (Educational Level) 00 is not in range [1, 16]. or Item 68 (Educational Level) " + "00 is not in range [98, 99].", "M3 Item 69 (Citizenship/Immigration Status): 0 is not in [1, 2, 3, 9]."} diff --git a/tdrs-backend/tdpservice/parsers/test/test_validators.py b/tdrs-backend/tdpservice/parsers/test/test_validators.py index 521abaaf5b..d729efc6e0 100644 --- a/tdrs-backend/tdpservice/parsers/test/test_validators.py +++ b/tdrs-backend/tdpservice/parsers/test/test_validators.py @@ -5,6 +5,7 @@ from datetime import date from .. import validators from ..row_schema import RowSchema +from ..fields import Field from tdpservice.parsers.test.factories import TanfT1Factory, TanfT2Factory, TanfT3Factory, TanfT5Factory, TanfT6Factory from tdpservice.parsers.test.factories import SSPM5Factory @@ -44,39 +45,39 @@ def test_or_validators(): """Test `or_validators` gives a valid result.""" value = "2" validator = validators.or_validators(validators.matches(("2")), validators.matches(("3"))) - assert validator(value, RowSchema(), "friendly_name", "item_no") == (True, None) - assert validator("3", RowSchema(), "friendly_name", "item_no") == (True, None) - assert validator("5", RowSchema(), "friendly_name", "item_no") == ( + assert validator(value, RowSchema(), "friendly_name", "item_no", 'inline') == (True, None) + assert validator("3", RowSchema(), "friendly_name", "item_no", 'inline') == (True, None) + assert validator("5", RowSchema(), "friendly_name", "item_no", 'inline') == ( False, - "T1 Item item_no (friendly_name): 5 does not match 2. or " - "T1 Item item_no (friendly_name): 5 does not match 3." + "Item item_no (friendly_name) 5 does not match 2. or " + "Item item_no (friendly_name) 5 does not match 3." ) validator = validators.or_validators(validators.matches(("2")), validators.matches(("3")), validators.matches(("4"))) - assert validator(value, RowSchema(), "friendly_name", "item_no") == (True, None) + assert validator(value, RowSchema(), "friendly_name", "item_no", 'inline') == (True, None) value = "3" - assert validator(value, RowSchema(), "friendly_name", "item_no") == (True, None) + assert validator(value, RowSchema(), "friendly_name", "item_no", 'inline') == (True, None) value = "4" - assert validator(value, RowSchema(), "friendly_name", "item_no") == (True, None) + assert validator(value, RowSchema(), "friendly_name", "item_no", 'inline') == (True, None) value = "5" - assert validator(value, RowSchema(), "friendly_name", "item_no") == ( + assert validator(value, RowSchema(), "friendly_name", "item_no", 'inline') == ( False, - 'T1 Item item_no (friendly_name): 5 does not match 2. or ' - 'T1 Item item_no (friendly_name): 5 does not match 3. or ' - 'T1 Item item_no (friendly_name): 5 does not match 4.' + "Item item_no (friendly_name) 5 does not match 2. or " + "Item item_no (friendly_name) 5 does not match 3. or " + "Item item_no (friendly_name) 5 does not match 4." ) validator = validators.or_validators(validators.matches((2)), validators.matches((3)), validators.isLargerThan(4)) - assert validator(5, RowSchema(), "friendly_name", "item_no") == (True, None) - assert validator(1, RowSchema(), "friendly_name", "item_no") == ( + assert validator(5, RowSchema(), "friendly_name", "item_no", 'inline') == (True, None) + assert validator(1, RowSchema(), "friendly_name", "item_no", 'inline') == ( False, - "T1 Item item_no (friendly_name): 1 does not match 2. " - "or T1 Item item_no (friendly_name): 1 does not " - "match 3. or T1 Item item_no (friendly_name): 1 is not larger than 4." + "Item item_no (friendly_name) 1 does not match 2. or " + "Item item_no (friendly_name) 1 does not match 3. or " + "Item item_no (friendly_name) 1 is not larger than 4." ) def test_if_validators(): @@ -86,14 +87,28 @@ def test_if_validators(): condition_field_name="Field1", condition_function=validators.matches('1'), result_field_name="Field2", result_function=validators.matches('2'), ) - assert validator(value, RowSchema()) == (True, None, ['Field1', 'Field2']) + assert validator(value, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='Field1', friendly_name='field 1'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='Field2', friendly_name='field 2'), + ] + )) == (True, None, ['Field1', 'Field2']) validator = validator = validators.if_then_validator( condition_field_name="Field1", condition_function=validators.matches('1'), result_field_name="Field2", result_function=validators.matches('1'), ) - result = validator(value, RowSchema()) - assert result == (False, 'if Field1 :1 validator1 passed then Field2 T1 Item -1 (Field2): 2 does not match 1.', + result = validator(value, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='Field1', friendly_name='field 1'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='Field2', friendly_name='field 2'), + ] + )) + assert result == (False, 'if Field1 :1 validator1 passed then Item 2 (field 2) 2 does not match 1.', ['Field1', 'Field2']) @@ -102,7 +117,7 @@ def test_and_validators(): validator = validators.and_validators(validators.isLargerThan(2), validators.isLargerThan(0)) assert validator(1, RowSchema(), "friendly_name", "item_no") == ( False, - 'T1 Item item_no (friendly_name): 1 is not larger than 2.' + 'Item item_no (friendly_name) 1 is not larger than 2.' ) assert validator(3, RowSchema(), "friendly_name", "item_no") == (True, None) @@ -138,8 +153,9 @@ def test_validate__FAM_AFF__SSN(): def test_quarterIsValid(value, valid): """Test `quarterIsValid`.""" val = validators.quarterIsValid() - result = val(value, RowSchema(), "friendly_name", "item_no") + result = val(value, RowSchema(), "friendly_name", "item_no", None) + errorText = None if valid else f"T1 Item item_no (friendly_name): {value[-1:]} is not a valid quarter." errorText = None if valid else f"T1 Item item_no (friendly_name): {value[-1:]} is not a valid quarter." assert result == (valid, errorText) @@ -152,7 +168,7 @@ def test_validateSSN(): value = "111111111" options = [str(i) * 9 for i in range(0, 10)] - result = val(value, RowSchema(), "friendly_name", "item_no") + result = val(value, RowSchema(), "friendly_name", "item_no", None) assert result == (False, f"T1 Item item_no (friendly_name): {value} is in {options}.") def test_validateRace(): @@ -163,7 +179,7 @@ def test_validateRace(): assert result == (True, None) value = 3 - result = val(value, RowSchema(), "friendly_name", "item_no") + result = val(value, RowSchema(), "friendly_name", "item_no", None) assert result == ( False, f"T1 Item item_no (friendly_name): {value} is not greater than or equal to 0 or smaller than or equal to 2." @@ -177,7 +193,7 @@ def test_validateRptMonthYear(): assert result == (True, None) value = "T1 " - result = val(value, RowSchema(), "friendly_name", "item_no") + result = val(value, RowSchema(), "friendly_name", "item_no", None) assert result == ( False, f"T1 Item item_no (friendly_name): The value: {value[2:8]}, does not " @@ -185,7 +201,7 @@ def test_validateRptMonthYear(): ) value = "T1189912" - result = val(value, RowSchema(), "friendly_name", "item_no") + result = val(value, RowSchema(), "friendly_name", "item_no", None) assert result == ( False, f"T1 Item item_no (friendly_name): The value: {value[2:8]}, does not follow " @@ -193,7 +209,7 @@ def test_validateRptMonthYear(): ) value = "T1202013" - result = val(value, RowSchema(), "friendly_name", "item_no") + result = val(value, RowSchema(), "friendly_name", "item_no", None) assert result == ( False, f"T1 Item item_no (friendly_name): The value: {value[2:8]}, does " @@ -216,10 +232,11 @@ def test_matches_returns_invalid(): value = 'TEST' validator = validators.matches('test') - is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no") + is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no", None) assert is_valid is False assert error == 'T1 Item item_no (friendly_name): TEST does not match test.' + assert error == 'T1 Item item_no (friendly_name): TEST does not match test.' def test_oneOf_returns_valid(): @@ -237,7 +254,7 @@ def test_oneOf_returns_valid(): options = ["17-55"] validator = validators.oneOf(options) - is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no") + is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no", None) assert is_valid is True assert error is None @@ -249,16 +266,17 @@ def test_oneOf_returns_invalid(): options = [17, 24, 36] validator = validators.oneOf(options) - is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no") + is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no", None) assert is_valid is False assert error == 'T1 Item item_no (friendly_name): 64 is not in [17, 24, 36].' + assert error == 'T1 Item item_no (friendly_name): 64 is not in [17, 24, 36].' value = 65 options = ["17-55"] validator = validators.oneOf(options) - is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no") + is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no", None) assert is_valid is False assert error == 'T1 Item item_no (friendly_name): 65 is not in [17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, ' \ @@ -292,7 +310,7 @@ def test_between_returns_invalid(): value = 47 validator = validators.between(48, 400) - is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no") + is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no", None) assert is_valid is False assert error == 'T1 Item item_no (friendly_name): 47 is not between 48 and 400.' @@ -320,7 +338,7 @@ def test_between_returns_invalid(): def test_isNumber(value, expected_is_valid, expected_error): """Test `isNumber` validator.""" validator = validators.isNumber() - is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no") + is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no", None) assert is_valid == expected_is_valid assert error == expected_error @@ -338,9 +356,10 @@ def test_date_month_is_valid_returns_invalid(): """Test `dateMonthIsValid` gives an invalid result.""" value = '20191327' validator = validators.dateMonthIsValid() - is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no") + is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no", None) assert is_valid is False assert error == 'T1 Item item_no (friendly_name): 13 is not a valid month.' + assert error == 'T1 Item item_no (friendly_name): 13 is not a valid month.' def test_date_day_is_valid_returns_valid(): @@ -356,9 +375,10 @@ def test_date_day_is_valid_returns_invalid(): """Test `dateDayIsValid` gives an invalid result.""" value = '20191132' validator = validators.dateDayIsValid() - is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no") + is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no", None) assert is_valid is False assert error == 'T1 Item item_no (friendly_name): 32 is not a valid day.' + assert error == 'T1 Item item_no (friendly_name): 32 is not a valid day.' def test_olderThan(): @@ -369,7 +389,7 @@ def test_olderThan(): assert validator(value) == (True, None) value = 20240101 - result = validator(value, RowSchema(), "friendly_name", "item_no") + result = validator(value, RowSchema(), "friendly_name", "item_no", None) assert result == ( False, f"T1 Item item_no (friendly_name): {str(value)[:4]} must be less than or equal to " @@ -385,7 +405,7 @@ def test_dateYearIsLargerThan(): assert validator(value) == (True, None) value = 18990101 - assert validator(value, RowSchema(), "friendly_name", "item_no") == ( + assert validator(value, RowSchema(), "friendly_name", "item_no", None) == ( False, f"T1 Item item_no (friendly_name): Year {str(value)[:4]} must be larger than {year}." ) @@ -396,10 +416,11 @@ def test_between_returns_invalid_for_string_value(): value = '047' validator = validators.between(100, 400) - is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no") + is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no", None) assert is_valid is False assert error == 'T1 Item item_no (friendly_name): 047 is not between 100 and 400.' + assert error == 'T1 Item item_no (friendly_name): 047 is not between 100 and 400.' def test_recordHasLength_returns_valid(): @@ -418,17 +439,18 @@ def test_recordHasLength_returns_invalid(): value = 'abcd123' validator = validators.recordHasLength(22) - is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no") + is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no", None) assert is_valid is False assert error == 'T1: record length is 7 characters but must be 22.' + assert error == 'T1: record length is 7 characters but must be 22.' def test_hasLengthGreaterThan_returns_valid(): """Test `hasLengthGreaterThan` gives a valid result.""" value = 'abcd123' validator = validators.hasLengthGreaterThan(6) - is_valid, error = validator(value, None, "friendly_name", "item_no") + is_valid, error = validator(value, None, "friendly_name", "item_no", None) assert is_valid is True assert error is None @@ -451,7 +473,7 @@ def test_recordHasLengthBetween_returns_valid(): upper = 15 validator = validators.recordHasLengthBetween(lower, upper) - is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no") + is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no", None) assert is_valid is True assert error is None @@ -464,10 +486,11 @@ def test_recordHasLengthBetween_returns_invalid(): upper = 1 validator = validators.recordHasLengthBetween(lower, upper) - is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no") + is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no", None) assert is_valid is False assert error == f"T1: record length of {len(value)} characters is not in the range [{lower}, {upper}]." + assert error == f"T1: record length of {len(value)} characters is not in the range [{lower}, {upper}]." def test_intHasLength_returns_valid(): @@ -486,10 +509,11 @@ def test_intHasLength_returns_invalid(): value = '1a3' validator = validators.intHasLength(22) - is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no") + is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no", None) assert is_valid is False assert error == 'T1 Item item_no (friendly_name): 1a3 does not have exactly 22 digits.' + assert error == 'T1 Item item_no (friendly_name): 1a3 does not have exactly 22 digits.' def test_contains_returns_valid(): @@ -508,10 +532,11 @@ def test_contains_returns_invalid(): value = '12345abcde' validator = validators.contains('6789') - is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no") + is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no", None) assert is_valid is False assert error == 'T1 Item item_no (friendly_name): 12345abcde does not contain 6789.' + assert error == 'T1 Item item_no (friendly_name): 12345abcde does not contain 6789.' def test_startsWith_returns_valid(): @@ -530,10 +555,11 @@ def test_startsWith_returns_invalid(): value = '12345abcde' validator = validators.startsWith('abc') - is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no") + is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no", None) assert is_valid is False assert error == 'T1 Item item_no (friendly_name): 12345abcde does not start with abc.' + assert error == 'T1 Item item_no (friendly_name): 12345abcde does not start with abc.' def test_notEmpty_returns_valid_full_string(): @@ -552,10 +578,11 @@ def test_notEmpty_returns_invalid_full_string(): value = ' ' validator = validators.notEmpty() - is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no") + is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no", None) assert is_valid is False assert error == 'T1 Item item_no (friendly_name): contains blanks between positions 0 and 9.' + assert error == 'T1 Item item_no (friendly_name): contains blanks between positions 0 and 9.' def test_notEmpty_returns_valid_substring(): @@ -574,10 +601,11 @@ def test_notEmpty_returns_invalid_substring(): value = '111 333' validator = validators.notEmpty(start=3, end=5) - is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no") + is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no", None) assert is_valid is False assert error == "T1 Item item_no (friendly_name): 111 333 contains blanks between positions 3 and 5." + assert error == "T1 Item item_no (friendly_name): 111 333 contains blanks between positions 3 and 5." def test_notEmpty_returns_nonexistent_substring(): @@ -585,17 +613,18 @@ def test_notEmpty_returns_nonexistent_substring(): value = '111 333' validator = validators.notEmpty(start=10, end=12) - is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no") + is_valid, error = validator(value, RowSchema(), "friendly_name", "item_no", None) assert is_valid is False assert error == "T1 Item item_no (friendly_name): 111 333 contains blanks between positions 10 and 12." + assert error == "T1 Item item_no (friendly_name): 111 333 contains blanks between positions 10 and 12." @pytest.mark.parametrize("test_input", [1, 2, 3, 4]) def test_quarterIsValid_returns_true_if_valid(test_input): """Test `quarterIsValid` gives a valid result for values 1-4.""" validator = validators.quarterIsValid() - is_valid, error = validator(test_input, RowSchema(), "friendly_name", "item_no") + is_valid, error = validator(test_input, RowSchema(), "friendly_name", "item_no", None) assert is_valid is True assert error is None @@ -605,16 +634,17 @@ def test_quarterIsValid_returns_true_if_valid(test_input): def test_quarterIsValid_returns_false_if_invalid(test_input): """Test `quarterIsValid` gives an invalid result for values not 1-4.""" validator = validators.quarterIsValid() - is_valid, error = validator(test_input, RowSchema(), "friendly_name", "item_no") + is_valid, error = validator(test_input, RowSchema(), "friendly_name", "item_no", 'prefix') assert is_valid is False assert error == f"T1 Item item_no (friendly_name): {test_input} is not a valid quarter." + assert error == f"T1 Item item_no (friendly_name): {test_input} is not a valid quarter." @pytest.mark.parametrize("value", ["T72020 ", "T720194", "T720200", "T720207", "T72020$"]) def test_calendarQuarterIsValid_returns_invalid(value): """Test `calendarQuarterIsValid` returns false on invalid input.""" val = validators.calendarQuarterIsValid(2, 7) - is_valid, error_msg = val(value, RowSchema(), "friendly_name", "item_no") + is_valid, error_msg = val(value, RowSchema(), "friendly_name", "item_no", None) assert is_valid is False assert error_msg == ( @@ -627,7 +657,7 @@ def test_calendarQuarterIsValid_returns_invalid(value): def test_calendarQuarterIsValid_returns_valid(value): """Test `calendarQuarterIsValid` returns false on invalid input.""" val = validators.calendarQuarterIsValid(2, 7) - is_valid, error_msg = val(value, RowSchema(), "friendly_name", "item_no") + is_valid, error_msg = val(value, RowSchema(), "friendly_name", "item_no", None) assert is_valid is True assert error_msg is None @@ -661,11 +691,25 @@ def test_validate_food_stamps(self, record): ) record.RECEIVES_FOOD_STAMPS = 1 record.AMT_FOOD_STAMP_ASSISTANCE = 1 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='RECEIVES_FOOD_STAMPS', friendly_name='receives food stamps'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='AMT_FOOD_STAMP_ASSISTANCE', friendly_name='amt food stamps'), + ] + )) assert result == (True, None, ['RECEIVES_FOOD_STAMPS', 'AMT_FOOD_STAMP_ASSISTANCE']) record.AMT_FOOD_STAMP_ASSISTANCE = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='RECEIVES_FOOD_STAMPS', friendly_name='receives food stamps'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='AMT_FOOD_STAMP_ASSISTANCE', friendly_name='amt food stamps'), + ] + )) assert result[0] is False def test_validate_subsidized_child_care(self, record): @@ -676,12 +720,26 @@ def test_validate_subsidized_child_care(self, record): ) record.RECEIVES_SUB_CC = 4 record.AMT_SUB_CC = 1 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='RECEIVES_SUB_CC', friendly_name='receives sub cc'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='AMT_SUB_CC', friendly_name='amt sub cc'), + ] + )) assert result == (True, None, ['RECEIVES_SUB_CC', 'AMT_SUB_CC']) record.RECEIVES_SUB_CC = 4 record.AMT_SUB_CC = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='RECEIVES_SUB_CC', friendly_name='receives sub cc'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='AMT_SUB_CC', friendly_name='amt sub cc'), + ] + )) assert result[0] is False def test_validate_cash_amount_and_nbr_months(self, record): @@ -690,12 +748,26 @@ def test_validate_cash_amount_and_nbr_months(self, record): condition_field_name='CASH_AMOUNT', condition_function=validators.isLargerThan(0), result_field_name='NBR_MONTHS', result_function=validators.isLargerThan(0), ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='CASH_AMOUNT', friendly_name='cash amt'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='NBR_MONTHS', friendly_name='nbr months'), + ] + )) assert result == (True, None, ['CASH_AMOUNT', 'NBR_MONTHS']) record.CASH_AMOUNT = 1 record.NBR_MONTHS = -1 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='CASH_AMOUNT', friendly_name='cash amt'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='NBR_MONTHS', friendly_name='nbr months'), + ] + )) assert result[0] is False def test_validate_child_care(self, record): @@ -704,12 +776,26 @@ def test_validate_child_care(self, record): condition_field_name='CC_AMOUNT', condition_function=validators.isLargerThan(0), result_field_name='CHILDREN_COVERED', result_function=validators.isLargerThan(0), ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='CC_AMOUNT', friendly_name='cc amt'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='CHILDREN_COVERED', friendly_name='chldrn coverd'), + ] + )) assert result == (True, None, ['CC_AMOUNT', 'CHILDREN_COVERED']) record.CC_AMOUNT = 1 record.CHILDREN_COVERED = -1 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='CC_AMOUNT', friendly_name='cc amt'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='CHILDREN_COVERED', friendly_name='chldrn coverd'), + ] + )) assert result[0] is False val = validators.if_then_validator( @@ -718,7 +804,14 @@ def test_validate_child_care(self, record): ) record.CC_AMOUNT = 10 record.CC_NBR_MONTHS = -1 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='CC_AMOUNT', friendly_name='cc amt'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='CC_NBR_MONTHS', friendly_name='cc nbr mnths'), + ] + )) assert result[0] is False def test_validate_transportation(self, record): @@ -727,12 +820,26 @@ def test_validate_transportation(self, record): condition_field_name='TRANSP_AMOUNT', condition_function=validators.isLargerThan(0), result_field_name='TRANSP_NBR_MONTHS', result_function=validators.isLargerThan(0), ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='TRANSP_AMOUNT', friendly_name='transp amt'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='TRANSP_NBR_MONTHS', friendly_name='transp nbr months'), + ] + )) assert result == (True, None, ['TRANSP_AMOUNT', 'TRANSP_NBR_MONTHS']) record.TRANSP_AMOUNT = 1 record.TRANSP_NBR_MONTHS = -1 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='TRANSP_AMOUNT', friendly_name='transp amt'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='TRANSP_NBR_MONTHS', friendly_name='transp nbr months'), + ] + )) assert result[0] is False def test_validate_transitional_services(self, record): @@ -741,12 +848,26 @@ def test_validate_transitional_services(self, record): condition_field_name='TRANSITION_SERVICES_AMOUNT', condition_function=validators.isLargerThan(0), result_field_name='TRANSITION_NBR_MONTHS', result_function=validators.isLargerThan(0), ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='TRANSITION_SERVICES_AMOUNT', friendly_name='transition serv amt'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='TRANSITION_NBR_MONTHS', friendly_name='transition nbr months'), + ] + )) assert result == (True, None, ['TRANSITION_SERVICES_AMOUNT', 'TRANSITION_NBR_MONTHS']) record.TRANSITION_SERVICES_AMOUNT = 1 record.TRANSITION_NBR_MONTHS = -1 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='TRANSITION_SERVICES_AMOUNT', friendly_name='transition serv amt'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='TRANSITION_NBR_MONTHS', friendly_name='transition nbr months'), + ] + )) assert result[0] is False def test_validate_other(self, record): @@ -755,12 +876,26 @@ def test_validate_other(self, record): condition_field_name='OTHER_AMOUNT', condition_function=validators.isLargerThan(0), result_field_name='OTHER_NBR_MONTHS', result_function=validators.isLargerThan(0), ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='OTHER_AMOUNT', friendly_name='other amt'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='OTHER_NBR_MONTHS', friendly_name='other nbr months'), + ] + )) assert result == (True, None, ['OTHER_AMOUNT', 'OTHER_NBR_MONTHS']) record.OTHER_AMOUNT = 1 record.OTHER_NBR_MONTHS = -1 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='OTHER_AMOUNT', friendly_name='other amt'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='OTHER_NBR_MONTHS', friendly_name='other nbr months'), + ] + )) assert result[0] is False def test_validate_reasons_for_amount_of_assistance_reductions(self, record): @@ -770,12 +905,26 @@ def test_validate_reasons_for_amount_of_assistance_reductions(self, record): result_field_name='WORK_REQ_SANCTION', result_function=validators.oneOf((1, 2)), ) record.SANC_REDUCTION_AMT = 1 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='SANC_REDUCTION_AMT', friendly_name='sanc reduction amt'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='WORK_REQ_SANCTION', friendly_name='work req sanction'), + ] + )) assert result == (True, None, ['SANC_REDUCTION_AMT', 'WORK_REQ_SANCTION']) record.SANC_REDUCTION_AMT = 10 record.WORK_REQ_SANCTION = -1 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='SANC_REDUCTION_AMT', friendly_name='sanc reduction amt'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='WORK_REQ_SANCTION', friendly_name='work req sanction'), + ] + )) assert result[0] is False def test_validate_sum(self, record): @@ -792,7 +941,22 @@ def test_validate_sum(self, record): record.TRANSP_AMOUNT = 0 record.TRANSITION_SERVICES_AMOUNT = 0 record.OTHER_AMOUNT = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='AMT_FOOD_STAMP_ASSISTANCE', friendly_name='amt food stamp assis'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='AMT_SUB_CC', friendly_name='amt sub cc'), + Field(item=3, startIndex=4, endIndex=5, type='string', + name='CC_AMOUNT', friendly_name='cc amt'), + Field(item=4, startIndex=5, endIndex=6, type='string', + name='TRANSP_AMOUNT', friendly_name='transp amt'), + Field(item=5, startIndex=6, endIndex=7, type='string', + name='TRANSITION_SERVICES_AMOUNT', friendly_name='transition serv amt'), + Field(item=6, startIndex=7, endIndex=8, type='string', + name='OTHER_AMOUNT', friendly_name='other amt'), + ] + )) assert result[0] is False @@ -815,12 +979,26 @@ def test_validate_ssn(self, record): ) record.SSN = "999989999" record.FAMILY_AFFILIATION = 1 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='SSN', friendly_name='ssn'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'SSN']) record.FAMILY_AFFILIATION = 1 record.SSN = "999999999" - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='SSN', friendly_name='ssn'), + ] + )) assert result[0] is False def test_validate_race_ethnicity(self, record): @@ -832,7 +1010,14 @@ def test_validate_race_ethnicity(self, record): condition_field_name='FAMILY_AFFILIATION', condition_function=validators.oneOf((1, 2, 3)), result_field_name=race, result_function=validators.isInLimits(1, 2), ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name=race, friendly_name='race'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', race]) record.FAMILY_AFFILIATION = 0 @@ -841,7 +1026,14 @@ def test_validate_race_ethnicity(self, record): condition_field_name='FAMILY_AFFILIATION', condition_function=validators.oneOf((1, 2, 3)), result_field_name=race, result_function=validators.isInLimits(1, 2) ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name=race, friendly_name='race'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', race]) def test_validate_marital_status(self, record): @@ -851,12 +1043,26 @@ def test_validate_marital_status(self, record): result_field_name='MARITAL_STATUS', result_function=validators.isInLimits(1, 5), ) record.FAMILY_AFFILIATION = 1 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='MARITAL_STATUS', friendly_name='married?'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'MARITAL_STATUS']) record.FAMILY_AFFILIATION = 3 record.MARITAL_STATUS = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='MARITAL_STATUS', friendly_name='married?'), + ] + )) assert result[0] is False def test_validate_parent_with_minor(self, record): @@ -865,11 +1071,25 @@ def test_validate_parent_with_minor(self, record): condition_field_name='FAMILY_AFFILIATION', condition_function=validators.isInLimits(1, 3), result_field_name='PARENT_MINOR_CHILD', result_function=validators.isInLimits(1, 3), ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='PARENT_MINOR_CHILD', friendly_name='parent minor child'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'PARENT_MINOR_CHILD']) record.PARENT_MINOR_CHILD = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='PARENT_MINOR_CHILD', friendly_name='parent minor child'), + ] + )) assert result[0] is False def test_validate_education_level(self, record): @@ -883,12 +1103,26 @@ def test_validate_education_level(self, record): "98", "99")), ) record.FAMILY_AFFILIATION = 3 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='EDUCATION_LEVEL', friendly_name='education level'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'EDUCATION_LEVEL']) record.FAMILY_AFFILIATION = 1 record.EDUCATION_LEVEL = "00" - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='EDUCATION_LEVEL', friendly_name='education level'), + ] + )) assert result[0] is False def test_validate_citizenship(self, record): @@ -898,12 +1132,26 @@ def test_validate_citizenship(self, record): result_field_name='CITIZENSHIP_STATUS', result_function=validators.oneOf((1, 2)), ) record.FAMILY_AFFILIATION = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='CITIZENSHIP_STATUS', friendly_name='citizenship status'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'CITIZENSHIP_STATUS']) record.FAMILY_AFFILIATION = 1 record.CITIZENSHIP_STATUS = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='CITIZENSHIP_STATUS', friendly_name='citizenship status'), + ] + )) assert result[0] is False def test_validate_cooperation_with_child_support(self, record): @@ -913,12 +1161,26 @@ def test_validate_cooperation_with_child_support(self, record): result_field_name='COOPERATION_CHILD_SUPPORT', result_function=validators.oneOf((1, 2, 9)), ) record.FAMILY_AFFILIATION = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='COOPERATION_CHILD_SUPPORT', friendly_name='cooperation child support'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'COOPERATION_CHILD_SUPPORT']) record.FAMILY_AFFILIATION = 1 record.COOPERATION_CHILD_SUPPORT = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='COOPERATION_CHILD_SUPPORT', friendly_name='cooperation child support'), + ] + )) assert result[0] is False def test_validate_employment_status(self, record): @@ -928,12 +1190,26 @@ def test_validate_employment_status(self, record): result_field_name='EMPLOYMENT_STATUS', result_function=validators.isInLimits(1, 3), ) record.FAMILY_AFFILIATION = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='EMPLOYMENT_STATUS', friendly_name='employment status'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'EMPLOYMENT_STATUS']) record.FAMILY_AFFILIATION = 3 record.EMPLOYMENT_STATUS = 4 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='EMPLOYMENT_STATUS', friendly_name='employment status'), + ] + )) assert result[0] is False def test_validate_work_eligible_indicator(self, record): @@ -946,12 +1222,26 @@ def test_validate_work_eligible_indicator(self, record): ), ) record.FAMILY_AFFILIATION = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='WORK_ELIGIBLE_INDICATOR', friendly_name='work eligible indicator'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'WORK_ELIGIBLE_INDICATOR']) record.FAMILY_AFFILIATION = 1 record.WORK_ELIGIBLE_INDICATOR = "00" - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='WORK_ELIGIBLE_INDICATOR', friendly_name='work eligible indicator'), + ] + )) assert result[0] is False def test_validate_work_participation(self, record): @@ -963,12 +1253,26 @@ def test_validate_work_participation(self, record): '19', '99']), ) record.FAMILY_AFFILIATION = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='WORK_PART_STATUS', friendly_name='work part status'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'WORK_PART_STATUS']) record.FAMILY_AFFILIATION = 2 record.WORK_PART_STATUS = "04" - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='WORK_PART_STATUS', friendly_name='work part status'), + ] + )) assert result[0] is False val = validators.if_then_validator( @@ -979,7 +1283,14 @@ def test_validate_work_participation(self, record): ) record.WORK_PART_STATUS = "99" record.WORK_ELIGIBLE_INDICATOR = "01" - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='WORK_ELIGIBLE_INDICATOR', friendly_name='work eligible indicator'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='WORK_PART_STATUS', friendly_name='work part status'), + ] + )) assert result[0] is False @@ -999,12 +1310,26 @@ def test_validate_ssn(self, record): condition_field_name='FAMILY_AFFILIATION', condition_function=validators.matches(1), result_field_name='SSN', result_function=validators.notOneOf(("999999999", "000000000")), ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='SSN', friendly_name='social'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'SSN']) record.FAMILY_AFFILIATION = 1 record.SSN = "999999999" - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='SSN', friendly_name='social'), + ] + )) assert result[0] is False def test_validate_t3_race_ethnicity(self, record): @@ -1016,7 +1341,14 @@ def test_validate_t3_race_ethnicity(self, record): condition_field_name='FAMILY_AFFILIATION', condition_function=validators.oneOf((1, 2)), result_field_name=race, result_function=validators.oneOf((1, 2)), ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name=race, friendly_name='race'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', race]) record.FAMILY_AFFILIATION = 0 @@ -1025,7 +1357,14 @@ def test_validate_t3_race_ethnicity(self, record): condition_field_name='FAMILY_AFFILIATION', condition_function=validators.oneOf((1, 2)), result_field_name=race, result_function=validators.oneOf((1, 2)), ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name=race, friendly_name='race'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', race]) def test_validate_relationship_hoh(self, record): @@ -1036,12 +1375,26 @@ def test_validate_relationship_hoh(self, record): ) record.FAMILY_AFFILIATION = 0 record.RELATIONSHIP_HOH = "04" - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='RELATIONSHIP_HOH', friendly_name='relationship hoh'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'RELATIONSHIP_HOH']) record.FAMILY_AFFILIATION = 1 record.RELATIONSHIP_HOH = "01" - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='RELATIONSHIP_HOH', friendly_name='relationship hoh'), + ] + )) assert result[0] is False def test_validate_t3_education_level(self, record): @@ -1051,12 +1404,26 @@ def test_validate_t3_education_level(self, record): result_field_name='EDUCATION_LEVEL', result_function=validators.notMatches("99"), ) record.FAMILY_AFFILIATION = 1 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='EDUCATION_LEVEL', friendly_name='ed lev'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'EDUCATION_LEVEL']) record.FAMILY_AFFILIATION = 1 record.EDUCATION_LEVEL = "99" - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='EDUCATION_LEVEL', friendly_name='ed lev'), + ] + )) assert result[0] is False def test_validate_t3_citizenship(self, record): @@ -1066,12 +1433,26 @@ def test_validate_t3_citizenship(self, record): result_field_name='CITIZENSHIP_STATUS', result_function=validators.oneOf((1, 2)), ) record.FAMILY_AFFILIATION = 1 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='CITIZENSHIP_STATUS', friendly_name='cit stat'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'CITIZENSHIP_STATUS']) record.FAMILY_AFFILIATION = 1 record.CITIZENSHIP_STATUS = 3 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='CITIZENSHIP_STATUS', friendly_name='cit stat'), + ] + )) assert result[0] is False val = validators.if_then_validator( @@ -1080,7 +1461,14 @@ def test_validate_t3_citizenship(self, record): ) record.FAMILY_AFFILIATION = 2 record.CITIZENSHIP_STATUS = 3 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='CITIZENSHIP_STATUS', friendly_name='cit stat'), + ] + )) assert result[0] is False @@ -1099,26 +1487,58 @@ def test_validate_ssn(self, record): result_field_name='SSN', result_function=validators.isNumber() ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='SSN', friendly_name='social'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'SSN']) record.SSN = "abc" record.FAMILY_AFFILIATION = 2 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='SSN', friendly_name='social'), + ] + )) assert result[0] is False def test_validate_ssn_citizenship(self, record): """Test cat3 validator for SSN/citizenship.""" val = validators.validate__FAM_AFF__SSN() - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='SSN', friendly_name='social'), + Field(item=3, startIndex=4, endIndex=5, type='string', + name='CITIZENSHIP_STATUS', friendly_name='cit stat'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'CITIZENSHIP_STATUS', 'SSN']) record.FAMILY_AFFILIATION = 2 record.SSN = "000000000" - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='SSN', friendly_name='social'), + Field(item=3, startIndex=4, endIndex=5, type='string', + name='CITIZENSHIP_STATUS', friendly_name='cit stat'), + ] + )) assert result[0] is False def test_validate_race_ethnicity(self, record): @@ -1128,10 +1548,17 @@ def test_validate_race_ethnicity(self, record): for race in races: val = validators.if_then_validator( condition_field_name='FAMILY_AFFILIATION', condition_function=validators.isInLimits(1, 3), - result_field_name='RACE_HISPANIC', result_function=validators.isInLimits(1, 2) + result_field_name=race, result_function=validators.isInLimits(1, 2) ) - result = val(record, RowSchema()) - assert result == (True, None, ['FAMILY_AFFILIATION', 'RACE_HISPANIC']) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name=race, friendly_name='social'), + ] + )) + assert result == (True, None, ['FAMILY_AFFILIATION', race]) record.FAMILY_AFFILIATION = 1 record.RACE_HISPANIC = 0 @@ -1145,7 +1572,14 @@ def test_validate_race_ethnicity(self, record): condition_field_name='FAMILY_AFFILIATION', condition_function=validators.isInLimits(1, 3), result_field_name=race, result_function=validators.isInLimits(1, 2) ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name=race, friendly_name='social'), + ] + )) assert result[0] is False def test_validate_marital_status(self, record): @@ -1156,13 +1590,27 @@ def test_validate_marital_status(self, record): ) record.FAMILY_AFFILIATION = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='MARITAL_STATUS', friendly_name='marital status'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'MARITAL_STATUS']) record.FAMILY_AFFILIATION = 2 record.MARITAL_STATUS = 6 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='MARITAL_STATUS', friendly_name='marital status'), + ] + )) assert result[0] is False def test_validate_parent_minor(self, record): @@ -1173,13 +1621,27 @@ def test_validate_parent_minor(self, record): ) record.FAMILY_AFFILIATION = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='PARENT_MINOR_CHILD', friendly_name='parent minor child'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'PARENT_MINOR_CHILD']) record.FAMILY_AFFILIATION = 2 record.PARENT_MINOR_CHILD = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='PARENT_MINOR_CHILD', friendly_name='parent minor child'), + ] + )) assert result[0] is False def test_validate_education(self, record): @@ -1193,13 +1655,27 @@ def test_validate_education(self, record): ) record.FAMILY_AFFILIATION = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='EDUCATION_LEVEL', friendly_name='education level'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'EDUCATION_LEVEL']) record.FAMILY_AFFILIATION = 2 record.EDUCATION_LEVEL = "0" - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='EDUCATION_LEVEL', friendly_name='education level'), + ] + )) assert result[0] is False def test_validate_citizenship_status(self, record): @@ -1210,13 +1686,27 @@ def test_validate_citizenship_status(self, record): ) record.FAMILY_AFFILIATION = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='CITIZENSHIP_STATUS', friendly_name='citizenship status'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'CITIZENSHIP_STATUS']) record.FAMILY_AFFILIATION = 1 record.CITIZENSHIP_STATUS = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='CITIZENSHIP_STATUS', friendly_name='citizenship status'), + ] + )) assert result[0] is False def test_validate_oasdi_insurance(self, record): @@ -1227,13 +1717,27 @@ def test_validate_oasdi_insurance(self, record): ) record.DATE_OF_BIRTH = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='DATE_OF_BIRTH', friendly_name='dob'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='REC_OASDI_INSURANCE', friendly_name='rec oasdi insurance'), + ] + )) assert result == (True, None, ['DATE_OF_BIRTH', 'REC_OASDI_INSURANCE']) record.DATE_OF_BIRTH = 200001 record.REC_OASDI_INSURANCE = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='DATE_OF_BIRTH', friendly_name='dob'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='REC_OASDI_INSURANCE', friendly_name='rec oasdi insurance'), + ] + )) assert result[0] is False def test_validate_federal_disability(self, record): @@ -1244,13 +1748,27 @@ def test_validate_federal_disability(self, record): ) record.FAMILY_AFFILIATION = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='REC_FEDERAL_DISABILITY', friendly_name='rec fed disability'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'REC_FEDERAL_DISABILITY']) record.FAMILY_AFFILIATION = 1 record.REC_FEDERAL_DISABILITY = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='REC_FEDERAL_DISABILITY', friendly_name='rec fed disability'), + ] + )) assert result[0] is False @@ -1267,12 +1785,30 @@ def test_sum_of_applications(self, record): val = validators.sumIsEqual("NUM_APPLICATIONS", ["NUM_APPROVED", "NUM_DENIED"]) record.NUM_APPLICATIONS = 2 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='NUM_APPLICATIONS', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='NUM_APPROVED', friendly_name='num approved'), + Field(item=2, startIndex=4, endIndex=5, type='string', + name='NUM_DENIED', friendly_name='num denied'), + ] + )) assert result == (True, None, ['NUM_APPLICATIONS', 'NUM_APPROVED', 'NUM_DENIED']) record.NUM_APPLICATIONS = 1 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='NUM_APPLICATIONS', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='NUM_APPROVED', friendly_name='num approved'), + Field(item=3, startIndex=4, endIndex=5, type='string', + name='NUM_DENIED', friendly_name='num denied'), + ] + )) assert result[0] is False @@ -1281,12 +1817,34 @@ def test_sum_of_families(self, record): val = validators.sumIsEqual("NUM_FAMILIES", ["NUM_2_PARENTS", "NUM_1_PARENTS", "NUM_NO_PARENTS"]) record.NUM_FAMILIES = 3 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='NUM_FAMILIES', friendly_name='num fam'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='NUM_2_PARENTS', friendly_name='num 2 parent'), + Field(item=3, startIndex=4, endIndex=5, type='string', + name='NUM_1_PARENTS', friendly_name='num 2 parent'), + Field(item=4, startIndex=5, endIndex=6, type='string', + name='NUM_NO_PARENTS', friendly_name='num 0 parent'), + ] + )) assert result == (True, None, ['NUM_FAMILIES', 'NUM_2_PARENTS', 'NUM_1_PARENTS', 'NUM_NO_PARENTS']) record.NUM_FAMILIES = 1 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='NUM_FAMILIES', friendly_name='num fam'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='NUM_2_PARENTS', friendly_name='num 2 parent'), + Field(item=3, startIndex=4, endIndex=5, type='string', + name='NUM_1_PARENTS', friendly_name='num 2 parent'), + Field(item=4, startIndex=5, endIndex=6, type='string', + name='NUM_NO_PARENTS', friendly_name='num 0 parent'), + ] + )) assert result[0] is False @@ -1295,12 +1853,30 @@ def test_sum_of_recipients(self, record): val = validators.sumIsEqual("NUM_RECIPIENTS", ["NUM_ADULT_RECIPIENTS", "NUM_CHILD_RECIPIENTS"]) record.NUM_RECIPIENTS = 2 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='NUM_RECIPIENTS', friendly_name='num recip'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='NUM_ADULT_RECIPIENTS', friendly_name='num adult recip'), + Field(item=3, startIndex=4, endIndex=5, type='string', + name='NUM_CHILD_RECIPIENTS', friendly_name='num child recip'), + ] + )) assert result == (True, None, ['NUM_RECIPIENTS', 'NUM_ADULT_RECIPIENTS', 'NUM_CHILD_RECIPIENTS']) record.NUM_RECIPIENTS = 1 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='NUM_RECIPIENTS', friendly_name='num recip'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='NUM_ADULT_RECIPIENTS', friendly_name='num adult recip'), + Field(item=3, startIndex=4, endIndex=5, type='string', + name='NUM_CHILD_RECIPIENTS', friendly_name='num child recip'), + ] + )) assert result[0] is False @@ -1319,11 +1895,25 @@ def test_fam_affil_ssn(self, record): result_field_name='SSN', result_function=validators.validateSSN(), ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='SSN', friendly_name='social'), + ] + )) assert result == (True, None, ["FAMILY_AFFILIATION", "SSN"]) record.SSN = '111111111' - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='SSN', friendly_name='social'), + ] + )) assert result[0] is False @@ -1335,7 +1925,14 @@ def test_validate_race_ethnicity(self, record): condition_field_name='FAMILY_AFFILIATION', condition_function=validators.isInLimits(1, 3), result_field_name=race, result_function=validators.isInLimits(1, 2), ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name=race, friendly_name='social'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', race]) def test_fam_affil_marital_stat(self, record): @@ -1345,12 +1942,26 @@ def test_fam_affil_marital_stat(self, record): result_field_name='MARITAL_STATUS', result_function=validators.isInLimits(1, 5), ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='MARITAL_STATUS', friendly_name='marital status'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'MARITAL_STATUS']) record.MARITAL_STATUS = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='MARITAL_STATUS', friendly_name='marital status'), + ] + )) assert result[0] is False def test_fam_affil_parent_with_minor(self, record): @@ -1360,12 +1971,26 @@ def test_fam_affil_parent_with_minor(self, record): result_field_name='PARENT_MINOR_CHILD', result_function=validators.isInLimits(1, 3), ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='PARENT_MINOR_CHILD', friendly_name='parent minor child'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'PARENT_MINOR_CHILD']) record.PARENT_MINOR_CHILD = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='PARENT_MINOR_CHILD', friendly_name='parent minor child'), + ] + )) assert result[0] is False def test_fam_affil_ed_level(self, record): @@ -1376,12 +2001,26 @@ def test_fam_affil_ed_level(self, record): validators.isInStringRange(1, 16), validators.isInStringRange(98, 99)), ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='EDUCATION_LEVEL', friendly_name='education level'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'EDUCATION_LEVEL']) record.EDUCATION_LEVEL = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='EDUCATION_LEVEL', friendly_name='education level'), + ] + )) assert result[0] is False def test_fam_affil_citz_stat(self, record): @@ -1391,12 +2030,26 @@ def test_fam_affil_citz_stat(self, record): result_field_name='CITIZENSHIP_STATUS', result_function=validators.isInLimits(1, 3), ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='CITIZENSHIP_STATUS', friendly_name='citizenship status'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'CITIZENSHIP_STATUS']) record.CITIZENSHIP_STATUS = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='CITIZENSHIP_STATUS', friendly_name='citizenship status'), + ] + )) assert result[0] is False def test_dob_oasdi_insur(self, record): @@ -1406,12 +2059,26 @@ def test_dob_oasdi_insur(self, record): result_field_name='REC_OASDI_INSURANCE', result_function=validators.isInLimits(1, 2), ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='DATE_OF_BIRTH', friendly_name='dob'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='REC_OASDI_INSURANCE', friendly_name='rec oasdi insurance'), + ] + )) assert result == (True, None, ['DATE_OF_BIRTH', 'REC_OASDI_INSURANCE']) record.REC_OASDI_INSURANCE = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='DATE_OF_BIRTH', friendly_name='dob'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='REC_OASDI_INSURANCE', friendly_name='rec oasdi insurance'), + ] + )) assert result[0] is False def test_fam_affil_fed_disability(self, record): @@ -1421,11 +2088,25 @@ def test_fam_affil_fed_disability(self, record): result_field_name='REC_FEDERAL_DISABILITY', result_function=validators.isInLimits(1, 2), ) - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='REC_FEDERAL_DISABILITY', friendly_name='rec fed disability'), + ] + )) assert result == (True, None, ['FAMILY_AFFILIATION', 'REC_FEDERAL_DISABILITY']) record.REC_FEDERAL_DISABILITY = 0 - result = val(record, RowSchema()) + result = val(record, RowSchema( + fields=[ + Field(item=1, startIndex=0, endIndex=2, type='string', + name='FAMILY_AFFILIATION', friendly_name='fam affil'), + Field(item=2, startIndex=2, endIndex=4, type='string', + name='REC_FEDERAL_DISABILITY', friendly_name='rec fed disability'), + ] + )) assert result[0] is False def test_is_quiet_preparser_errors(): diff --git a/tdrs-backend/tdpservice/parsers/validators.py b/tdrs-backend/tdpservice/parsers/validators.py index 9f99e928c4..e8320055a5 100644 --- a/tdrs-backend/tdpservice/parsers/validators.py +++ b/tdrs-backend/tdpservice/parsers/validators.py @@ -1,9 +1,12 @@ """Generic parser validator functions for use in schema definitions.""" -from .models import ParserErrorCategoryChoices -from .util import fiscal_to_calendar, year_month_to_year_quarter, clean_options_string import datetime import logging +from dataclasses import dataclass +from typing import Any +# from tdpservice.parsers.row_schema import RowSchema +from tdpservice.parsers.models import ParserErrorCategoryChoices +from tdpservice.parsers.util import fiscal_to_calendar, year_month_to_year_quarter, clean_options_string logger = logging.getLogger(__name__) @@ -22,27 +25,63 @@ def value_is_empty(value, length, extra_vals={}): return value is None or value in empty_values + +@dataclass +class ValidationErrorArgs: + """Dataclass for args to `make_validator` `error_func`s.""" + + value: Any + row_schema: object # RowSchema causes circular import + friendly_name: str + item_num: str + error_context_format: str = 'prefix' + + +def format_error_context(eargs: ValidationErrorArgs): + """Format the error message for consistency across cat2 validators.""" + match eargs.error_context_format: + case 'inline': + return f'Item {eargs.item_num} ({eargs.friendly_name})' + + case 'prefix' | _: + return f'{eargs.row_schema.record_type} Item {eargs.item_num} ({eargs.friendly_name}):' + + # higher order validator functions + def make_validator(validator_func, error_func): """Return a function accepting a value input and returning (bool, string) to represent validation state.""" - def validator(value, row_schema=None, friendly_name=None, item_num=None): + def validator(value, row_schema=None, friendly_name=None, item_num=None, error_context_format='prefix'): + eargs = ValidationErrorArgs( + value=value, + row_schema=row_schema, + friendly_name=friendly_name, + item_num=item_num, + error_context_format=error_context_format + ) + try: if validator_func(value): return (True, None) - return (False, error_func(value, row_schema, friendly_name, item_num)) + return (False, error_func(eargs)) except Exception as e: logger.debug(f"Caught exception in validator. Exception: {e}") - return (False, error_func(value, row_schema, friendly_name, item_num)) + return (False, error_func(eargs)) return validator def or_validators(*args, **kwargs): """Return a validator that is true only if one of the validators is true.""" return ( - lambda value, row_schema, friendly_name, item_num: (True, None) - if any([validator(value, row_schema, friendly_name, item_num)[0] for validator in args]) - else (False, " or ".join([validator(value, row_schema, friendly_name, item_num)[1] for validator in args])) + lambda value, row_schema, friendly_name, + item_num, error_context_format='inline': (True, None) + if any([ + validator(value, row_schema, friendly_name, item_num, error_context_format)[0] for validator in args + ]) + else (False, " or ".join([ + validator(value, row_schema, friendly_name, item_num, error_context_format)[1] for validator in args + ])) ) @@ -50,14 +89,14 @@ def and_validators(validator1, validator2): """Return a validator that is true only if both validators are true.""" return ( lambda value, row_schema, friendly_name, item_num: (True, None) - if (validator1(value, row_schema, friendly_name, item_num)[0] and validator2(value, row_schema, - friendly_name, item_num)[0]) + if (validator1(value, row_schema, friendly_name, item_num, 'inline')[0] + and validator2(value, row_schema, friendly_name, item_num, 'inline')[0]) else ( False, - (validator1(value, row_schema, friendly_name, item_num)[1]) - if validator1(value, row_schema, friendly_name, item_num)[1] is not None + (validator1(value, row_schema, friendly_name, item_num, 'inline')[1]) + if validator1(value, row_schema, friendly_name, item_num, 'inline')[1] is not None else "" + " and " + validator2(value)[1] - if validator2(value, row_schema, friendly_name, item_num)[1] is not None + if validator2(value, row_schema, friendly_name, item_num, 'inline')[1] is not None else "", ) ) @@ -69,9 +108,9 @@ def or_priority_validators(validators=[]): """ def or_priority_validators_func(value, rows_schema, friendly_name=None, item_num=None): for validator in validators: - if not validator(value, rows_schema, friendly_name, item_num)[0]: + if not validator(value, rows_schema, friendly_name, item_num, 'inline')[0]: return (False, validator(value, rows_schema, - friendly_name, item_num)[1]) + friendly_name, item_num, 'inline')[1]) return (True, None) return or_priority_validators_func @@ -80,14 +119,13 @@ def or_priority_validators_func(value, rows_schema, friendly_name=None, item_num def extended_and_validators(*args, **kwargs): """Return a validator that is true only if all validators are true.""" def returned_func(value, row_schema, friendly_name, item_num): - if all([validator(value, row_schema, friendly_name, item_num)[0] for validator in args]): + if all([validator(value, row_schema, friendly_name, item_num, 'inline')[0] for validator in args]): return (True, None) else: return (False, "".join( [ - " and " + validator(value, row_schema, - friendly_name, item_num)[1] if validator(value, row_schema, - friendly_name, item_num)[0] else "" + " and " + validator(value, row_schema, friendly_name, item_num, 'inline')[1] + if validator(value, row_schema, friendly_name, item_num, 'inline')[0] else "" for validator in args ] )) @@ -115,9 +153,23 @@ def if_then_validator_func(value, row_schema): value[result_field_name] if type(value) is dict else getattr(value, result_field_name) ) - # TODO: There is some work to be done here to get the actual friendly name and item numbers of the fields - validator1_result = condition_function(value1, row_schema, condition_field_name, "-1") - validator2_result = result_function(value2, row_schema, result_field_name, "-1") + condition_field = row_schema.get_field_by_name(condition_field_name) + result_field = row_schema.get_field_by_name(result_field_name) + + validator1_result = condition_function( + value1, + row_schema, + condition_field.friendly_name, + condition_field.item, + 'inline' + ) + validator2_result = result_function( + value2, + row_schema, + result_field.friendly_name, + result_field.item, + 'inline' + ) if not validator1_result[0]: returned_value = (True, None, [condition_field_name, result_field_name]) @@ -137,7 +189,7 @@ def if_then_validator_func(value, row_schema): ending_error = "validator2 passed" error_message = (f"if {condition_field_name} " + (center_error) + - f" then {result_field_name} " + ending_error) + " then " + ending_error) else: error_message = None @@ -179,7 +231,8 @@ def sumIsEqualFunc(value, row_schema): def field_year_month_with_header_year_quarter(): """Validate that the field year and month match the header year and quarter.""" - def validate_reporting_month_year_fields_header(line, row_schema, friendly_name, item_num): + def validate_reporting_month_year_fields_header( + line, row_schema, friendly_name, item_num, error_context_format=None): field_month_year = row_schema.get_field_values_by_names(line, ['RPT_MONTH_YEAR']).get('RPT_MONTH_YEAR') df_quarter = row_schema.datafile.quarter @@ -222,10 +275,8 @@ def recordHasLength(length): """Validate that value (string or array) has a length matching length param.""" return make_validator( lambda value: len(value) == length, - lambda value, - row_schema, - friendly_name, - item_num: f"{row_schema.record_type}: record length is {len(value)} characters but must be {length}.", + lambda eargs: f"{eargs.row_schema.record_type}: record length is " + f"{len(eargs.value)} characters but must be {length}.", ) @@ -233,10 +284,11 @@ def recordHasLengthBetween(lower, upper, error_func=None): """Validate that value (string or array) has a length matching length param.""" return make_validator( lambda value: len(value) >= lower and len(value) <= upper, - lambda value, row_schema, friendly_name, item_num: error_func(value, lower, upper) + lambda eargs: error_func(eargs.value, lower, upper) if error_func else - f"{row_schema.record_type}: record length of {len(value)} characters is not in the range [{lower}, {upper}].", + f"{eargs.row_schema.record_type}: record length of {len(eargs.value)} " + f"characters is not in the range [{lower}, {upper}].", ) @@ -244,8 +296,7 @@ def caseNumberNotEmpty(start=0, end=None): """Validate that string value isn't only blanks.""" return make_validator( lambda value: not _is_empty(value, start, end), - lambda value, row_schema, - friendly_name, item_num: f'{row_schema.record_type}: Case number {str(value)} cannot contain blanks.' + lambda eargs: f'{eargs.row_schema.record_type}: Case number {str(eargs.value)} cannot contain blanks.' ) @@ -254,29 +305,21 @@ def calendarQuarterIsValid(start=0, end=None): return make_validator( lambda value: value[start:end].isnumeric() and int(value[start:end - 1]) >= 2020 and int(value[end - 1:end]) > 0 and int(value[end - 1:end]) < 5, - lambda value, - row_schema, - friendly_name, - item_num: f"{row_schema.record_type}: {value[start:end]} is invalid. Calendar Quarter must be a numeric " - "representing the Calendar Year and Quarter formatted as YYYYQ", + lambda eargs: f"{eargs.row_schema.record_type}: {eargs.value[start:end]} is invalid. " + "Calendar Quarter must be a numeric representing the Calendar Year and Quarter formatted as YYYYQ", ) # generic validators -def format_error_context(row_schema, friendly_name, item_num): - """Format the error message for consistency across cat2 validators.""" - return f'{row_schema.record_type} Item {item_num} ({friendly_name})' - - def matches(option, error_func=None): """Validate that value is equal to option.""" return make_validator( lambda value: value == option, - lambda value, row_schema, friendly_name, item_num: error_func(option) + lambda eargs: error_func(option) if error_func - else f"{format_error_context(row_schema, friendly_name, item_num)}: {value} does not match {option}.", + else f"{format_error_context(eargs)} {eargs.value} does not match {option}.", ) @@ -284,8 +327,7 @@ def notMatches(option): """Validate that value is not equal to option.""" return make_validator( lambda value: value != option, - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: {value} matches {option}." + lambda eargs: f"{format_error_context(eargs)} {eargs.value} matches {option}." ) @@ -306,9 +348,8 @@ def check_option(value, options): return make_validator( lambda value: check_option(value, options), - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: " - f"{value} is not in {clean_options_string(options)}." + lambda eargs: + f"{format_error_context(eargs)} {eargs.value} is not in {clean_options_string(options)}." ) @@ -316,9 +357,8 @@ def notOneOf(options=[]): """Validate that value exists in the provided options array.""" return make_validator( lambda value: value not in options, - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: " - f"{value} is in {clean_options_string(options)}." + lambda eargs: + f"{format_error_context(eargs)} {eargs.value} is in {clean_options_string(options)}." ) @@ -326,8 +366,8 @@ def between(min, max): """Validate value, when casted to int, is greater than min and less than max.""" return make_validator( lambda value: int(value) > min and int(value) < max, - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: {value} is not between {min} and {max}.", + lambda eargs: + f"{format_error_context(eargs)} {eargs.value} is not between {min} and {max}.", ) @@ -335,10 +375,8 @@ def fieldHasLength(length): """Validate that the field value (string or array) has a length matching length param.""" return make_validator( lambda value: len(value) == length, - lambda value, - row_schema, - friendly_name, - item_num: f"{row_schema.record_type} field length is {len(value)} characters but must be {length}.", + lambda eargs: + f"{eargs.row_schema.record_type} field length is {len(eargs.value)} characters but must be {length}.", ) @@ -346,10 +384,8 @@ def hasLengthGreaterThan(val, error_func=None): """Validate that value (string or array) has a length greater than val.""" return make_validator( lambda value: len(value) >= val, - lambda value, - row_schema, - friendly_name, - item_num: f"Value length {len(value)} is not greater than {val}.", + lambda eargs: + f"Value length {len(eargs.value)} is not greater than {val}.", ) @@ -357,9 +393,8 @@ def intHasLength(num_digits): """Validate the number of digits in an integer.""" return make_validator( lambda value: sum(c.isdigit() for c in str(value)) == num_digits, - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: " - f"{value} does not have exactly {num_digits} digits.", + lambda eargs: + f"{format_error_context(eargs)} {eargs.value} does not have exactly {num_digits} digits.", ) @@ -367,8 +402,7 @@ def contains(substring): """Validate that string value contains the given substring param.""" return make_validator( lambda value: value.find(substring) != -1, - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: {value} does not contain {substring}.", + lambda eargs: f"{format_error_context(eargs)} {eargs.value} does not contain {substring}.", ) @@ -376,9 +410,9 @@ def startsWith(substring, error_func=None): """Validate that string value starts with the given substring param.""" return make_validator( lambda value: value.startswith(substring), - lambda value, row_schema, friendly_name, item_num: error_func(substring) + lambda eargs: error_func(substring) if error_func - else f"{format_error_context(row_schema, friendly_name, item_num)}: {value} does not start with {substring}.", + else f"{format_error_context(eargs)} {eargs.value} does not start with {substring}.", ) @@ -386,8 +420,7 @@ def isNumber(): """Validate that value can be casted to a number.""" return make_validator( lambda value: str(value).strip().isnumeric(), - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: {value} is not a number." + lambda eargs: f"{format_error_context(eargs)} {eargs.value} is not a number." ) @@ -395,8 +428,7 @@ def isAlphaNumeric(): """Validate that value is alphanumeric.""" return make_validator( lambda value: value.isalnum(), - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: {value} is not alphanumeric." + lambda eargs: f"{format_error_context(eargs)} {eargs.value} is not alphanumeric." ) @@ -404,8 +436,7 @@ def isBlank(): """Validate that string value is blank.""" return make_validator( lambda value: value.isspace(), - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: {value} is not blank." + lambda eargs: f"{format_error_context(eargs)} {eargs.value} is not blank." ) @@ -413,8 +444,7 @@ def isInStringRange(lower, upper): """Validate that string value is in a specific range.""" return make_validator( lambda value: int(value) >= lower and int(value) <= upper, - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: {value} is not in range [{lower}, {upper}].", + lambda eargs: f"{format_error_context(eargs)} {eargs.value} is not in range [{lower}, {upper}].", ) @@ -422,8 +452,7 @@ def isStringLargerThan(val): """Validate that string value is larger than val.""" return make_validator( lambda value: int(value) > val, - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: {value} is not larger than {val}.", + lambda eargs: f"{format_error_context(eargs)} {eargs.value} is not larger than {val}.", ) @@ -438,9 +467,9 @@ def notEmpty(start=0, end=None): """Validate that string value isn't only blanks.""" return make_validator( lambda value: not _is_empty(value, start, end), - lambda value, row_schema, friendly_name, item_num: - f'{format_error_context(row_schema, friendly_name, item_num)}: {str(value)} contains blanks ' - f'between positions {start} and {end if end else len(str(value))}.' + lambda eargs: + f'{format_error_context(eargs)} {str(eargs.value)} contains blanks ' + f'between positions {start} and {end if end else len(str(eargs.value))}.' ) @@ -448,9 +477,9 @@ def isEmpty(start=0, end=None): """Validate that string value is only blanks.""" return make_validator( lambda value: _is_empty(value, start, end), - lambda value, row_schema, friendly_name, item_num: - f'{format_error_context(row_schema, friendly_name, item_num)}: {value} is not blank ' - f'between positions {start} and {end if end else len(value)}.' + lambda eargs: + f'{format_error_context(eargs)} {eargs.value} is not blank ' + f'between positions {start} and {end if end else len(eargs.value)}.' ) @@ -458,8 +487,7 @@ def notZero(number_of_zeros=1): """Validate that value is not zero.""" return make_validator( lambda value: value != "0" * number_of_zeros, - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: {value} is zero." + lambda eargs: f"{format_error_context(eargs)} {eargs.value} is zero." ) @@ -467,8 +495,7 @@ def isLargerThan(LowerBound): """Validate that value is larger than the given value.""" return make_validator( lambda value: float(value) > LowerBound if value is not None else False, - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: {value} is not larger than {LowerBound}.", + lambda eargs: f"{format_error_context(eargs)} {eargs.value} is not larger than {LowerBound}.", ) @@ -476,8 +503,7 @@ def isSmallerThan(UpperBound): """Validate that value is smaller than the given value.""" return make_validator( lambda value: value < UpperBound, - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: {value} is not smaller than {UpperBound}.", + lambda eargs: f"{format_error_context(eargs)} {eargs.value} is not smaller than {UpperBound}.", ) @@ -485,8 +511,7 @@ def isLargerThanOrEqualTo(LowerBound): """Validate that value is larger than the given value.""" return make_validator( lambda value: value >= LowerBound, - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: {value} is not larger than {LowerBound}.", + lambda eargs: f"{format_error_context(eargs)} {eargs.value} is not larger than {LowerBound}.", ) @@ -494,8 +519,7 @@ def isSmallerThanOrEqualTo(UpperBound): """Validate that value is smaller than the given value.""" return make_validator( lambda value: value <= UpperBound, - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: {value} is not smaller than {UpperBound}.", + lambda eargs: f"{format_error_context(eargs)} {eargs.value} is not smaller than {UpperBound}.", ) @@ -503,8 +527,8 @@ def isInLimits(LowerBound, UpperBound): """Validate that value is in a range including the limits.""" return make_validator( lambda value: int(value) >= LowerBound and int(value) <= UpperBound, - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: {value} is not larger or equal " + lambda eargs: + f"{format_error_context(eargs)} {eargs.value} is not larger or equal " f"to {LowerBound} and smaller or equal to {UpperBound}." ) @@ -514,16 +538,14 @@ def dateMonthIsValid(): """Validate that in a monthyear combination, the month is a valid month.""" return make_validator( lambda value: int(str(value)[4:6]) in range(1, 13), - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: {str(value)[4:6]} is not a valid month.", + lambda eargs: f"{format_error_context(eargs)} {str(eargs.value)[4:6]} is not a valid month.", ) def dateDayIsValid(): """Validate that in a monthyearday combination, the day is a valid day.""" return make_validator( lambda value: int(str(value)[6:]) in range(1, 32), - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: {str(value)[6:]} is not a valid day.", + lambda eargs: f"{format_error_context(eargs)} {str(eargs.value)[6:]} is not a valid day.", ) @@ -531,8 +553,8 @@ def olderThan(min_age): """Validate that value is larger than min_age.""" return make_validator( lambda value: datetime.date.today().year - int(str(value)[:4]) > min_age, - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: {str(value)[:4]} must be less " + lambda eargs: + f"{format_error_context(eargs)} {str(eargs.value)[:4]} must be less " f"than or equal to {datetime.date.today().year - min_age} to meet the minimum age requirement." ) @@ -541,9 +563,7 @@ def dateYearIsLargerThan(year): """Validate that in a monthyear combination, the year is larger than the given year.""" return make_validator( lambda value: int(str(value)[:4]) > year, - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: " - f"Year {str(value)[:4]} must be larger than {year}.", + lambda eargs: f"{format_error_context(eargs)} Year {str(eargs.value)[:4]} must be larger than {year}.", ) @@ -551,8 +571,7 @@ def quarterIsValid(): """Validate in a year quarter combination, the quarter is valid.""" return make_validator( lambda value: int(str(value)[-1]) > 0 and int(str(value)[-1]) < 5, - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: {str(value)[-1]} is not a valid quarter.", + lambda eargs: f"{format_error_context(eargs)} {str(eargs.value)[-1]} is not a valid quarter.", ) @@ -561,8 +580,7 @@ def validateSSN(): options = [str(i) * 9 for i in range(0, 10)] return make_validator( lambda value: value not in options, - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: {value} is in {options}." + lambda eargs: f"{format_error_context(eargs)} {eargs.value} is in {options}." ) @@ -570,8 +588,8 @@ def validateRace(): """Validate race.""" return make_validator( lambda value: value >= 0 and value <= 2, - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: {value} is not greater than or equal to 0 " + lambda eargs: + f"{format_error_context(eargs)} {eargs.value} is not greater than or equal to 0 " "or smaller than or equal to 2." ) @@ -582,8 +600,8 @@ def validateRptMonthYear(): lambda value: value[2:8].isdigit() and int(value[2:6]) > 1900 and value[6:8] in {"01", "02", "03", "04", "05", "06", "07", "08", "09", "10", "11", "12"}, - lambda value, row_schema, friendly_name, item_num: - f"{format_error_context(row_schema, friendly_name, item_num)}: The value: {value[2:8]}, " + lambda eargs: + f"{format_error_context(eargs)} The value: {eargs.value[2:8]}, " "does not follow the YYYYMM format for Reporting Year and Month.", ) diff --git a/tdrs-backend/tdpservice/search_indexes/admin/filters.py b/tdrs-backend/tdpservice/search_indexes/admin/filters.py index 517d27e487..36e1e66daa 100644 --- a/tdrs-backend/tdpservice/search_indexes/admin/filters.py +++ b/tdrs-backend/tdpservice/search_indexes/admin/filters.py @@ -61,9 +61,23 @@ def _get_lookup_choices(self, request): else: type_query = Query(type=STT.EntityType.STATE) | Query(type=STT.EntityType.TERRITORY) queryset = queryset.filter(type_query) - return (queryset.distinct().order_by('name').values_list('name', flat=True)) + def lookups(self, request, model_admin): + """Available options in dropdown.""" + listing_record_type = [i for i in str(request.path).lower().split('/') if i not in ['/', '']][-1] + options = [] + if 'tribal' in listing_record_type: + objects = STT.objects.filter(type=STT.EntityType.TRIBE) + elif 'ssp' in listing_record_type: + objects = STT.objects.filter(ssp=True) + else: + objects = STT.objects.filter(Query(type=STT.EntityType.STATE) | Query(type=STT.EntityType.TERRITORY)) + for obj in objects: + options.append((obj.stt_code, _(obj.name))) + + return options + class FiscalPeriodFilter(SimpleListFilter): """Simple filter class to filter records based on datafile fiscal year.""" diff --git a/tdrs-backend/tdpservice/search_indexes/migrations/0029_tanf_tribal_ssp_alter_verbose_names.py b/tdrs-backend/tdpservice/search_indexes/migrations/0029_tanf_tribal_ssp_alter_verbose_names.py new file mode 100644 index 0000000000..768c91d1cb --- /dev/null +++ b/tdrs-backend/tdpservice/search_indexes/migrations/0029_tanf_tribal_ssp_alter_verbose_names.py @@ -0,0 +1,97 @@ +# Generated by Django 3.2.15 on 2024-05-15 16:49 + +from django.db import migrations + + +class Migration(migrations.Migration): + + dependencies = [ + ('search_indexes', '0028_education_level_to_string'), + ] + + operations = [ + migrations.AlterModelOptions( + name='ssp_m1', + options={'verbose_name': 'SSP M1'}, + ), + migrations.AlterModelOptions( + name='ssp_m2', + options={'verbose_name': 'SSP M2'}, + ), + migrations.AlterModelOptions( + name='ssp_m3', + options={'verbose_name': 'SSP M3'}, + ), + migrations.AlterModelOptions( + name='ssp_m4', + options={'verbose_name': 'SSP M4'}, + ), + migrations.AlterModelOptions( + name='ssp_m5', + options={'verbose_name': 'SSP M5'}, + ), + migrations.AlterModelOptions( + name='ssp_m6', + options={'verbose_name': 'SSP M6'}, + ), + migrations.AlterModelOptions( + name='ssp_m7', + options={'verbose_name': 'SSP M7'}, + ), + migrations.AlterModelOptions( + name='tanf_t1', + options={'verbose_name': 'TANF T1'}, + ), + migrations.AlterModelOptions( + name='tanf_t2', + options={'verbose_name': 'TANF T2'}, + ), + migrations.AlterModelOptions( + name='tanf_t3', + options={'verbose_name': 'TANF T3'}, + ), + migrations.AlterModelOptions( + name='tanf_t4', + options={'verbose_name': 'TANF T4'}, + ), + migrations.AlterModelOptions( + name='tanf_t5', + options={'verbose_name': 'TANF T5'}, + ), + migrations.AlterModelOptions( + name='tanf_t6', + options={'verbose_name': 'TANF T6'}, + ), + migrations.AlterModelOptions( + name='tanf_t7', + options={'verbose_name': 'TANF T7'}, + ), + migrations.AlterModelOptions( + name='tribal_tanf_t1', + options={'verbose_name': 'Tribal TANF T1'}, + ), + migrations.AlterModelOptions( + name='tribal_tanf_t2', + options={'verbose_name': 'Tribal TANF T2'}, + ), + migrations.AlterModelOptions( + name='tribal_tanf_t3', + options={'verbose_name': 'Tribal TANF T3'}, + ), + migrations.AlterModelOptions( + name='tribal_tanf_t4', + options={'verbose_name': 'Tribal TANF T4'}, + ), + migrations.AlterModelOptions( + name='tribal_tanf_t5', + options={'verbose_name': 'Tribal TANF T5'}, + ), + migrations.AlterModelOptions( + name='tribal_tanf_t6', + options={'verbose_name': 'Tribal TANF T6'}, + ), + migrations.AlterModelOptions( + name='tribal_tanf_t7', + options={'verbose_name': 'Tribal TANF T7'}, + ), + ] diff --git a/tdrs-backend/tdpservice/search_indexes/models/ssp.py b/tdrs-backend/tdpservice/search_indexes/models/ssp.py index bb58403234..78c035db2d 100644 --- a/tdrs-backend/tdpservice/search_indexes/models/ssp.py +++ b/tdrs-backend/tdpservice/search_indexes/models/ssp.py @@ -12,6 +12,11 @@ class SSP_M1(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'SSP M1' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile, @@ -78,6 +83,11 @@ class SSP_M2(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'SSP M2' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile, @@ -165,6 +175,11 @@ class SSP_M3(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'SSP M3' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile, @@ -206,6 +221,11 @@ class SSP_M4(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'SSP M4' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile, @@ -240,6 +260,11 @@ class SSP_M5(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'SSP M5' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile, @@ -286,6 +311,11 @@ class SSP_M6(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'SSP M6' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile, @@ -318,6 +348,11 @@ class SSP_M7(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'SSP M7' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile, diff --git a/tdrs-backend/tdpservice/search_indexes/models/tanf.py b/tdrs-backend/tdpservice/search_indexes/models/tanf.py index bac3f008da..c57ab89ed9 100644 --- a/tdrs-backend/tdpservice/search_indexes/models/tanf.py +++ b/tdrs-backend/tdpservice/search_indexes/models/tanf.py @@ -12,6 +12,11 @@ class TANF_T1(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'TANF T1' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile, @@ -81,6 +86,11 @@ class TANF_T2(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'TANF T2' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile, @@ -170,6 +180,11 @@ class TANF_T3(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'TANF T3' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile, @@ -211,6 +226,11 @@ class TANF_T4(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'TANF T4' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile, @@ -243,6 +263,11 @@ class TANF_T5(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'TANF T5' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile, @@ -292,6 +317,11 @@ class TANF_T6(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'TANF T6' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile, @@ -329,6 +359,11 @@ class TANF_T7(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'TANF T7' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile, diff --git a/tdrs-backend/tdpservice/search_indexes/models/tribal.py b/tdrs-backend/tdpservice/search_indexes/models/tribal.py index fd5ee805d1..d428183364 100644 --- a/tdrs-backend/tdpservice/search_indexes/models/tribal.py +++ b/tdrs-backend/tdpservice/search_indexes/models/tribal.py @@ -14,6 +14,11 @@ class Tribal_TANF_T1(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'Tribal TANF T1' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile, @@ -83,6 +88,11 @@ class Tribal_TANF_T2(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'Tribal TANF T2' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile, @@ -155,6 +165,11 @@ class Tribal_TANF_T3(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'Tribal TANF T3' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile, @@ -195,6 +210,11 @@ class Tribal_TANF_T4(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'Tribal TANF T4' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile, @@ -226,6 +246,11 @@ class Tribal_TANF_T5(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'Tribal TANF T5' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile, @@ -274,6 +299,11 @@ class Tribal_TANF_T6(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'Tribal TANF T6' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile, @@ -310,6 +340,11 @@ class Tribal_TANF_T7(models.Model): Mapped to an elastic search index. """ + class Meta: + """Meta class for the model.""" + + verbose_name = 'Tribal TANF T7' + id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False) datafile = models.ForeignKey( DataFile,