Tribal Section 1 Parsing & Validation

[elipe17]

Summary of Changes

• Added support for Tribal TANF section 1 parsing and validation
• Added new pre-check error that manages tribe code header inconsistencies
  Pull request closes Tribal TANF Active Data (01) File Parsing - As a data analyst, I want my data files to be parsed so that my submission can be validated/accepted by TDP #1114

How to Test

cd tdrs-frontend && docker-compose up
cd tdrs-backend && docker-compose up

• Open http://localhost:3000/ and sign in.
• Submit both ADS.E2J.FTP1.TS142 and tribal_section_1_inconsistency.txt.
• Verify the first file generates Tribal T1/T2/T3 records.
• Verify the second file does not generate any records and throws a pre-check error describing an inconsistency with the tribe code and other header fields.

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

• Data File parsing logic is in place such that data file submission can be completed by user and parsed by system for full submission once errors are resolved.
• Data Errors tell user specifically where in the file they occur, and therefore what exactly needs to be fixed in order for the submission to be complete
• TDP gives the user all data errors for a submission in a way they can easily save or export so they can take the issues back to their case management software and fix the issues then recreate an export for resubmission to TDP.
• Data Errors must be issued in human readable plain language so that an English speaker can easily understand what is needed.
• Testing Checklist has been run and all tests pass
• README is updated, if necessary

Deliverable 2: Tested Code

• Are all areas of code introduced in this PR meaningfully tested?
  • If this PR introduces backend code changes, are they meaningfully tested?
  • If this PR introduces frontend code changes, are they meaningfully tested?
• Are code coverage minimums met?
  • Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
  • Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

• Are backend code style checks passing on CircleCI?
• Are frontend code style checks passing on CircleCI?
• Are code maintainability principles being followed?

Deliverable 4: Accessible

• Does this PR complete the epic?
• Are links included to any other gov-approved PRs associated with epic?
• Does PR include documentation for Raft's a11y review?
• Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

• Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

• Does this PR provide background for why coding decisions were made?
• If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces dependencies, are their licenses documented?
• Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

• Does the OWASP Scan pass on CircleCI?
• Do manual code review and manual testing detect any new security issues?
• If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

• the purpose of the research
• methods used to conduct the research
• who participated in the research
• what was tested and how
• impact of research on TDP
• (if applicable) final design mockups produced for TDP development

[ADPennington]

@elipe17 this is in pretty good shape. some testing notes and questions below. You'll notice that most of the comments reference other clean-up tickets. Let me know if it makes sense to keep those fixes in other tix. The only exception might be CASE_NUMBER.

---

Header/Trailer

• tribes will often space-fill or zero-fill the state_fips. space-filling raises an error message: state_fips is required but a value was not provided.
• currently not storing FIPS_CODE from header (state_fips) ❓
• currently not storing TRIBE_CODE from header (tribe_code) ❓
• how will OFA staff who query the db know which STTs each record is associated with? seems like we need the abovementioned fields in every schema? possibly a new ticket? ❓

T1, T2, and T3

• verified that tribal section 1 files have T1,T2, and T3 records have length == 122. this was implemented here ✔️
• records with RPT_MONTH_YEAR needs better validator for YYYY. update dateYearIsLargerThan() validator logic for RPT_MONTH_YEAR  #2793
• CASE_NUMBER validator needs an update to handle edgy case numbers. see comment here
• COUNTY_FIPS_CODE validation correct but also related to a bug in FTANF executables that erroneously drops leading zeros. As tech lead, I need validation checks to be consistent with FTANF validation checks.  #2749
• DATE_OF_BIRTH needs logic from As an STT, I need to know if there are issues with the DOBs reported in my data files #2746
• SSN needs logic from Update validateSSN validator to allow for all 0s,1s,...,9s as valid possible values except in cat3 scenarios.  #2771
• a few suggestions/comments related to friendly names and range of values I missed.

---

---

for @reitermb --please double check that these are captured in the hackmd:

• cat 2 error messages look like the following ([Error Audit] Category 2 error messages clean-up  #2693):

  • 2 does not match 1.
  • 3 is not in [1, 2].
  • 3 is not larger or equal to 1 and smaller or equal to 2.
  • family exempt time limits is required but a value was not provided.
  • 04 is not in range [0, 3]. or 04 is not in range [5, 9]. or 04 is not in range [11, 19]. or 04 does not match 99.

• cat 3 error mesages look like the following ([Error Audit] Category 3 error messages clean-up #2792):

  • if cash amount :502 validator1 passed then number of months 0 is not larger than 0.
  • if child care amount :3 validator1 passed then children covered 0 is not larger than 0.
  • if sanction reduction amount :3 validator1 passed then work requirement sanction 0 is not in (1, 2).
  • The sum of ('amount of food stamp assistance', 'amount of subsidized child care', 'cash amount', 'child care amount', 'transportation amount', 'transition services amount', 'other amount') is not larger than 0.
  • if family affiliation :1 validator1 passed then work participation status 04 is not in range [1, 3]. or 04 is not in range [5, 9]. or 04 is not in range [11, 19]. or 04 does not match 99.
  • If family affiliation ==2 and citizenship status==1 or 2, then social security number - ssn != 000000000 -- 999999999.

[elipe17]

@ADPennington I updated based on your review. I think all the validator changes should live in those tickets you called out. That way we can update all sections for programs in one go. I would also like to make the fips_code/tribe_code addition to the models/documents it's own ticket. Adding those to the models that are missing them will create a new migration. The migration generated will be a big one since it will have to impute a new column and value for the DB tables that don't have it. Those tables are pretty big which will cause a reasonably long running migration.

[ADPennington]

@ADPennington I updated based on your review. I think all the validator changes should live in those tickets you called out. That way we can update all sections for programs in one go. I would also like to make the fips_code/tribe_code addition to the models/documents it's own ticket. Adding those to the models that are missing them will create a new migration. The migration generated will be a big one since it will have to impute a new column and value for the DB tables that don't have it. Those tables are pretty big which will cause a reasonably long running migration.

sounds good thank you @elipe17. we may need to separate new tickets:

• for fips and tribe code storage in the record
• to address the case number validator. currently that ticket is focused on TANF Section 1 but the validator is used in SSP and Tribal TANF.

[ADPennington]

@elipe17 this is ready for approval as soon as this comment is addressed.

[ADPennington]

thank you @elipe17 🚀