SSP Section 2 Parsing & Validation

[elipe17]

Summary of Changes

• Updated code to parse and validate SSP section 2 records based on provided spreadsheet.
  Pull request closes SSP Closed Data (02) File Parsing - As a data analyst, I want my data files to be parsed so that my submission can be validated/accepted by TDP #1118

How to Test

cd tdrs-frontend && docker-compose up
cd tdrs-backend && docker-compose up

1. Log in as user who can submit ssp datafiles
2. submit ADS.E2J.NDM2.MS24
3. Verify M4 and M5 records are created. There will also be a lot of parser errors
4. Run the unit tests

Deliverables

More details on how deliverables herein are assessed included here.

Deliverable 1: Accepted Features

Checklist of ACs:

• Data File parsing logic is in place such that data file submission can be completed by user and parsed by system for full submission once errors are resolved.
• Data Errors tell user specifically where in the file they occur, and therefore what exactly needs to be fixed in order for the submission to be complete
• TDP gives the user all data errors for a submission in a way they can easily save or export so they can take the issues back to their case management software and fix the issues then recreate an export for resubmission to TDP.
• Data Errors must be issued in human readable plain language so that an english speaker can easily understand what is needed.

Deliverable 2: Tested Code

• Are all areas of code introduced in this PR meaningfully tested?
  • If this PR introduces backend code changes, are they meaningfully tested?
  • If this PR introduces frontend code changes, are they meaningfully tested?
• Are code coverage minimums met?
  • Frontend coverage: [insert coverage %] (see CodeCov Report comment in PR)
  • Backend coverage: [insert coverage %] (see CodeCov Report comment in PR)

Deliverable 3: Properly Styled Code

• Are backend code style checks passing on CircleCI?
• Are frontend code style checks passing on CircleCI?
• Are code maintainability principles being followed?

Deliverable 4: Accessible

• Does this PR complete the epic?
• Are links included to any other gov-approved PRs associated with epic?
• Does PR include documentation for Raft's a11y review?
• Did automated and manual testing with iamjolly and ttran-hub using Accessibility Insights reveal any errors introduced in this PR?

Deliverable 5: Deployed

• Was the code successfully deployed via automated CircleCI process to development on Cloud.gov?

Deliverable 6: Documented

• Does this PR provide background for why coding decisions were made?
• If this PR introduces backend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces frontend code, is that code easy to understand and sufficiently documented, both inline and overall?
• If this PR introduces dependencies, are their licenses documented?
• Can reviewer explain and take ownership of these elements presented in this code review?

Deliverable 7: Secure

• Does the OWASP Scan pass on CircleCI?
• Do manual code review and manual testing detect any new security issues?
• If new issues detected, is investigation and/or remediation plan documented?

Deliverable 8: User Research

Research product(s) clearly articulate(s):

• the purpose of the research
• methods used to conduct the research
• who participated in the research
• what was tested and how
• impact of research on TDP
• (if applicable) final design mockups produced for TDP development

[codecov]

Codecov Report

Merging #2724 (624828f) into develop (b614166) will decrease coverage by 0.04%.
Report is 3 commits behind head on develop.
The diff coverage is 100.00%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2724      +/-   ##
===========================================
- Coverage    92.83%   92.79%   -0.04%     
===========================================
  Files           44      246     +202     
  Lines          977     5568    +4591     
  Branches       160      480     +320     
===========================================
+ Hits           907     5167    +4260     
- Misses          37      308     +271     
- Partials        33       93      +60     

Flag	Coverage Δ
dev-backend	92.79% <100.00%> (?)
dev-frontend	92.83% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
tdrs-backend/tdpservice/parsers/parse.py	86.87% <100.00%> (ø)
...s-backend/tdpservice/parsers/schema_defs/header.py	100.00% <ø> (ø)
...end/tdpservice/parsers/schema_defs/ssp/__init__.py	100.00% <100.00%> (ø)
...s-backend/tdpservice/parsers/schema_defs/ssp/m1.py	100.00% <100.00%> (ø)
...s-backend/tdpservice/parsers/schema_defs/ssp/m2.py	100.00% <100.00%> (ø)
...s-backend/tdpservice/parsers/schema_defs/ssp/m3.py	100.00% <100.00%> (ø)
...s-backend/tdpservice/parsers/schema_defs/ssp/m4.py	100.00% <100.00%> (ø)
...s-backend/tdpservice/parsers/schema_defs/ssp/m5.py	100.00% <100.00%> (ø)
...s-backend/tdpservice/parsers/schema_defs/ssp/m6.py	100.00% <100.00%> (ø)
...s-backend/tdpservice/parsers/schema_defs/ssp/m7.py	100.00% <100.00%> (ø)
... and 18 more

... and 174 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f1f15c1...624828f. Read the comment docs.

[jtimpe]

fix the missing validator and this is good to go! a couple additional questions for clarification

[ADPennington]

@elipe17 @jtimpe in good shape! please note:

• i suggested changes for a few validators that i missed. reflected in workbook as well.
• wanted to confirm the space-filled fields are accounted for.
• can you briefly summarize what changed in the t1-t7 and m1-m3 schemas? just formatting? want to make sure i dont need to re-test those.

[jtimpe]

@ADPennington

can you briefly summarize what changed in the t1-t7 and m1-m3 schemas? just formatting? want to make sure i dont need to re-test those.

these changes are formatting that comes from running the python utility black. i think the unit test coverage should be sufficient to say it doesn't cause an issue. the formatter will likely need to be re-run for ssp section 1 at some point, though, as my changes didn't have the formatting applied.

[ADPennington]

looks great @elipe17 @jtimpe 🚀