Merge branch 'develop' into 3004-clean-and-reparse-cmd

raft-tech · Jul 17, 2024 · bcf550b · bcf550b
2 parents 09d2974 + fd5042e
commit bcf550b
Show file tree

Hide file tree

Showing 11 changed files with 1,286 additions and 273 deletions.
diff --git a/docs/Sprint-Review/sprint-101-summary.md b/docs/Sprint-Review/sprint-101-summary.md
@@ -0,0 +1,87 @@
+# sprint-101-summary
+
+6/5/2024 - 6/18/2024
+
+**Dev:**
+
+_**Prioritized DAC and Notifications Work**_&#x20;
+
+* As sys admin, I want to be able to reparse datafile sets #2978
+* As a software engineer, I want to be able to test django-admin-508 #3008
+* As tech lead, I need the STT filter for search\_indexes to be updated #2950
+* As a data analyst I want to be notified of approaching data deadlines #2473
+* add `SENDGRID_API_KEY` to deploy.backend.sh #2677
+* Implement (small) data lifecycle (backup/archive ES) #3004
+* As a developer I want to test django-508 repo #2980\
+
+
+**DevOps:**
+
+_**Successful deployments across environments and pipeline stability investments**_
+
+* Application health monitoring #831
+
+**Design:**
+
+_**Close out error guide work, coordinate with dev on a plan for Cat 3 problems introduced by Cat 2 work, support spec-writing for upcoming work, and continued error audit dev ticket refinement.**_
+
+* Error Report Guide #2847 is going through final edits&#x20;
+  * Walk-on Dear Colleague letter link update to this PR (or spin up a separate ticket if deployment of the letter to OFA's website doesn't align to this)
+* Deliver spec for #3014 (Blanked-out values in Submission History)
+* \#3021 Updated KC Release Notes & Update Indicator FAQ - stretch goal for this sprint
+* Write follow-on / spec tickets from #2909 findings - stretch/ongoing lift
+* Category 3 error messages clean-up #2792 - stretch/ongoing lift
+
+## Tickets
+
+### Completed/Merged
+
+* [#2980 As a developer I want to test django-508 repo](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2980)
+* [#2892 Correct misleading error message for unaligned reporting year/q against header year/q](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2892)
+* [#2909 \[Research Spike\] OOtB OFA Kibana Experience & DIGIT Data Access](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2909)
+* [#2991 As tech lead, I need the sftp file transfer feature to be deprecated](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2991)
+* [#2847 \[Design Deliverable\] Error Report Knowledge Center Explainer](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2847)
+* [#3024 2897 follow-on for a11y-related enhancement ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3024)
+* [#2897 As a data analyst I want finalized language and guidance resources in Submission History & Error Reports ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2897)
+
+### Submitted (QASP Review, OCIO Review)
+
+* [#2133 \[Dev\] Enhancement for Request Access form (Tribe discoverability) ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2133)
+* [#3023 as STT approved user, I need my IP address whitelisted so i can access TDP](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3023)
+* [#3000 \[Design Deliverable\] TDP Poster for summer 2024 conferences](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3000)
+* [#2795 As tech lead, I need TDP to detect duplicate records within a file and not store them in the db. ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2795)
+* [#2693 \[Error Audit\] Category 2 error messages clean-up ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2693)
+* [#2801 Friendly name cleanup ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2801)
+* [#2883 Pre-Made Reporting Dashboards on Kibana](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2883)
+* [#3021 \[Design Deliverable\] Updated KC Release Notes & Update Indicator FAQ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3021)
+* [#2954 Extend SESSION\_COOKIE\_AGE](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2954)
+
+### Ready to Merge
+
+*
+
+### Closed (Not Merged)
+
+* [#2491 Create root-level docker-compose configuration file(s)](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2491)
+* [#1690 As a system admin, I need a way to be redirected to frontend from DAC](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/1690)
+* [#2351 As a user I want to be notified when the files are being scanned or uploaded when I push upload button](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2351)
+* [#2591 Allow `manage.py` commands to be run by circleci](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2591)
+
+### Moved to Next Sprint&#x20;
+
+**In Progress**&#x20;
+
+* [#3004 Implement (small) data lifecycle (backup/archive ES)](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3004)
+* [#831 \[Spike\] As a Tech Lead, I want to get alerts when there is a backend or frontend error that affects an STT user ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/831)
+* [#2978 As sys admin, I want to be able to reparse datafile sets](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2978)
+
+#### Blocked
+
+*
+
+**Raft Review**
+
+* [#2950 As tech lead, I need the STT filter for search\_indexes to be updated ](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2950)
+* [#3008 As a software engineer, I want to be able to test django-admin-508](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3008)
+* [#3016 Spike - Cat2 Validator Improvement](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/3016)
+* [#2473  As a data analyst I want to be notified of approaching data deadlines](https://app.zenhub.com/workspaces/sprint-board-5f18ab06dfd91c000f7e682e/issues/gh/raft-tech/tanf-app/2473)
diff --git a/tdrs-backend/tdpservice/data_files/test/test_api.py b/tdrs-backend/tdpservice/data_files/test/test_api.py
@@ -100,10 +100,10 @@ def assert_error_report_tanf_file_content_matches_with_friendly_names(response):
 
         assert ws.cell(row=1, column=1).value == "Please refer to the most recent versions of the coding " \
             + "instructions (linked below) when looking up items and allowable values during the data revision process"
-        assert ws.cell(row=8, column=COL_ERROR_MESSAGE).value == ("if Cash Amount :873 validator1 passed then Cash and "
-                                                                  "Cash Equivalents: Number of Months T1 Item -1 (Cash "
-                                                                  "and Cash Equivalents: Number of Months): 0 is not "
-                                                                  "larger than 0.")
+        assert ws.cell(row=8, column=COL_ERROR_MESSAGE).value == (
+            "if Cash Amount :873 validator1 passed then Item 21B "
+            "(Cash and Cash Equivalents: Number of Months) 0 is not larger than 0."
+        )
 
     @staticmethod
     def assert_error_report_ssp_file_content_matches_with_friendly_names(response):
@@ -134,9 +134,10 @@ def assert_error_report_file_content_matches_without_friendly_names(response):
 
         assert ws.cell(row=1, column=1).value == "Please refer to the most recent versions of the coding " \
             + "instructions (linked below) when looking up items and allowable values during the data revision process"
-        assert ws.cell(row=8, column=COL_ERROR_MESSAGE).value == ("if CASH_AMOUNT :873 validator1 passed then "
-                                                                  "NBR_MONTHS T1 Item -1 (NBR_MONTHS): 0 is not "
-                                                                  "larger than 0.")
+        assert ws.cell(row=8, column=COL_ERROR_MESSAGE).value == (
+            "if CASH_AMOUNT :873 validator1 passed then Item 21B "
+            "(Cash and Cash Equivalents: Number of Months) 0 is not larger than 0."
+        )
 
     @staticmethod
     def assert_data_file_exists(data_file_data, version, user):

diff --git a/tdrs-backend/tdpservice/parsers/row_schema.py b/tdrs-backend/tdpservice/parsers/row_schema.py
@@ -1,7 +1,7 @@
 """Row schema for datafile."""
 from .models import ParserErrorCategoryChoices
 from .fields import Field, TransformField
-from .validators import value_is_empty, format_error_context
+from .validators import value_is_empty, format_error_context, ValidationErrorArgs
 import logging
 
 logger = logging.getLogger(__name__)
@@ -154,12 +154,20 @@ def run_field_validators(self, instance, generate_error):
                         )
             elif field.required:
                 is_valid = False
+                eargs = ValidationErrorArgs(
+                    value=value,
+                    row_schema=self,
+                    friendly_name=field.friendly_name,
+                    item_num=field.item,
+                    error_context_format='prefix'
+                )
+
                 errors.append(
                     generate_error(
                         schema=self,
                         error_category=ParserErrorCategoryChoices.FIELD_VALUE,
                         error_message=(
-                            f"{format_error_context(self, field.friendly_name, field.item)}: "
+                            f"{format_error_context(eargs)} "
                             "field is required but a value was not provided."
                         ),
                         record=instance,

diff --git a/tdrs-backend/tdpservice/parsers/test/test_parse.py b/tdrs-backend/tdpservice/parsers/test/test_parse.py
@@ -1673,10 +1673,12 @@ def test_parse_m2_cat2_invalid_37_38_39_file(m2_cat2_invalid_37_38_39_file, dfs)
 
     assert parser_errors.count() == 3
 
-    error_msgs = {"M2 Item 37 (Educational Level): 00 is not in range [1, 16]. or M2 Item 37 (Educational Level): " +
-                  "00 is not in range [98, 99].",
-                  "M2 Item 38 (Citizenship/Immigration Status): 0 is not in [1, 2, 3, 9].",
-                  "M2 Item 39 (Cooperated with Child Support): 0 is not in [1, 2, 9]."}
+    error_msgs = {
+        "Item 37 (Educational Level) 00 is not in range [1, 16]. or "
+        "Item 37 (Educational Level) 00 is not in range [98, 99].",
+        "M2 Item 38 (Citizenship/Immigration Status): 0 is not in [1, 2, 3, 9].",
+        "M2 Item 39 (Cooperated with Child Support): 0 is not in [1, 2, 9]."
+    }
     for e in parser_errors:
         assert e.error_message in error_msgs
 
@@ -1697,10 +1699,10 @@ def test_parse_m3_cat2_invalid_68_69_file(m3_cat2_invalid_68_69_file, dfs):
 
     assert parser_errors.count() == 4
 
-    error_msgs = {"M3 Item 68 (Educational Level): 00 is not in range [1, 16]. or M3 Item 68 (Educational Level): " +
+    error_msgs = {"Item 68 (Educational Level) 00 is not in range [1, 16]. or Item 68 (Educational Level) " +
                   "00 is not in range [98, 99].",
                   "M3 Item 69 (Citizenship/Immigration Status): 0 is not in [1, 2, 3, 9].",
-                  "M3 Item 68 (Educational Level): 00 is not in range [1, 16]. or M3 Item 68 (Educational Level): " +
+                  "Item 68 (Educational Level) 00 is not in range [1, 16]. or Item 68 (Educational Level) " +
                   "00 is not in range [98, 99].",
                   "M3 Item 69 (Citizenship/Immigration Status): 0 is not in [1, 2, 3, 9]."}