Ticket #2616: populate suborg and portfolio script #2670

zandercymatics · 2024-08-28T20:54:49Z

Ticket

Resolves #2616

Changes

Adds a create_federal_portfolio script that takes the FederalAgency as a param
Adds a get_default_user function on User that returns the "System" user
Documentation
Tests

Context for reviewers

Background

This script automatically creates a portfolio for a given FederalAgency. A lot of information about these agencies are scattered throughout different tables, so this script synthesizes that.

As part of the AC, this script should be runnable more than once. This means that if you try to rerun this on a record that already exists, it will prompt the user where possible and update rather than attempt to recreate the record.

Other information

This is difficulty to accurately test without good data. I would suggest testing this PR on getgov-ag

As noted in the readme, the script has two params:

parse_domains <= adds the created portfolio record to domains (specifically - the domain information object)
parse_requests <= adds the created portfolio record to domain requests

Note: this only applies to one agency at a time. To pick a record to populate, go to the FederalAgency table and copy the agency name. For example, "AMTRAK".

Then run the script as so: ./manage.py create_federal_agency "AMTRAK" --parse_domains --parse_requests. Note that you can run the script as many times as you like, so you can also test this by just passing the parse_domains flag. You can also just pass in --both.

Given that no senior official data exists on ag, you will need to create a senior official record or use an existing one. When creating it, make sure to associate the senior official record to the given federal agency that you are testing.

Setup

You will need to test:

The created portfolio (organization_name, federal_agency, organization_type, creator, notes, senior_official)
Check that the given portfolio and right suborg is attached to domains when --parse_domains is ran
Check that the given portfolio and right suborg is attached to domain requests when --parse_requests is ran
Check all created Suborganization records

Code Review Verification Steps

As the original developer, I have

Satisfied acceptance criteria and met development standards

Met the acceptance criteria, or will meet them in a subsequent PR
Created/modified automated tests
Added at least 2 developers as PR reviewers (only 1 will need to approve)
Messaged on Slack or in standup to notify the team that a PR is ready for review
Changes to “how we do things” are documented in READMEs and or onboarding guide
If any model was updated to modify/add/delete columns, makemigrations was ran and the associated migrations file has been commited.

Ensured code standards are met (Original Developer)

All new functions and methods are commented using plain language
Did dependency updates in Pipfile also get changed in requirements.txt?
Interactions with external systems are wrapped in try/except
Error handling exists for unusual or missing values

Validated user-facing changes (if applicable)

New pages have been added to .pa11yci file so that they will be tested with our automated accessibility testing
Checked keyboard navigability
Tested general usability, landmarks, page header structure, and links with a screen reader (such as Voiceover or ANDI)
Add at least 1 designer as PR reviewer

As a code reviewer, I have

Reviewed, tested, and left feedback about the changes

Pulled this branch locally and tested it
Reviewed this code and left comments
Checked that all code is adequately covered by tests
Made it clear which comments need to be addressed before this work is merged
If any model was updated to modify/add/delete columns, makemigrations was ran and the associated migrations file has been commited.

Ensured code standards are met (Code reviewer)

All new functions and methods are commented using plain language
Interactions with external systems are wrapped in try/except
Error handling exists for unusual or missing values
(Rarely needed) Did dependency updates in Pipfile also get changed in requirements.txt?

Validated user-facing changes as a developer

Note: Multiple code reviewers can share the checklists above, a second reviewers should not make a duplicate checklist

As a designer reviewer, I have

Verified that the changes match the design intention

Checked that the design translated visually
Checked behavior
Checked different states (empty, one, some, error)
Checked for landmarks, page heading structure, and links
Tried to break the intended flow

Validated user-facing changes as a designer

Checked keyboard navigability
Tested general usability, landmarks, page header structure, and links with a screen reader (such as Voiceover or ANDI)
Tested with multiple browsers (check off which ones were used)
- Chrome
- Microsoft Edge
- FireFox
- Safari
(Rarely needed) Tested as both an analyst and applicant user

Screenshots

github-actions · 2024-08-28T20:59:54Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2024-08-29T14:40:48Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2024-08-29T16:04:41Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2024-08-29T16:46:08Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2024-08-29T16:52:33Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2024-08-29T17:31:43Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2024-08-29T17:38:23Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2024-08-29T17:46:49Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2024-08-29T18:18:04Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2024-08-29T20:22:57Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2024-08-29T21:22:36Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2024-08-30T14:44:43Z

🥳 Successfully deployed to developer sandbox ag.

zandercymatics · 2024-08-30T15:06:44Z

src/registrar/management/commands/create_federal_portfolio.py

+        parser.add_argument(
+            "--parse_requests",
+            action=argparse.BooleanOptionalAction,
+            help="Adds portfolio to DomainRequests",
+        )
+        parser.add_argument(
+            "--parse_domains",
+            action=argparse.BooleanOptionalAction,
+            help="Adds portfolio to DomainInformation",
+        )


@abroddrick What do you think of this particular approach? This is related to this comment. It seemed simpler to check for in handle but the UX of the script is important too! Let me know

I just saw you comment actually and replied. I prefer having "both" so I don't have to worry about missing a parameter it also makes running it a bit more straight forward in test envs when we don't really care what gets added.

That makes sense!

abroddrick · 2024-09-06T17:23:21Z

docs/operations/data_migration.md

@@ -838,3 +838,45 @@ Example: `cf ssh getgov-za`

 ### Running locally
 ```docker-compose exec app ./manage.py populate_domain_request_dates```
+
+## Create federal portfolio
+This script takes the name of a `FederalAgency` (like 'AMTRAK') and does the following:


The additional context of when this may fail etc should probably be in here as well (in a summed up format).

zandercymatics · 2024-09-06T19:23:49Z

@abroddrick

This wasn't an ac, suborg just has name and portfolio it should not modify an existing suborg, it should just skip it if a suborg already exists. Curious why this was added, as someone using this I don't want the extra step of having to confirm making suborgs when that is something that must happen with this script. Can you give more details here?

Ah! So, this dialog never occurs when you are creating suborgs, so you won't need to confirm when running the script in that fashion. For our use case, this prompt only occurs if you rerun the script two or more times on the same organization.

All of these Y/N dialog options only ever show in that particular scenario, they do not show when you run it for the first time.

Basically, I wrote this script such that it knows if you rerun it on the same org twice. This is a low-cost add and it has a lot of benefits, imo. I feel that it makes sense here. This should be documented however. Thoughts?

zandercymatics · 2024-09-06T19:55:03Z

If this is going to be a prompt, then it would be helpful to know what would be modified prior to the y/n prompt so the decision to modify can be made based on what is going to be changed
Also will the script remove suborgs if there are no longer domains with that suborg name?

Agreed on showing the changes! This script does not delete suborgs whatsoever, would you want that?

This should be red error. Every domain as organization name which is suborg. If for some reason there is no suborg this is a critical error. Again fed agency is not a string it comes from the fed agency value on domains

I agree! Just wanted to flag that with our current stable db we may have a few of those so maybe I'll leave some additional info. Basically, we may see this on a few of them

github-actions · 2024-09-10T14:39:45Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2024-09-10T20:30:19Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2024-09-11T15:10:53Z

🥳 Successfully deployed to developer sandbox ag.

src/registrar/management/commands/create_federal_portfolio.py

github-actions · 2024-09-11T15:17:13Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2024-09-11T16:49:11Z

🥳 Successfully deployed to developer sandbox ag.

Matt-Spence

Great work.

github-actions · 2024-09-12T14:27:45Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2024-09-12T14:43:23Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2024-09-12T14:52:06Z

🥳 Successfully deployed to developer sandbox ag.

github-actions · 2024-09-12T14:54:47Z

🥳 Successfully deployed to developer sandbox ag.

docs/operations/data_migration.md

github-actions · 2024-09-12T15:10:07Z

🥳 Successfully deployed to developer sandbox ag.

zandercymatics added 3 commits August 28, 2024 14:10

Create skeleton

061a096

Fix bugs

e5b1b5a

Update create_federal_portfolio.py

d35acc8

zandercymatics requested a review from abroddrick as a code owner August 28, 2024 20:54

Merge branch 'main' into ag/2616-populate-suborg-and-portfolio-script

27993e5

Add some better logging and options

8e6115a

linting

8d324a8

Comments

bfbce5a

Simplify logic

b162ae8

zandercymatics added 2 commits August 29, 2024 11:34

Avoid using an early return (consistency)

b27913f

Update create_federal_portfolio.py

12eab5c

use defaults

11d662a

Merge branch 'main' into ag/2616-populate-suborg-and-portfolio-script

e50805f

Unit tests

91e62a6

Update create_federal_portfolio.py

f228867

Remove logic for location information

b3b415b

zandercymatics commented Aug 30, 2024

View reviewed changes

lint

47b7ee4

abroddrick reviewed Sep 6, 2024

View reviewed changes

Merge branch 'main' into ag/2616-populate-suborg-and-portfolio-script

fbabd20

Add more logging detail

27f2157

zandercymatics commented Sep 11, 2024

View reviewed changes

src/registrar/management/commands/create_federal_portfolio.py Outdated Show resolved Hide resolved

Update src/registrar/management/commands/create_federal_portfolio.py

97c7826

Matt-Spence self-assigned this Sep 11, 2024

Merge branch 'main' into ag/2616-populate-suborg-and-portfolio-script

cf0671e

Matt-Spence approved these changes Sep 11, 2024

View reviewed changes

zandercymatics added 2 commits September 12, 2024 08:23

doc update

f2c4897

Merge branch 'main' into ag/2616-populate-suborg-and-portfolio-script

3ececbb

More detailed logging on senior official

060dbfc

zandercymatics added 2 commits September 12, 2024 08:47

Update create_federal_portfolio.py

87d35eb

Doc update

d787fcb

zandercymatics changed the title ~~(getgov-ag) Ticket #2616: populate suborg and portfolio script~~ Ticket #2616: populate suborg and portfolio script Sep 12, 2024

zandercymatics commented Sep 12, 2024

View reviewed changes

docs/operations/data_migration.md Outdated Show resolved Hide resolved

Update docs/operations/data_migration.md

066e796

zandercymatics merged commit 91b69a3 into main Sep 12, 2024
10 checks passed

zandercymatics deleted the ag/2616-populate-suborg-and-portfolio-script branch September 12, 2024 15:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ticket #2616: populate suborg and portfolio script #2670

Ticket #2616: populate suborg and portfolio script #2670

zandercymatics commented Aug 28, 2024 •

edited

Loading

github-actions bot commented Aug 28, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 30, 2024

zandercymatics Aug 30, 2024 •

edited

Loading

abroddrick Sep 4, 2024

zandercymatics Sep 4, 2024

abroddrick Sep 6, 2024

zandercymatics commented Sep 6, 2024 •

edited

Loading

zandercymatics commented Sep 6, 2024 •

edited

Loading

github-actions bot commented Sep 10, 2024

github-actions bot commented Sep 10, 2024

github-actions bot commented Sep 11, 2024

github-actions bot commented Sep 11, 2024

github-actions bot commented Sep 11, 2024

Matt-Spence left a comment

github-actions bot commented Sep 12, 2024

github-actions bot commented Sep 12, 2024

github-actions bot commented Sep 12, 2024

github-actions bot commented Sep 12, 2024

github-actions bot commented Sep 12, 2024

Ticket #2616: populate suborg and portfolio script #2670

Ticket #2616: populate suborg and portfolio script #2670

Conversation

zandercymatics commented Aug 28, 2024 • edited Loading

Ticket

Changes

Context for reviewers

Background

Other information

Setup

Code Review Verification Steps

As the original developer, I have

Satisfied acceptance criteria and met development standards

Ensured code standards are met (Original Developer)

Validated user-facing changes (if applicable)

As a code reviewer, I have

Reviewed, tested, and left feedback about the changes

Ensured code standards are met (Code reviewer)

Validated user-facing changes as a developer

As a designer reviewer, I have

Verified that the changes match the design intention

Validated user-facing changes as a designer

Screenshots

github-actions bot commented Aug 28, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Aug 30, 2024

zandercymatics Aug 30, 2024 • edited Loading

Choose a reason for hiding this comment

abroddrick Sep 4, 2024

Choose a reason for hiding this comment

zandercymatics Sep 4, 2024

Choose a reason for hiding this comment

abroddrick Sep 6, 2024

Choose a reason for hiding this comment

zandercymatics commented Sep 6, 2024 • edited Loading

zandercymatics commented Sep 6, 2024 • edited Loading

github-actions bot commented Sep 10, 2024

github-actions bot commented Sep 10, 2024

github-actions bot commented Sep 11, 2024

github-actions bot commented Sep 11, 2024

github-actions bot commented Sep 11, 2024

Matt-Spence left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 12, 2024

github-actions bot commented Sep 12, 2024

github-actions bot commented Sep 12, 2024

github-actions bot commented Sep 12, 2024

github-actions bot commented Sep 12, 2024

zandercymatics commented Aug 28, 2024 •

edited

Loading

zandercymatics Aug 30, 2024 •

edited

Loading

zandercymatics commented Sep 6, 2024 •

edited

Loading

zandercymatics commented Sep 6, 2024 •

edited

Loading