Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add table/instance filtering to speed up debugging process. #148

Merged
merged 3 commits into from
Sep 27, 2023

Conversation

jdangerx
Copy link
Member

This makes it so you can do something like:

> xbrl_extract $PUDL_INPUT/2021/ferc1-xbrl-2021.zip foo.sqlite --instance-pattern "PJM" --requested-tables "electric_operations_and_maintenance_expenses_320_duration" "identification_001_duration" --metadata-path foo-md.json

And take a look-see at what the extractor is putting out, as well as stuff breakpoints in places without having to add conditional breakpoints like "if the table is blah and the instance is blah, then breakpoint()".

Some timings just for fun:

> time xbrl_extract $PUDL_INPUT/2021/ferc1-xbrl-2021.zip foo.sqlite --instance-pattern "PJM" --requested-tables "electric_operations_and_maintenance_expenses_320_duration" "identification_001_duration" --metadata-path foo-md.json
2023-09-27 13:01:52 [    INFO] catalystcoop.ferc_xbrl_extractor.xbrl:265 Parsing taxonomy from https://eCollection.ferc.gov/taxonomy/form1/2022-01-01/form/form1/form-1_2022-01-01.xsd
2023-09-27 13:01:56 [    INFO] catalystcoop.ferc_xbrl_extractor.xbrl:149 Finished batch 1/1

________________________________________________________
Executed in    5.89 secs    fish           external
   usr time    7.20 secs    0.31 millis    7.20 secs
   sys time    2.07 secs    1.75 millis    2.07 secs

@codecov
Copy link

codecov bot commented Sep 27, 2023

Codecov Report

All modified lines are covered by tests ✅

Comparison is base (61525e5) 93.09% compared to head (2ca562e) 93.29%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #148      +/-   ##
==========================================
+ Coverage   93.09%   93.29%   +0.20%     
==========================================
  Files           8        8              
  Lines         594      597       +3     
==========================================
+ Hits          553      557       +4     
+ Misses         41       40       -1     
Files Coverage Δ
src/ferc_xbrl_extractor/cli.py 94.54% <100.00%> (+0.20%) ⬆️
src/ferc_xbrl_extractor/datapackage.py 99.35% <100.00%> (+0.64%) ⬆️
src/ferc_xbrl_extractor/xbrl.py 91.56% <100.00%> (+0.10%) ⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@zschira zschira left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good and seems pretty handy

@jdangerx
Copy link
Member Author

Sort of seems like I probably broke some of this behavior that used to exist, since there was some old logic to hook into. But I guess it's fixed now! I'm gonna add a test to make codecov happy and then merge.

@jdangerx jdangerx merged commit 2ddcc8c into main Sep 27, 2023
14 checks passed
@jdangerx jdangerx deleted the speedy-development branch September 27, 2023 22:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants