Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Total axes #118

Merged
merged 11 commits into from
Aug 18, 2023
Merged

Total axes #118

merged 11 commits into from
Aug 18, 2023

Conversation

zschira
Copy link
Member

@zschira zschira commented Aug 17, 2023

This PR changes the extractor behavior so missing Axes will be treated as Totals.

@codecov
Copy link

codecov bot commented Aug 17, 2023

Codecov Report

Patch coverage: 100.00% and project coverage change: +11.46% 🎉

Comparison is base (0572a99) 77.25% compared to head (19ae97c) 88.72%.

Additional details and impacted files
@@             Coverage Diff             @@
##             main     #118       +/-   ##
===========================================
+ Coverage   77.25%   88.72%   +11.46%     
===========================================
  Files           8        8               
  Lines         554      541       -13     
===========================================
+ Hits          428      480       +52     
+ Misses        126       61       -65     
Files Changed Coverage Δ
src/ferc_xbrl_extractor/arelle_interface.py 100.00% <100.00%> (ø)
src/ferc_xbrl_extractor/cli.py 77.27% <100.00%> (ø)
src/ferc_xbrl_extractor/datapackage.py 98.52% <100.00%> (+10.41%) ⬆️
src/ferc_xbrl_extractor/instance.py 97.43% <100.00%> (+40.52%) ⬆️
src/ferc_xbrl_extractor/taxonomy.py 86.20% <100.00%> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@zschira zschira requested a review from jdangerx August 17, 2023 21:06
@zschira zschira marked this pull request as draft August 17, 2023 21:57
@zschira zschira marked this pull request as ready for review August 18, 2023 15:37
Copy link
Member

@jdangerx jdangerx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only actually blocking thing is the question about whether dropna() is doing what we want it to do. But I would feel warm and fuzzy inside if you engaged with my various suggestions, too!

Broader questions I wanted to reiterate at the top level:

  • do we need to filter by context dimensions in get_facts? it might make sense to just grab all the facts. then we can filter contexts by whether or not each context fits with the schema.primary_key, and avoid passing extra state around.
  • should we consider not smearing the concept names into their own columns? I'm sure there was some sort of decision made about this in the ancient times, but what was the rationale there?
  • should we have the output dataframes be indexed by schema.primary_key?

Finally, when I was testing the code I'm suggesting, I found a couple weird things that I needed to handle:

  • empty files, obviously
  • filings with duplicated ("c_id", "name") values in facts... that seemed surprising.

src/ferc_xbrl_extractor/instance.py Show resolved Hide resolved
src/ferc_xbrl_extractor/instance.py Outdated Show resolved Hide resolved
tests/integration/datapackage_test.py Show resolved Hide resolved
src/ferc_xbrl_extractor/datapackage.py Outdated Show resolved Hide resolved
src/ferc_xbrl_extractor/datapackage.py Show resolved Hide resolved
src/ferc_xbrl_extractor/datapackage.py Outdated Show resolved Hide resolved
src/ferc_xbrl_extractor/datapackage.py Outdated Show resolved Hide resolved
src/ferc_xbrl_extractor/instance.py Show resolved Hide resolved
@zschira zschira merged commit f37907f into main Aug 18, 2023
1 of 6 checks passed
@zschira zschira deleted the total_axes branch August 18, 2023 20:46
@jdangerx jdangerx mentioned this pull request Aug 21, 2023
@zschira zschira restored the total_axes branch August 23, 2023 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

FERC Form 1 Statement of Income Contains Holes
2 participants