Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Opening a jwst file with many (~550) extensions (level 2 output for nirspec mos mode) take ~9 seconds (when
skip_fits_update=True
). Opening the same file withskip_fits_update=False
takes longer, ~26 seconds. Surprisingly, a large portion of the added time is spent in calls toget_hdu
.Using
cProfile
andskip_fits_update=False
opening now takes ~60 seconds (due to profiling overhead) and the rendered profile (with snakeviz) is as follows:zooming into
_load_from_schema
reveals ~40 seconds inget_hdu
This PR adds a
hdu_cache
to skip repeated indexing the hdulist (which in some conditions for this file takes 2-3 ms). With this PR opening the same file withskip_fits_update=False
(and no profiling) takes 12 seconds and withskip_fits_update=True
still takes 9 seconds (most of this is spent inasdf.open
as the tree is quite large). RunningcProfile
withskip_fits_update=False
takes 24 seconds and zooming in toget_hdu
reveals 5 seconds spent inget_hdu
(20% down from 66% without this PR):Checklist
CHANGES.rst
(either inBug Fixes
orChanges to API
)