Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: memoize primitives #19930

Merged
merged 1 commit into from
May 2, 2022

Conversation

betodealmeida
Copy link
Member

SUMMARY

In some of the views we're caching DatasourceName namedtuples to Redis using the memoized_func decorator. We found that in certain cases the object that comes back from the cache is a list of [table, schema], instead of DatasourceName(table=table, schema=schema), which breaks the view.

I changed the code in models/core.py so that the return value from the memoized functions is a primitive (a tuple of (table, schema)), and we rebuild the namedtuple in views/core.py, to prevent this bug from happening.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

N/A

TESTING INSTRUCTIONS

Updated unit tests.

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

@codecov
Copy link

codecov bot commented May 2, 2022

Codecov Report

Merging #19930 (a7ce679) into master (7b3d0f0) will decrease coverage by 0.02%.
The diff coverage is 50.00%.

@@            Coverage Diff             @@
##           master   #19930      +/-   ##
==========================================
- Coverage   66.53%   66.51%   -0.03%     
==========================================
  Files        1715     1715              
  Lines       65060    65060              
  Branches     6723     6723              
==========================================
- Hits        43289    43274      -15     
- Misses      20060    20075      +15     
  Partials     1711     1711              
Flag Coverage Δ
hive 52.90% <0.00%> (ø)
mysql 81.94% <50.00%> (ø)
postgres 81.97% <50.00%> (-0.02%) ⬇️
presto 52.75% <0.00%> (ø)
python 82.36% <50.00%> (-0.05%) ⬇️
sqlite ?
unit 48.03% <16.66%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
superset/db_engine_specs/base.py 88.01% <0.00%> (-0.34%) ⬇️
superset/utils/cache.py 75.00% <ø> (ø)
superset/models/core.py 89.33% <50.00%> (ø)
superset/views/core.py 77.06% <50.00%> (ø)
superset/db_engine_specs/sqlite.py 91.89% <100.00%> (-5.41%) ⬇️
superset/reports/commands/log_prune.py 85.71% <0.00%> (-3.58%) ⬇️
superset/utils/celery.py 86.20% <0.00%> (-3.45%) ⬇️
superset/connectors/sqla/utils.py 88.75% <0.00%> (-2.50%) ⬇️
superset/result_set.py 96.77% <0.00%> (-1.62%) ⬇️
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7b3d0f0...a7ce679. Read the comment docs.

@betodealmeida betodealmeida merged commit 1ebdaac into apache:master May 2, 2022
In the example above the result for `1+2` will be stored under the key of name "1+2",
in the `cache_manager.data_cache` cache.

Note: this decorator should be used only with functions that return primitives,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yey!

sadpandajoe pushed a commit to preset-io/superset that referenced this pull request May 2, 2022
@sadpandajoe
Copy link
Member

🏷️ preset:2022.17

schema=schema,
force=True,
cache=database.table_cache_enabled,
cache_timeout=database.table_cache_timeout,
Copy link
Member

@eschutho eschutho May 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@beto what would happen if this value returns from cache prior to this change and it's not a list of strings but rather the DatasourceName object?

Copy link
Member Author

@betodealmeida betodealmeida May 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great question, the cool thing about this solution is that it still works with old cached values:

>>> from typing import NamedTuple
>>>
>>> class DatasourceName(NamedTuple):
...     table: str
...     schema: str
...
>>> a = DatasourceName(table='t', schema='s')
>>> DatasourceName(*a)
DaasourceName(table='t', schema='s')

hughhhh pushed a commit to hve-labs/superset that referenced this pull request May 11, 2022
philipher29 pushed a commit to ValtechMobility/superset that referenced this pull request Jun 9, 2022
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 2.0.0 labels Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels preset:2022.17 size/L 🚢 2.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants