Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: reuse existing row on dual write if available #20661

Merged
merged 1 commit into from
Jul 16, 2022

Conversation

eschutho
Copy link
Member

@eschutho eschutho commented Jul 9, 2022

SUMMARY

We're seeing some integrity errors

psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "sl_columns_pkey"
DETAIL:  Key (id)=(x) already exists.

on import.
To avoid a duplicate primary key violation, I'm copying the existing row from the database instead of copying just the id and trying to use Sqlalchemy to merge the data since the add/merge functionality isn't in this module.

TESTING INSTRUCTIONS

This is an intermittent error and difficult to reproduce but happens for some people on import.

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags:
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

@eschutho eschutho marked this pull request as draft July 9, 2022 00:35
@eschutho eschutho marked this pull request as ready for review July 11, 2022 16:29
@eschutho eschutho force-pushed the elizabeth/no-autoflush-import branch from a1238b0 to 3b59914 Compare July 15, 2022 18:27
@pull-request-size pull-request-size bot added size/L and removed size/S labels Jul 15, 2022
@eschutho eschutho force-pushed the elizabeth/no-autoflush-import branch from 3b59914 to 84910b6 Compare July 15, 2022 18:49
@codecov
Copy link

codecov bot commented Jul 15, 2022

Codecov Report

Merging #20661 (84910b6) into master (b39a3d8) will increase coverage by 0.03%.
The diff coverage is 87.50%.

@@            Coverage Diff             @@
##           master   #20661      +/-   ##
==========================================
+ Coverage   66.82%   66.86%   +0.03%     
==========================================
  Files        1752     1752              
  Lines       65616    65574      -42     
  Branches     6938     6938              
==========================================
- Hits        43849    43845       -4     
+ Misses      20007    19969      -38     
  Partials     1760     1760              
Flag Coverage Δ
hive 53.87% <62.50%> (+0.05%) ⬆️
mysql 82.47% <87.50%> (+0.08%) ⬆️
postgres 82.54% <87.50%> (+0.08%) ⬆️
presto 53.73% <62.50%> (+0.05%) ⬆️
python 82.98% <87.50%> (+0.09%) ⬆️
sqlite 82.35% <87.50%> (+0.11%) ⬆️
unit ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
superset/connectors/sqla/models.py 90.33% <87.50%> (+0.01%) ⬆️
superset/initialization/__init__.py 89.93% <0.00%> (-1.74%) ⬇️
superset/databases/commands/validate.py 74.13% <0.00%> (-1.67%) ⬇️
superset/utils/log.py 93.70% <0.00%> (-1.40%) ⬇️
superset/views/base.py 75.36% <0.00%> (-1.36%) ⬇️
...t/annotation_layers/annotations/commands/delete.py 84.00% <0.00%> (-1.19%) ⬇️
superset/dashboards/permalink/commands/get.py 81.25% <0.00%> (-1.11%) ⬇️
superset/annotation_layers/commands/bulk_delete.py 84.61% <0.00%> (-1.10%) ⬇️
superset/css_templates/commands/bulk_delete.py 87.50% <0.00%> (-0.97%) ⬇️
...rset/queries/saved_queries/commands/bulk_delete.py 87.50% <0.00%> (-0.97%) ⬇️
... and 91 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b39a3d8...84910b6. Read the comment docs.

@eschutho eschutho changed the title fix: do not autoflush on import query fix: reuse existing row on dual write if available Jul 15, 2022
Copy link
Member

@betodealmeida betodealmeida left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable... I hope we can finalize the migration and get rid of this code soon-ish.

@eschutho eschutho merged commit bbaa4cc into apache:master Jul 16, 2022
rdubois pushed a commit to ftom/superset that referenced this pull request Sep 3, 2022
@eschutho eschutho deleted the elizabeth/no-autoflush-import branch October 21, 2022 21:00
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 2.1.0 labels Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels size/L 🚢 2.1.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants