Make self-hosted backup/restore work excellently

[hubertdeng123]

Currently, the backup/restore workflow experience is not great, and users are unhappy about it. Backup/restore is best practices for upgrading self-hosted versions, and is particularly important when the postgres version or our DB models change. It is also used for migrating from self-hosted to SaaS. The issue about backup/restore being broken is the most commented open source issue in the Sentry repo, and has as many unique commenters as the most commented issues in the self-hosted repo.

Backup/restore is also important for users that want to migrate to SaaS. To make this experience better, we should

Self-hosted issues:

1. Fix backup/restore flow sentry#36868
2. sentry_environment is not backed up and breaks alerts logic self-hosted#1965
3. DB duplicate key violation errors during backup/restore self-hosted#1972

SaaS related concerns:

1. Make sure the documented workflow for migrating works
2. See if we can make the self-hosted to SaaS migrations we handle easier for ourselves

Future possibilities enabled from this ticket:

1. Migrate data from a single org deployment of Sentry to a multi-org one self-hosted#1403
2. One-click SaaS migration
3. Migration from one region to another (e.g. US -> EU)

Task list:

1. Try following SaaS migration docs and self-hosted backup/restore docs and see what breaks. Experiment with changing settings to work different parts of what is backed up.
2. Investigate and fix Deserialization errors during backup/restore
3. Investigate and fix db integrity errors during backup/restore
4. Write Sentry tests to cover what is currently broken with documented workflows.
5. Fix backup restore to pass new Sentry tests
6. Remove headers from backup/restore  self-hosted#1995
7. Add integration tests for backup/restore workflow self-hosted#1918
8. Dogfood backup/restore self-hosted#1986
9. vanguard post

┆Issue is synchronized with this Jira Epic by Unito

[hubertdeng123]

So far, there are three main issues with backup/restore that I am aware of.

1. Backup file includes garbage headers
2. Deserialization errors during import
3. Unique key violation during import

[hubertdeng123]

Weekly Update 2/13

• merge fix(backup/restore): Add TypeError handling for de-pickling PickledObjectField sentry#44328
• investigate unique key violation error
• fix sentry_environment is not backed up and breaks alerts logic self-hosted#1965

[hubertdeng123]

Weekly Update 2/21

• merge fix(backup/restore): Warn users of integrity violations during backup/restore sentry#44705
• investigate foreign key violation error

[hubertdeng123]

Weekly Update 2/27

• Remove headers in backup file
• Put up draft PR for integration test for backup/restore in self-hosted

Ended up not wrapping up PR for integration tests due to pivoting between other tasks

[hubertdeng123]

Weekly Update 3/6

• Merge Add backup/restore integration tests self-hosted#2012
• put draft PR up Add opt in error monitoring to reset and clean scripts self-hosted#2021

[hubertdeng123]

Weekly Update 3/13

• merge Add opt in error monitoring to reset and clean scripts self-hosted#2021
• merge Add Backup/restore scripts self-hosted#2029

[hubertdeng123]

Weekly Update 3/20

• [ ]Tasks for this Epic are now finished. Will do one final team review before closing
• vanguard post

[hubertdeng123]

shipped vanguard post! closing this now :)

[chadwhitacre]

ftr https://vanguard.getsentry.net/p/clfk7tmpe000es60l743x5cuy 🔒