Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

backup/restore: handle external storage error gracefully #49942

Closed
Leavrth opened this issue Dec 30, 2023 · 5 comments · Fixed by #50541
Closed

backup/restore: handle external storage error gracefully #49942

Leavrth opened this issue Dec 30, 2023 · 5 comments · Fixed by #50541
Labels
affects-6.5 affects-7.1 affects-7.5 component/br This issue is related to BR of TiDB. report/customer Customers have encountered this bug. severity/moderate type/bug The issue is confirmed as a bug.

Comments

@Leavrth
Copy link
Contributor

Leavrth commented Dec 30, 2023

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. backup/restore/stream

2. What did you expect to see? (Required)

success

3. What did you see instead (Required)

failed for external storage error

failed to put object rusoto error timeout after 15mins for upload in s3 storage

https://github.com/pingcap/tidb/blob/d6aeaf043f6d0b4f5718321e799b0a25792e4562/br/pkg/backup/client.go#L1446

read: connection reset by peer

failed when read the response.body in s3 ReadFile

4. What is your TiDB version? (Required)

master
@Leavrth Leavrth added type/bug The issue is confirmed as a bug. severity/moderate component/br This issue is related to BR of TiDB. labels Dec 30, 2023
@Leavrth
Copy link
Contributor Author

Leavrth commented Dec 30, 2023

The same issue
#49462

@Leavrth Leavrth mentioned this issue Jan 18, 2024
Merged
13 tasks
@BornChanger
Copy link
Contributor

/label affects-6.5

@BornChanger
Copy link
Contributor

/label affects-7.1

@BornChanger
Copy link
Contributor

/label affects-7.5

ti-chi-bot bot pushed a commit that referenced this issue Jan 24, 2024
@Leavrth
br: add more retry strategy (s3.ReadFile: body reader / pushBackup: b…
d6ef1c7 
…ackoffer) (#50541)

close #49942
This was referenced Jan 24, 2024
Merged
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Jan 24, 2024
@ti-chi-bot
br: add more retry strategy: s3.ReadFile: body reader (#50541) (#50706)
c12144a 
close #49942
ti-chi-bot bot pushed a commit that referenced this issue Feb 4, 2024
@ti-chi-bot
br: add more retry strategy (s3.ReadFile: body reader / pushBackup: b…
ae6bd23 
…ackoffer) (#50541) (#50708)

close #49942
ti-chi-bot bot pushed a commit that referenced this issue Feb 27, 2024
@ti-chi-bot
br: add more retry strategy (s3.ReadFile: body reader / pushBackup: b…
619708c 
…ackoffer) (#50541) (#50707)

close #49942
@seiya-annie
Copy link

/found customer

@ti-chi-bot ti-chi-bot bot added the report/customer Customers have encountered this bug. label Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-6.5 affects-7.1 affects-7.5 component/br This issue is related to BR of TiDB. report/customer Customers have encountered this bug. severity/moderate type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

br: add more retry strategy (s3.ReadFile: body reader / pushBackup: backoffer) Leavrth/tidb
3 participants
@Leavrth @seiya-annie @BornChanger