Skip to content
This repository has been archived by the owner on Jul 24, 2024. It is now read-only.

Add a flag to control whether to backup/restore SEQUENCEs #305

Open
kennytm opened this issue May 25, 2020 · 0 comments
Open

Add a flag to control whether to backup/restore SEQUENCEs #305

kennytm opened this issue May 25, 2020 · 0 comments
Labels
difficulty/1-easy Easy issue Priority/P0 Top priority issue. Must have an associated milestone question Further information is requested type/feature-request New feature or request

Comments

@kennytm
Copy link
Collaborator

kennytm commented May 25, 2020

Feature Request

Describe your feature request related problem:

After #242, we will unconditionally backup and restore sequences. However, sequence support is still considered somewhat buggy, so we'd like to add a switch to control whether to backup and/or restore sequences.

Problems involving sequence backup & restore:

  • The backed up sequence skips all cached values. For example, even if a TiDB server only used up to ID = 10, it may have already allocated up to ID = 1000 in its cache, so the backed up sequence will have ID = 1001. The 900 numbers are wasted. Note that the same behavior will manifest if we use multiple TiDB servers.
  • The backed up sequence value does not respect snapshot, so the backed up value may be even higher.
  • Cycles are not yet properly supported.

Note that sequences will never "decrease", so there is no risk over-valuating a sequence other than wasting numbers.

Describe the feature you'd like:

  1. Add a switch to disable sequence backup (???)
  2. Add a switch to disable sequence restore

If a sequence is not restored, tables depending on that sequence will still be successfully restored, but INSERT operations will fail. This is expected behavior.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

@kennytm kennytm added the type/feature-request New feature or request label May 25, 2020
kennytm added a commit to kennytm/br that referenced this issue May 27, 2020
* restore: fix alter auto increment id for no-primary-key table (pingcap#139)

* restore: fix alter auto increment id for no-primary-key table

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add integration test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* *: use oracle package to manipulate ts and test gc safe point  (pingcap#121)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* cmd: convert version command to flags (pingcap#144)

* cmd: convert version command to flags

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: add changelog and github templates (pingcap#143)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: merge tidb-tools/pkg/restore-util (pingcap#146)

* restore-util: Implement split/scatter (pingcap#274)

* implement split/scatter

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* init test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* redesign output/input of the lib

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update dependency

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add commments and more tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add ScanRegions interface to Client

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix potential data race

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* Update pkg/restore-util/client.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update dependency

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* resolve conflicts

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix prefix rewrite

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add RewriteRule/skip failed scatter region/retry the SplitRegion

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check if region has peer

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* more logs

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add split retry interval (pingcap#277)

* reset dependencies to release-3.1

* add split retry interval

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix go.sum

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: wait for scatter region sequentially  (pingcap#279)

* wait for scatter region sequentially

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add on split hook (pingcap#281)

* restore-util: add on split hook

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Nil check onSplit

Co-Authored-By: kennytm <kennytm@gmail.com>

* restore-util: fix returned new region is nil (pingcap#283)

* restore-util: fix returned new region is nil

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* more logs

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* *: gofmt

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: call onSplit on splitByRewriteRules (pingcap#285)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore-util: fix overlapped error message (pingcap#293)

* restore-util: fix overlapped error message

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix log message

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* reduce error trace

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: log warning when cannot find matched rewrite rule (pingcap#299)

* restore-util: add method to set placement rules and store labels (pingcap#301)

* restore-util: add method to set placement rules and store labels

Signed-off-by: disksing <i@disksing.com>

* minor fix

Signed-off-by: disksing <i@disksing.com>

* address comment

Signed-off-by: disksing <i@disksing.com>

* add GetPlacementRules

Signed-off-by: disksing <i@disksing.com>

* fix test

Signed-off-by: disksing <i@disksing.com>

* restore-util: support batch split (pingcap#300)

* restore-util: support batch split

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* go fmt

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* address commits

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update pkg/restore-util/split.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* add onSplit callback

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add upper bound time for waiting for scatter (pingcap#305)

* restore: fix scatter regions failed

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* stop waiting for scatter after 3min

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: fix wrong url (pingcap#306)

Signed-off-by: disksing <i@disksing.com>

* restore-util: add warning about unmatched table id (pingcap#313)

* restore-util: support table partition

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* warn table id does not match

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add unit tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: Neil Shen <overvenus@gmail.com>

* fix compile error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Ian <ArGregoryIan@gmail.com>
Co-authored-by: Neil Shen <overvenus@gmail.com>

* *: prune tidb-tools

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: address linters suggestions

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: merge restoreutil into restore

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comment

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com>
Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: disksing <i@disksing.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>

* Fixed handling for a dbName that do not exist in the backup being restored (pingcap#148)

* Fixed handling for a dbName that do not exist in the backup being restored

* Fixed handling for a dbName that do not exist in the backup being restored

* validate: fix debug meta test ci (pingcap#153)

* validate: fix debug meta test ci

* *: extracts runBackup/runRestore in cmd into pkg/task (pingcap#156)

* *: extracts runBackup/runRestore in cmd into pkg/task

Defines a "Config" structure to store the parsed flags.

Use the "black-white-list" structure to define what tables/databases to
backup/restore.

* go.mod: update tidb to v4.0.0-beta

* restore: fix restore summary log (pingcap#150)

Co-authored-by: kennytm <kennytm@gmail.com>

* restore: enhance error handling (pingcap#152)

* restore: enhance error handling

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* unit test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix region epoch error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* remove `Restore*`

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address lint

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add debug log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* Update pkg/restore/import.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix retry error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* handle RegionNotFound error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: kennytm <kennytm@gmail.com>

Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com>
Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: disksing <i@disksing.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>
Co-authored-by: Kolbe Kegel <kolbe@kolbekegel.com>
kennytm added a commit to kennytm/br that referenced this issue May 27, 2020
* restore: fix alter auto increment id for no-primary-key table (pingcap#139)

* restore: fix alter auto increment id for no-primary-key table

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add integration test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* *: use oracle package to manipulate ts and test gc safe point  (pingcap#121)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* cmd: convert version command to flags (pingcap#144)

* cmd: convert version command to flags

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: add changelog and github templates (pingcap#143)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: merge tidb-tools/pkg/restore-util (pingcap#146)

* restore-util: Implement split/scatter (pingcap#274)

* implement split/scatter

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* init test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* redesign output/input of the lib

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update dependency

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add commments and more tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add ScanRegions interface to Client

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix potential data race

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* Update pkg/restore-util/client.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update dependency

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* resolve conflicts

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix prefix rewrite

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add RewriteRule/skip failed scatter region/retry the SplitRegion

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check if region has peer

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* more logs

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add split retry interval (pingcap#277)

* reset dependencies to release-3.1

* add split retry interval

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix go.sum

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: wait for scatter region sequentially  (pingcap#279)

* wait for scatter region sequentially

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add on split hook (pingcap#281)

* restore-util: add on split hook

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Nil check onSplit

Co-Authored-By: kennytm <kennytm@gmail.com>

* restore-util: fix returned new region is nil (pingcap#283)

* restore-util: fix returned new region is nil

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* more logs

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* *: gofmt

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: call onSplit on splitByRewriteRules (pingcap#285)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore-util: fix overlapped error message (pingcap#293)

* restore-util: fix overlapped error message

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix log message

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* reduce error trace

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: log warning when cannot find matched rewrite rule (pingcap#299)

* restore-util: add method to set placement rules and store labels (pingcap#301)

* restore-util: add method to set placement rules and store labels

Signed-off-by: disksing <i@disksing.com>

* minor fix

Signed-off-by: disksing <i@disksing.com>

* address comment

Signed-off-by: disksing <i@disksing.com>

* add GetPlacementRules

Signed-off-by: disksing <i@disksing.com>

* fix test

Signed-off-by: disksing <i@disksing.com>

* restore-util: support batch split (pingcap#300)

* restore-util: support batch split

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* go fmt

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* address commits

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update pkg/restore-util/split.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* add onSplit callback

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add upper bound time for waiting for scatter (pingcap#305)

* restore: fix scatter regions failed

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* stop waiting for scatter after 3min

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: fix wrong url (pingcap#306)

Signed-off-by: disksing <i@disksing.com>

* restore-util: add warning about unmatched table id (pingcap#313)

* restore-util: support table partition

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* warn table id does not match

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add unit tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: Neil Shen <overvenus@gmail.com>

* fix compile error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Ian <ArGregoryIan@gmail.com>
Co-authored-by: Neil Shen <overvenus@gmail.com>

* *: prune tidb-tools

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: address linters suggestions

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: merge restoreutil into restore

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comment

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com>
Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: disksing <i@disksing.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>

* Fixed handling for a dbName that do not exist in the backup being restored (pingcap#148)

* Fixed handling for a dbName that do not exist in the backup being restored

* Fixed handling for a dbName that do not exist in the backup being restored

* validate: fix debug meta test ci (pingcap#153)

* validate: fix debug meta test ci

* *: extracts runBackup/runRestore in cmd into pkg/task (pingcap#156)

* *: extracts runBackup/runRestore in cmd into pkg/task

Defines a "Config" structure to store the parsed flags.

Use the "black-white-list" structure to define what tables/databases to
backup/restore.

* go.mod: update tidb to v4.0.0-beta

* restore: fix restore summary log (pingcap#150)

Co-authored-by: kennytm <kennytm@gmail.com>

* restore: enhance error handling (pingcap#152)

* restore: enhance error handling

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* unit test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix region epoch error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* remove `Restore*`

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address lint

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add debug log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* Update pkg/restore/import.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix retry error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* handle RegionNotFound error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: kennytm <kennytm@gmail.com>

* Incremental BR: support DDL (pingcap#155)

* support backup&restore ddl

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* integration tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update kvproto

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix integration tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* reduce cyclomatic complexity of `runRestore`

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add unit test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* disable fast checksum in incremental br

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix no valid key error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address lint

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Reduce TiDB dependencies (pingcap#158)

* utils: exclude mock_cluster outside of unit test

* utils: remove unused ResultSetToStringSlice()

* *: abstract away dependencies of tidb/session into a Glue interface

* *: fix hound lint

* util,mock: move utils.MockCluster to mock.Cluster

* restore: fix test build failure

Co-authored-by: 3pointer <qdlc2010@gmail.com>

* go.mod: update tidb (pingcap#168)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* BR support TLS (pingcap#161)

* *: support tls

* move tikv.driver to glue

* fix comments

* upgrade golangci and prepare for go 1.14 (pingcap#171)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* backup: add raw backup command (pingcap#101)

* backup: add raw backup command

* restore: speed up retry on not leader (pingcap#179)

* tests: stable cluster start up

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: fix unbound var

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: speed retry on not leader

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: add --cacert flag

Signed-off-by: Neil Shen <overvenus@gmail.com>

* make codecov green

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* conn, restore: paginate scan regions (pingcap#165)

* conn, restore: paginate scan regions

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: large timeout

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Batch restore (pingcap#167)

* *: unify Range and RangeTree

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: split restore files into small batch

Signed-off-by: Neil Shen <overvenus@gmail.com>

* task: set default restore concurrency to 128

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: unused table worker pool

Signed-off-by: Neil Shen <overvenus@gmail.com>

* summary: sum up repeated duration and int

Signed-off-by: Neil Shen <overvenus@gmail.com>

* rtree: move rtree from utils to pkg

Signed-off-by: Neil Shen <overvenus@gmail.com>

* README, docker: add quick start (pingcap#181)

* README, docker: add quick start

Signed-off-by: Neil Shen <overvenus@gmail.com>

* cmd: disable some TiDB log

Signed-off-by: Neil Shen <overvenus@gmail.com>

* docker: build go-ycsb automatically

Signed-off-by: Neil Shen <overvenus@gmail.com>

* cmd: add TODO about TiDB logs

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: update tidb dependency build with go1.14 (pingcap#176)

* *: add license header (pingcap#182)

* rtree: move checkFile into backup

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: add license header

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Update LICENSE.md

Co-Authored-By: kennytm <kennytm@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>

* conn: support not shutting down the storage when closing the connection (pingcap#185)

Co-authored-by: 3pointer <luancheng@pingcap.com>

* conn: use GetDomain to avoid some TiDB breaking changes (pingcap#186)

* conn: use GetDomain to avoid some TiDB breaking changes

Signed-off-by: Neil Shen <overvenus@gmail.com>

* minor usability improvement

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>

* fix check safepoint & unhide experimental features (pingcap#175)

* backup: check safepoint for last backup ts

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check lastbackupts > 0

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* unhide experimental features

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update tests/br_z_gc_safepoint/run.sh

Co-Authored-By: kennytm <kennytm@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>

* support backupts (pingcap#172)

* support backupts

* address comment

* address comment

* fix space

* *: update pd deps to v4 (pingcap#184)

Co-authored-by: 3pointer <luancheng@pingcap.com>

* restore: support online restore (pingcap#114)

Signed-off-by: disksing <i@disksing.com>

* metrics: add grafana scripts (pingcap#140)

* add grafana scripts

* fix

Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: glorv <glorvs@163.com>
Co-authored-by: kennytm <kennytm@gmail.com>

* filter out all TiFlash nodes when retrieving lists of stores from PD (pingcap#187)

* conn: ignore nodes with label engine=tiflash

* conn: disallow TiFlash on restore, only skip TiFlash on backup

* Create integration test for S3 storage (pingcap#174)

* Fix summary log (pingcap#191)

* *: fix restore summary log after restore logic changed to files

* fix

* fix

* fix

Co-authored-by: kennytm <kennytm@gmail.com>

* Implement Raw Restore (pingcap#104)

* Update kvproto

* Implement raw restore

* fix build

* Set range for file importer

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Remove unnecessary comments

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* check cf and support multi ranges in BackupMeta

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Check files' cf; address comments

* adjust structure to keep consistent with master

* Fix build

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix build and make check, avoid accessing TiDB in rawkv mode

* Fix test

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix tests

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix broken logic after merging master

* Update pkg/task/restore_raw.go

Co-Authored-By: Neil Shen <overvenus@gmail.com>

* Address comments

* Address comments

* Mark raw restore as experimental

* Fix build

* Address comments

* test: Add check for deleting data and partial backup

* Fix build

* Add license header

* fix ci

* fix ci

Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com>

* update dep

* restore: remove tiflash replica before restore (pingcap#194)

* restore: remove tiflash replica before restore

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* rename errSplit variable

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check replica count by region info

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* cleanup

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* save tiflash replica count to backupmeta

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix save crcxor

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix decode the key of placement rule

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address lint

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* close domain after restoring tiflash-replica

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update pkg/task/restore.go

Co-Authored-By: 3pointer <qdlc2010@gmail.com>

Co-authored-by: 3pointer <qdlc2010@gmail.com>

* summary: put summary log at last (pingcap#197)

* summary: put summary log at last

* fix switch sql

* *: abstract the progress channel (updateCh) into the glue package (pingcap#196)

* *: abstract the progress channel (updateCh) into the glue package

* restore: fix crash in truncateTS() when the bound is unlimited

* task: fix comment

Co-authored-by: Ian <ArGregoryIan@gmail.com>

* *: refline logs (pingcap#189)

* adapt pd/tidb related 3.1 changes

* tests: disable TLS test (pingcap#204)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: add S3 quick start and few enhancement of log (pingcap#202)

* README, docker: add quick start about S3 storage

Signed-off-by: Neil Shen <overvenus@gmail.com>

* pkg/summary: make sure to output correct summary

Signed-off-by: Neil Shen <overvenus@gmail.com>

* cmd, tests: log to terminal if BR_LOG_TO_TERM is set

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Update pkg/task/common.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: cat log if br fails

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>

* restore: add error field to `DownloadResponse` (pingcap#195)

* restore: add error field to `DownloadResponse`

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore: populate restore cancel error (pingcap#207)

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>

* enhance usability of br (pingcap#208)

* silenceUsage only when parse cmd flags failed

* udpate tidb

Co-authored-by: kennytm <kennytm@gmail.com>

* task: do not run checksum if restore failed (pingcap#209)

* fix incremental bug in llroad test (pingcap#199)

* restore: filter same table ddl

* *: do not return error when backup/restore data is empty

* fix create database double during incremental restore

* add tests

* fix ci

* address comment

* add skip create sqls (pingcap#211)

* Revert "tests: disable TLS test (pingcap#204)" (pingcap#218)

This reverts commit e168a60.

* doc: add `minio` to dependence list. (pingcap#221)

The README of test omitted `minio` in the dependence list, which is needed for run the integration test.

Co-authored-by: Neil Shen <overvenus@gmail.com>

* move waiting reject stores in import file (pingcap#222)

* move wait rejectstores into import files

* restore: use new table id to search placementRules

* Update pkg/restore/import.go

Co-Authored-By: Neil Shen <overvenus@gmail.com>

* Update pkg/restore/import.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix ci

Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: kennytm <kennytm@gmail.com>

* Max index length (pingcap#220)

* restore: set max-index-length to max

* restore:add max-index-length params

* address comment

* address comment

* glue: create schema/table directly with info (pingcap#216)

* glue: create schema/table directly with info

* go.mod: change to use the master version

* gluetidb: fix failure to create schema

* gluetidb: exclude non-public indices when restoring

* go.mod: removed unused replace

Co-authored-by: 3pointer <luancheng@pingcap.com>

* update deps

Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com>
Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: disksing <i@disksing.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>
Co-authored-by: Kolbe Kegel <kolbe@pingcap.com>
Co-authored-by: WangXiangUSTC <wx347249478@gmail.com>
Co-authored-by: glorv <glorvs@163.com>
Co-authored-by: MyonKeminta <9948422+MyonKeminta@users.noreply.github.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com>
Co-authored-by: 庄天翼 <zty0826@gmail.com>
Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>
@kennytm kennytm added difficulty/1-easy Easy issue Priority/P0 Top priority issue. Must have an associated milestone question Further information is requested labels May 28, 2020
@kennytm kennytm added this to the v4.0.1 milestone May 28, 2020
YuJuncen added a commit to YuJuncen/br that referenced this issue Jun 16, 2020
* restore: merge tidb-tools/pkg/restore-util (pingcap#146)

* restore-util: Implement split/scatter (pingcap#274)

* implement split/scatter

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* init test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* redesign output/input of the lib

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update dependency

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add commments and more tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add ScanRegions interface to Client

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix potential data race

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* Update pkg/restore-util/client.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update dependency

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* resolve conflicts

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix prefix rewrite

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add RewriteRule/skip failed scatter region/retry the SplitRegion

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check if region has peer

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* more logs

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add split retry interval (pingcap#277)

* reset dependencies to release-3.1

* add split retry interval

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix go.sum

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: wait for scatter region sequentially  (pingcap#279)

* wait for scatter region sequentially

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add on split hook (pingcap#281)

* restore-util: add on split hook

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Nil check onSplit

Co-Authored-By: kennytm <kennytm@gmail.com>

* restore-util: fix returned new region is nil (pingcap#283)

* restore-util: fix returned new region is nil

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* more logs

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* *: gofmt

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: call onSplit on splitByRewriteRules (pingcap#285)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore-util: fix overlapped error message (pingcap#293)

* restore-util: fix overlapped error message

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix log message

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* reduce error trace

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: log warning when cannot find matched rewrite rule (pingcap#299)

* restore-util: add method to set placement rules and store labels (pingcap#301)

* restore-util: add method to set placement rules and store labels

Signed-off-by: disksing <i@disksing.com>

* minor fix

Signed-off-by: disksing <i@disksing.com>

* address comment

Signed-off-by: disksing <i@disksing.com>

* add GetPlacementRules

Signed-off-by: disksing <i@disksing.com>

* fix test

Signed-off-by: disksing <i@disksing.com>

* restore-util: support batch split (pingcap#300)

* restore-util: support batch split

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* go fmt

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* address commits

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update pkg/restore-util/split.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* add onSplit callback

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add upper bound time for waiting for scatter (pingcap#305)

* restore: fix scatter regions failed

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* stop waiting for scatter after 3min

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: fix wrong url (pingcap#306)

Signed-off-by: disksing <i@disksing.com>

* restore-util: add warning about unmatched table id (pingcap#313)

* restore-util: support table partition

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* warn table id does not match

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add unit tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: Neil Shen <overvenus@gmail.com>

* fix compile error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Ian <ArGregoryIan@gmail.com>
Co-authored-by: Neil Shen <overvenus@gmail.com>

* *: prune tidb-tools

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: address linters suggestions

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: merge restoreutil into restore

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comment

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com>
Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: disksing <i@disksing.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>

* Fixed handling for a dbName that do not exist in the backup being restored (pingcap#148)

* Fixed handling for a dbName that do not exist in the backup being restored

* Fixed handling for a dbName that do not exist in the backup being restored

* validate: fix debug meta test ci (pingcap#153)

* validate: fix debug meta test ci

* *: extracts runBackup/runRestore in cmd into pkg/task (pingcap#156)

* *: extracts runBackup/runRestore in cmd into pkg/task

Defines a "Config" structure to store the parsed flags.

Use the "black-white-list" structure to define what tables/databases to
backup/restore.

* go.mod: update tidb to v4.0.0-beta

* restore: fix restore summary log (pingcap#150)

Co-authored-by: kennytm <kennytm@gmail.com>

* restore: enhance error handling (pingcap#152)

* restore: enhance error handling

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* unit test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix region epoch error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* remove `Restore*`

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address lint

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add debug log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* Update pkg/restore/import.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix retry error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* handle RegionNotFound error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: kennytm <kennytm@gmail.com>

* Incremental BR: support DDL (pingcap#155)

* support backup&restore ddl

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* integration tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update kvproto

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix integration tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* reduce cyclomatic complexity of `runRestore`

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add unit test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* disable fast checksum in incremental br

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix no valid key error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address lint

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Reduce TiDB dependencies (pingcap#158)

* utils: exclude mock_cluster outside of unit test

* utils: remove unused ResultSetToStringSlice()

* *: abstract away dependencies of tidb/session into a Glue interface

* *: fix hound lint

* util,mock: move utils.MockCluster to mock.Cluster

* restore: fix test build failure

Co-authored-by: 3pointer <qdlc2010@gmail.com>

* go.mod: update tidb (pingcap#168)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* BR support TLS (pingcap#161)

* *: support tls

* move tikv.driver to glue

* fix comments

* upgrade golangci and prepare for go 1.14 (pingcap#171)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* backup: add raw backup command (pingcap#101)

* backup: add raw backup command

* restore: speed up retry on not leader (pingcap#179)

* tests: stable cluster start up

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: fix unbound var

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: speed retry on not leader

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: add --cacert flag

Signed-off-by: Neil Shen <overvenus@gmail.com>

* make codecov green

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* conn, restore: paginate scan regions (pingcap#165)

* conn, restore: paginate scan regions

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: large timeout

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Batch restore (pingcap#167)

* *: unify Range and RangeTree

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: split restore files into small batch

Signed-off-by: Neil Shen <overvenus@gmail.com>

* task: set default restore concurrency to 128

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: unused table worker pool

Signed-off-by: Neil Shen <overvenus@gmail.com>

* summary: sum up repeated duration and int

Signed-off-by: Neil Shen <overvenus@gmail.com>

* rtree: move rtree from utils to pkg

Signed-off-by: Neil Shen <overvenus@gmail.com>

* README, docker: add quick start (pingcap#181)

* README, docker: add quick start

Signed-off-by: Neil Shen <overvenus@gmail.com>

* cmd: disable some TiDB log

Signed-off-by: Neil Shen <overvenus@gmail.com>

* docker: build go-ycsb automatically

Signed-off-by: Neil Shen <overvenus@gmail.com>

* cmd: add TODO about TiDB logs

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: update tidb dependency build with go1.14 (pingcap#176)

* *: add license header (pingcap#182)

* rtree: move checkFile into backup

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: add license header

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Update LICENSE.md

Co-Authored-By: kennytm <kennytm@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>

* conn: support not shutting down the storage when closing the connection (pingcap#185)

Co-authored-by: 3pointer <luancheng@pingcap.com>

* conn: use GetDomain to avoid some TiDB breaking changes (pingcap#186)

* conn: use GetDomain to avoid some TiDB breaking changes

Signed-off-by: Neil Shen <overvenus@gmail.com>

* minor usability improvement

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>

* fix check safepoint & unhide experimental features (pingcap#175)

* backup: check safepoint for last backup ts

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check lastbackupts > 0

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* unhide experimental features

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update tests/br_z_gc_safepoint/run.sh

Co-Authored-By: kennytm <kennytm@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>

* support backupts (pingcap#172)

* support backupts

* address comment

* address comment

* fix space

* *: update pd deps to v4 (pingcap#184)

Co-authored-by: 3pointer <luancheng@pingcap.com>

* restore: support online restore (pingcap#114)

Signed-off-by: disksing <i@disksing.com>

* metrics: add grafana scripts (pingcap#140)

* add grafana scripts

* fix

Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: glorv <glorvs@163.com>
Co-authored-by: kennytm <kennytm@gmail.com>

* filter out all TiFlash nodes when retrieving lists of stores from PD (pingcap#187)

* conn: ignore nodes with label engine=tiflash

* conn: disallow TiFlash on restore, only skip TiFlash on backup

* Create integration test for S3 storage (pingcap#174)

* Fix summary log (pingcap#191)

* *: fix restore summary log after restore logic changed to files

* fix

* fix

* fix

Co-authored-by: kennytm <kennytm@gmail.com>

* Implement Raw Restore (pingcap#104)

* Update kvproto

* Implement raw restore

* fix build

* Set range for file importer

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Remove unnecessary comments

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* check cf and support multi ranges in BackupMeta

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Check files' cf; address comments

* adjust structure to keep consistent with master

* Fix build

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix build and make check, avoid accessing TiDB in rawkv mode

* Fix test

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix tests

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix broken logic after merging master

* Update pkg/task/restore_raw.go

Co-Authored-By: Neil Shen <overvenus@gmail.com>

* Address comments

* Address comments

* Mark raw restore as experimental

* Fix build

* Address comments

* test: Add check for deleting data and partial backup

* Fix build

* Add license header

* fix ci

* fix ci

Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com>

* restore: remove tiflash replica before restore (pingcap#194)

* restore: remove tiflash replica before restore

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* rename errSplit variable

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check replica count by region info

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* cleanup

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* save tiflash replica count to backupmeta

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix save crcxor

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix decode the key of placement rule

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address lint

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* close domain after restoring tiflash-replica

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update pkg/task/restore.go

Co-Authored-By: 3pointer <qdlc2010@gmail.com>

Co-authored-by: 3pointer <qdlc2010@gmail.com>

* summary: put summary log at last (pingcap#197)

* summary: put summary log at last

* fix switch sql

* *: abstract the progress channel (updateCh) into the glue package (pingcap#196)

* *: abstract the progress channel (updateCh) into the glue package

* restore: fix crash in truncateTS() when the bound is unlimited

* task: fix comment

Co-authored-by: Ian <ArGregoryIan@gmail.com>

* *: refline logs (pingcap#189)

* tests: disable TLS test (pingcap#204)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: add S3 quick start and few enhancement of log (pingcap#202)

* README, docker: add quick start about S3 storage

Signed-off-by: Neil Shen <overvenus@gmail.com>

* pkg/summary: make sure to output correct summary

Signed-off-by: Neil Shen <overvenus@gmail.com>

* cmd, tests: log to terminal if BR_LOG_TO_TERM is set

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Update pkg/task/common.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: cat log if br fails

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>

* restore: add error field to `DownloadResponse` (pingcap#195)

* restore: add error field to `DownloadResponse`

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore: populate restore cancel error (pingcap#207)

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>

* enhance usability of br (pingcap#208)

* silenceUsage only when parse cmd flags failed

* udpate tidb

Co-authored-by: kennytm <kennytm@gmail.com>

* task: do not run checksum if restore failed (pingcap#209)

* fix incremental bug in llroad test (pingcap#199)

* restore: filter same table ddl

* *: do not return error when backup/restore data is empty

* fix create database double during incremental restore

* add tests

* fix ci

* address comment

* add skip create sqls (pingcap#211)

* Revert "tests: disable TLS test (pingcap#204)" (pingcap#218)

This reverts commit e168a60.

* doc: add `minio` to dependence list. (pingcap#221)

The README of test omitted `minio` in the dependence list, which is needed for run the integration test.

Co-authored-by: Neil Shen <overvenus@gmail.com>

* move waiting reject stores in import file (pingcap#222)

* move wait rejectstores into import files

* restore: use new table id to search placementRules

* Update pkg/restore/import.go

Co-Authored-By: Neil Shen <overvenus@gmail.com>

* Update pkg/restore/import.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix ci

Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: kennytm <kennytm@gmail.com>

* Max index length (pingcap#220)

* restore: set max-index-length to max

* restore:add max-index-length params

* address comment

* address comment

* glue: create schema/table directly with info (pingcap#216)

* glue: create schema/table directly with info

* go.mod: change to use the master version

* gluetidb: fix failure to create schema

* gluetidb: exclude non-public indices when restoring

* go.mod: removed unused replace

Co-authored-by: 3pointer <luancheng@pingcap.com>

Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com>
Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: disksing <i@disksing.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>
Co-authored-by: Kolbe Kegel <kolbe@pingcap.com>
Co-authored-by: WangXiangUSTC <wx347249478@gmail.com>
Co-authored-by: glorv <glorvs@163.com>
Co-authored-by: MyonKeminta <9948422+MyonKeminta@users.noreply.github.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com>
Co-authored-by: 庄天翼 <zty0826@gmail.com>
Co-authored-by: 山岚 <36239017+YuJuncen@users.noreply.github.com>
@3pointer 3pointer modified the milestones: v4.0.2, v4.0.3 Jun 19, 2020
3pointer added a commit to 3pointer/br that referenced this issue Jul 15, 2020
* restore: merge tidb-tools/pkg/restore-util (pingcap#146)

* restore-util: Implement split/scatter (pingcap#274)

* implement split/scatter

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* init test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* redesign output/input of the lib

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update dependency

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add commments and more tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add ScanRegions interface to Client

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix potential data race

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* Update pkg/restore-util/client.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update dependency

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* resolve conflicts

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix prefix rewrite

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add RewriteRule/skip failed scatter region/retry the SplitRegion

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check if region has peer

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* more logs

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add split retry interval (pingcap#277)

* reset dependencies to release-3.1

* add split retry interval

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix go.sum

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: wait for scatter region sequentially  (pingcap#279)

* wait for scatter region sequentially

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add on split hook (pingcap#281)

* restore-util: add on split hook

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Nil check onSplit

Co-Authored-By: kennytm <kennytm@gmail.com>

* restore-util: fix returned new region is nil (pingcap#283)

* restore-util: fix returned new region is nil

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* more logs

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* *: gofmt

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: call onSplit on splitByRewriteRules (pingcap#285)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore-util: fix overlapped error message (pingcap#293)

* restore-util: fix overlapped error message

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix log message

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* reduce error trace

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: log warning when cannot find matched rewrite rule (pingcap#299)

* restore-util: add method to set placement rules and store labels (pingcap#301)

* restore-util: add method to set placement rules and store labels

Signed-off-by: disksing <i@disksing.com>

* minor fix

Signed-off-by: disksing <i@disksing.com>

* address comment

Signed-off-by: disksing <i@disksing.com>

* add GetPlacementRules

Signed-off-by: disksing <i@disksing.com>

* fix test

Signed-off-by: disksing <i@disksing.com>

* restore-util: support batch split (pingcap#300)

* restore-util: support batch split

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* go fmt

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* address commits

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update pkg/restore-util/split.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* add onSplit callback

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add upper bound time for waiting for scatter (pingcap#305)

* restore: fix scatter regions failed

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* stop waiting for scatter after 3min

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: fix wrong url (pingcap#306)

Signed-off-by: disksing <i@disksing.com>

* restore-util: add warning about unmatched table id (pingcap#313)

* restore-util: support table partition

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* warn table id does not match

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add unit tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: Neil Shen <overvenus@gmail.com>

* fix compile error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Ian <ArGregoryIan@gmail.com>
Co-authored-by: Neil Shen <overvenus@gmail.com>

* *: prune tidb-tools

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: address linters suggestions

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: merge restoreutil into restore

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comment

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com>
Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: disksing <i@disksing.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>

* Fixed handling for a dbName that do not exist in the backup being restored (pingcap#148)

* Fixed handling for a dbName that do not exist in the backup being restored

* Fixed handling for a dbName that do not exist in the backup being restored

* validate: fix debug meta test ci (pingcap#153)

* validate: fix debug meta test ci

* *: extracts runBackup/runRestore in cmd into pkg/task (pingcap#156)

* *: extracts runBackup/runRestore in cmd into pkg/task

Defines a "Config" structure to store the parsed flags.

Use the "black-white-list" structure to define what tables/databases to
backup/restore.

* go.mod: update tidb to v4.0.0-beta

* restore: fix restore summary log (pingcap#150)

Co-authored-by: kennytm <kennytm@gmail.com>

* restore: enhance error handling (pingcap#152)

* restore: enhance error handling

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* unit test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix region epoch error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* remove `Restore*`

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address lint

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add debug log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* Update pkg/restore/import.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix retry error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* handle RegionNotFound error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: kennytm <kennytm@gmail.com>

* Incremental BR: support DDL (pingcap#155)

* support backup&restore ddl

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* integration tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update kvproto

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix integration tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* reduce cyclomatic complexity of `runRestore`

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add unit test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* disable fast checksum in incremental br

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix no valid key error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address lint

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Reduce TiDB dependencies (pingcap#158)

* utils: exclude mock_cluster outside of unit test

* utils: remove unused ResultSetToStringSlice()

* *: abstract away dependencies of tidb/session into a Glue interface

* *: fix hound lint

* util,mock: move utils.MockCluster to mock.Cluster

* restore: fix test build failure

Co-authored-by: 3pointer <qdlc2010@gmail.com>

* go.mod: update tidb (pingcap#168)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* BR support TLS (pingcap#161)

* *: support tls

* move tikv.driver to glue

* fix comments

* upgrade golangci and prepare for go 1.14 (pingcap#171)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* backup: add raw backup command (pingcap#101)

* backup: add raw backup command

* restore: speed up retry on not leader (pingcap#179)

* tests: stable cluster start up

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: fix unbound var

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: speed retry on not leader

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: add --cacert flag

Signed-off-by: Neil Shen <overvenus@gmail.com>

* make codecov green

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comments

Signed-off-by: Neil Shen <overvenus@gmail.com>

* conn, restore: paginate scan regions (pingcap#165)

* conn, restore: paginate scan regions

Signed-off-by: Neil Shen <overvenus@gmail.com>

* tests: large timeout

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Batch restore (pingcap#167)

* *: unify Range and RangeTree

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: split restore files into small batch

Signed-off-by: Neil Shen <overvenus@gmail.com>

* task: set default restore concurrency to 128

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: unused table worker pool

Signed-off-by: Neil Shen <overvenus@gmail.com>

* summary: sum up repeated duration and int

Signed-off-by: Neil Shen <overvenus@gmail.com>

* rtree: move rtree from utils to pkg

Signed-off-by: Neil Shen <overvenus@gmail.com>

* README, docker: add quick start (pingcap#181)

* README, docker: add quick start

Signed-off-by: Neil Shen <overvenus@gmail.com>

* cmd: disable some TiDB log

Signed-off-by: Neil Shen <overvenus@gmail.com>

* docker: build go-ycsb automatically

Signed-off-by: Neil Shen <overvenus@gmail.com>

* cmd: add TODO about TiDB logs

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: update tidb dependency build with go1.14 (pingcap#176)

* *: add license header (pingcap#182)

* rtree: move checkFile into backup

Signed-off-by: Neil Shen <overvenus@gmail.com>

* *: add license header

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Update LICENSE.md

Co-Authored-By: kennytm <kennytm@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>

* conn: support not shutting down the storage when closing the connection (pingcap#185)

Co-authored-by: 3pointer <luancheng@pingcap.com>

* conn: use GetDomain to avoid some TiDB breaking changes (pingcap#186)

* conn: use GetDomain to avoid some TiDB breaking changes

Signed-off-by: Neil Shen <overvenus@gmail.com>

* minor usability improvement

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>

* fix check safepoint & unhide experimental features (pingcap#175)

* backup: check safepoint for last backup ts

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check lastbackupts > 0

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* unhide experimental features

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comment

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update tests/br_z_gc_safepoint/run.sh

Co-Authored-By: kennytm <kennytm@gmail.com>

Co-authored-by: kennytm <kennytm@gmail.com>

* support backupts (pingcap#172)

* support backupts

* address comment

* address comment

* fix space

* *: update pd deps to v4 (pingcap#184)

Co-authored-by: 3pointer <luancheng@pingcap.com>

* restore: support online restore (pingcap#114)

Signed-off-by: disksing <i@disksing.com>

* metrics: add grafana scripts (pingcap#140)

* add grafana scripts

* fix

Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: glorv <glorvs@163.com>
Co-authored-by: kennytm <kennytm@gmail.com>

* filter out all TiFlash nodes when retrieving lists of stores from PD (pingcap#187)

* conn: ignore nodes with label engine=tiflash

* conn: disallow TiFlash on restore, only skip TiFlash on backup

* Create integration test for S3 storage (pingcap#174)

* Fix summary log (pingcap#191)

* *: fix restore summary log after restore logic changed to files

* fix

* fix

* fix

Co-authored-by: kennytm <kennytm@gmail.com>

* Implement Raw Restore (pingcap#104)

* Update kvproto

* Implement raw restore

* fix build

* Set range for file importer

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Remove unnecessary comments

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* check cf and support multi ranges in BackupMeta

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Check files' cf; address comments

* adjust structure to keep consistent with master

* Fix build

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix build and make check, avoid accessing TiDB in rawkv mode

* Fix test

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix tests

Signed-off-by: MyonKeminta <MyonKeminta@users.noreply.github.com>

* Fix broken logic after merging master

* Update pkg/task/restore_raw.go

Co-Authored-By: Neil Shen <overvenus@gmail.com>

* Address comments

* Address comments

* Mark raw restore as experimental

* Fix build

* Address comments

* test: Add check for deleting data and partial backup

* Fix build

* Add license header

* fix ci

* fix ci

Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: 3pointer <luancheng@pingcap.com>
Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com>

* restore: remove tiflash replica before restore (pingcap#194)

* restore: remove tiflash replica before restore

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* rename errSplit variable

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check replica count by region info

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* cleanup

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* save tiflash replica count to backupmeta

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix save crcxor

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix decode the key of placement rule

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address lint

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* close domain after restoring tiflash-replica

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update pkg/task/restore.go

Co-Authored-By: 3pointer <qdlc2010@gmail.com>

Co-authored-by: 3pointer <qdlc2010@gmail.com>

* summary: put summary log at last (pingcap#197)

* summary: put summary log at last

* fix switch sql

Co-authored-by: Neil Shen <overvenus@gmail.com>
Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com>
Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: disksing <i@disksing.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>
Co-authored-by: Kolbe Kegel <kolbe@pingcap.com>
Co-authored-by: WangXiangUSTC <wx347249478@gmail.com>
Co-authored-by: glorv <glorvs@163.com>
Co-authored-by: MyonKeminta <9948422+MyonKeminta@users.noreply.github.com>
Co-authored-by: MyonKeminta <MyonKeminta@users.noreply.github.com>
Co-authored-by: pingcap-github-bot <sre-bot@pingcap.com>
@overvenus overvenus removed this from the v4.0.3 milestone Jul 16, 2020
overvenus added a commit to overvenus/br-1 that referenced this issue Feb 1, 2021
* restore-util: Implement split/scatter (pingcap#274)

* implement split/scatter

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* init test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* redesign output/input of the lib

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update dependency

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add commments and more tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add ScanRegions interface to Client

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix potential data race

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* Update pkg/restore-util/client.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* update dependency

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* resolve conflicts

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix prefix rewrite

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add RewriteRule/skip failed scatter region/retry the SplitRegion

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* check if region has peer

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* more logs

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add split retry interval (pingcap#277)

* reset dependencies to release-3.1

* add split retry interval

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix go.sum

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: wait for scatter region sequentially  (pingcap#279)

* wait for scatter region sequentially

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add on split hook (pingcap#281)

* restore-util: add on split hook

Signed-off-by: Neil Shen <overvenus@gmail.com>

* Nil check onSplit

Co-Authored-By: kennytm <kennytm@gmail.com>

* restore-util: fix returned new region is nil (pingcap#283)

* restore-util: fix returned new region is nil

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* more logs

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* *: gofmt

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* fix log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: call onSplit on splitByRewriteRules (pingcap#285)

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore-util: fix overlapped error message (pingcap#293)

* restore-util: fix overlapped error message

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix log message

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* reduce error trace

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: log warning when cannot find matched rewrite rule (pingcap#299)

* restore-util: add method to set placement rules and store labels (pingcap#301)

* restore-util: add method to set placement rules and store labels

Signed-off-by: disksing <i@disksing.com>

* minor fix

Signed-off-by: disksing <i@disksing.com>

* address comment

Signed-off-by: disksing <i@disksing.com>

* add GetPlacementRules

Signed-off-by: disksing <i@disksing.com>

* fix test

Signed-off-by: disksing <i@disksing.com>

* restore-util: support batch split (pingcap#300)

* restore-util: support batch split

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* go fmt

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: kennytm <kennytm@gmail.com>

* address commits

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Update pkg/restore-util/split.go

Co-Authored-By: kennytm <kennytm@gmail.com>

* add onSplit callback

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: add upper bound time for waiting for scatter (pingcap#305)

* restore: fix scatter regions failed

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* stop waiting for scatter after 3min

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* restore-util: fix wrong url (pingcap#306)

Signed-off-by: disksing <i@disksing.com>

* restore-util: add warning about unmatched table id (pingcap#313)

* restore-util: support table partition

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix log

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* warn table id does not match

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* add unit tests

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* Apply suggestions from code review

Co-Authored-By: Neil Shen <overvenus@gmail.com>

* fix compile error

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* address comments

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

* fix test

Signed-off-by: 5kbpers <tangminghua@pingcap.com>

Co-authored-by: Ian <ArGregoryIan@gmail.com>
Co-authored-by: Neil Shen <overvenus@gmail.com>

* *: prune tidb-tools

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: address linters suggestions

Signed-off-by: Neil Shen <overvenus@gmail.com>

* restore: merge restoreutil into restore

Signed-off-by: Neil Shen <overvenus@gmail.com>

* address comment

Signed-off-by: Neil Shen <overvenus@gmail.com>

Co-authored-by: 5kbpers <20279863+5kbpers@users.noreply.github.com>
Co-authored-by: kennytm <kennytm@gmail.com>
Co-authored-by: disksing <i@disksing.com>
Co-authored-by: Ian <ArGregoryIan@gmail.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
difficulty/1-easy Easy issue Priority/P0 Top priority issue. Must have an associated milestone question Further information is requested type/feature-request New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants