Summary for unexpected replication lags & stops

[liuzix]

There have been multiple reports about unexpected replication lags and/or stops. We summarize the issue here and track the progress towards resolving it.

Issues that need addressing

• 1.1 Fix the known TiKV problem where cancelling an incremental scan in one connection causes other connections to not have resolved-ts's. Issues Simultaneous incremental scans interfere with each other causing replication to not proceed #1626, release-5.0 PR deregister region when resolver fails to build tikv/tikv#9987, master PR pending.
• 1.2 KVClient v1 seems not to be able to handle 50000+ regions, which causes very large resolved-ts lags (15 minutes+) and potentially other implications. We need to find a workaround or stabilize KVClient v2 as soon as possible.
• 1.3 KVClient v1 & v2 seem not to handle EOF of streams properly. We need a fix and proper tests for this. kv/client: fix the error handling for io.EOF may miss region reconnect #1641

Other action items

• 2.1 Better metrics to help diagnose similar problems.
• 2.2 Investigate data accumulating in grpc send buffer for some unknown reason.
• 2.3 Investigate TiCDC resolved-ts not moving forward when the upstream has many relatively large transactions.

[amyangfei]

KVClient v1 & v2 seem not to handle EOF of streams properly. We need a fix and proper tests for this.
Does this mean stream.Recv with io.EOF error is not handled properly, or something else? @liuzix