Support replace mode for lightning post-import conflict detection

[lyzx2001]

Feature Request

Is your feature request related to a problem? Please describe:

Currently lightning only supports remove mode for post-import conflict detection, but many customers request lightning to support replace mode for lightning post-import conflict detection.

Describe the feature you'd like:

replace mode for lightning post-import conflict detection:
To resolve rows with conflict, instead of deleting all the rows that are engaged in conflict (the algorithm for remove), we delete some rows with conflict and reserve other rows that can be kept and not cause conflict anymore. Under this circumstance, we only delete the necessary rows to resolve conflicts, so that we can keep more original rows than remove mode as long as the conflicts are resolved.

For instance, we have the following table structure:

create table test (c1 int primary key, c2 int);

And we have 2 rows:

1, 111
1, 222

remove mode would delete both 2 rows, but replace mode would only delete 1 row and reserve the other one.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

Design documentation:

https://pingcap.feishu.cn/docx/XD52dwxfdoewUQxDDRtcdUq4nod?from=from_copylink

Implementation steps:

1. Change the structure of the tables that record data KV conflicts and index KV conflicts. Add index to track and search rows with conflict faster.
   

   tidb/br/pkg/lightning/backend/local/duplicate.go

   Lines 710 to 714 in 1e73eee

   	if task.indexInfo == nil {
   	err = m.RecordDataConflictError(gCtx, stream)
   	} else {
   	err = m.RecordIndexConflictError(gCtx, stream, task.tableID, task.indexInfo)
   	}

2. Use local backend import, set duplicate-resolution = 'record' to record the conflict rows but not delete them from downstream, manually use TiKV's API to extract information from the tables, and implement the replace algorithm to check the feasibility of the method.

3. Modify ResolveAllConflictKeys function to support the replace algorithm that delete only the necessary rows to resolve conflict.
   

   tidb/br/pkg/lightning/backend/local/duplicate.go

   Line 1012 in 1e73eee

   err = local.errorMgr.ResolveAllConflictKeys(

4. Optimize executing logic and performance for the replace algorithm.

[lyzx2001]

/assign @lyzx2001