lightning import failed after inject pdleader io delay 1s

[Lily2025]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、lightning import
- storage: s3://benchmark/location-dumpling
s3Endpoint: http://minio.pingcap.net:9000
db: location
options: --enable-checkpoint false
config: |-
[lightning]
max-error = 9999999999
[tikv-importer]
duplicate-resolution = "record"
incremental-import = true
crc32Checksum: 2645404411
adminCheck:
- table: AreaMeshLake
indexes:
- areameshlakeid
backend: local
sortedKvDir: /tmp/sorted-kv-dir
- storage: s3://brie/lightning-mhy-100G-csv
s3Endpoint: http://minio.pingcap.net:9001
db: sysbench
backend: local
sortedKvDir: /tmp/sorted-kv-dir
options: "--enable-checkpoint false"
config: "[[mydumper.files]]\npattern = "sysbench-schema-create.sql"\nschema = "sysbench"\ntype = "schema-schema"\n[[mydumper.files]]\npattern = "sysbench.user_data1-schema.sql"\nschema = "sysbench"\ntable = "user_data1"\ntype = "table-schema"\n[[mydumper.files]]\npattern = "sysbench.user_data1.000000[0-3][0-9][0-9].csv"\nschema = "sysbench"\ntable = "user_data1"\ntype = "csv""
crc32Checksum: 619903217
2、inject pdleader io delay 1s
- name: ha_pdleader_io_delay(1s)
faultType: iochaos_io_delay(volumePath:/var/lib/pd,path:/var/lib/pd/data/**/*,delay:1s,percent:100)
selector: pdleader
warmUpTime: 1m
period: "@every 3m"
faultDuration: 1m
faultTotalRunTime: 20m
lightningList: *id001

2. What did you expect to see? (Required)

lightning import can success

3. What did you see instead (Required)

lightning import failed after inject pdleader io delay 1s

tidb lightning encountered error: [Lightning:Restore:ErrRestoreTable]restore table sysbench.user_data1 failed: Post "http://tc-pd:2379/pd/api/v1/config/region-label/rule": http: ContentLength=274 with Body length 0

github.com/pingcap/errors.AddStack
	/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174
github.com/pingcap/errors.Trace
	/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15
github.com/pingcap/tidb/br/pkg/pdutil.pdRequestWithCode
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:203
github.com/pingcap/tidb/br/pkg/pdutil.pdRequest
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:154
github.com/pingcap/tidb/br/pkg/pdutil.(*PdController).CreateOrUpdateRegionLabelRule
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:983
github.com/pingcap/tidb/br/pkg/pdutil.(*PdController).pauseSchedulerByKeyRangeWithTTL
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:1048
github.com/pingcap/tidb/br/pkg/pdutil.(*PdController).PauseSchedulersByKeyRange
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/pdutil/pd.go:1027
github.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).ImportEngine
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1523
github.com/pingcap/tidb/br/pkg/lightning/backend.(*ClosedEngine).Import
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/backend.go:365
github.com/pingcap/tidb/br/pkg/lightning/importer.(*TableImporter).importKV
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/importer/table_import.go:1242
github.com/pingcap/tidb/br/pkg/lightning/importer.(*TableImporter).importEngine
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/importer/table_import.go:891
github.com/pingcap/tidb/br/pkg/lightning/importer.(*TableImporter).importEngines.func3
	/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/importer/table_import.go:504
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1650

4. What is your TiDB version? (Required)

./tidb-server -V
Release Version: v7.5.0-alpha
Edition: Community
Git Commit Hash: af7b32c
Git Branch: heads/refs/tags/v7.5.0-alpha
UTC Build Time: 2023-10-19 11:43:11
GoVersion: go1.21.3
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2023-10-20T08:01:59.525+0800

[Lily2025]

/severity major
/assign lance6716

[lance6716]

I guess it's caused by

tidb/br/pkg/pdutil/pd.go

Lines 167 to 174 in 05b50a1

	req, err := http.NewRequestWithContext(ctx, method, reqURL, body)
	if err != nil {
	return 0, nil, errors.Trace(err)
	}
	var resp *http.Response
	count := 0
	for {
	resp, err = cli.Do(req) //nolint:bodyclose

client.Do will close req.Body, so in second retry body size is 0.