Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

import into failed when inject pdleader io delay 1s last for 1mins #48307

Closed
Lily2025 opened this issue Nov 6, 2023 · 2 comments · Fixed by #48312
Closed

import into failed when inject pdleader io delay 1s last for 1mins #48307

Lily2025 opened this issue Nov 6, 2023 · 2 comments · Fixed by #48312
Assignees
Labels
affects-6.1 affects-6.5 affects-7.1 affects-7.5 component/ddl This issue is related to DDL of TiDB. severity/major type/bug The issue is confirmed as a bug.

Comments

@Lily2025
Copy link

Lily2025 commented Nov 6, 2023

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

tidb_enable_dist_task='on'
1、import into
2、inject pdleader io delay 1s last for 1mins

case config
tag: "ha-test-import-into"
cases:
- name: ha_pd(leader)_io_delay(1s)
faultType: iochaos_io_delay(volumePath:/var/lib/pd,path:/var/lib/pd/data/**/*,delay:1s,percent:100)
warmUpTime: 1m
selector: pd(leader)
period: "@every 5m"
faultDuration: 1m
faultTotalRunTime: 20m
importInto:
- db: "importtest"
schema: "CREATE TABLE user_data1 (
uid bigint(20) unsigned NOT NULL AUTO_INCREMENT,
data mediumblob NOT NULL,
update_data blob NOT NULL,
create_time datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
update_time datetime DEFAULT NULL,
PRIMARY KEY (uid) /*T![clustered_index] CLUSTERED /,
KEY k_1 (update_time)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin AUTO_INCREMENT=5850196;"
cmd: "import into user_data1 from 's3://brie/lightning-mhy-100G-csv/sysbench.user_data1.000001
.csv?access-key=xxx\u0026secret-access-key=xxx\u0026endpoint=http://minio.pingcap.net:9001\u0026force-path-style=true' WITH DETACHED,skip_rows=1"
timeout: 120m
crc32Checksum: 3582177512

2. What did you expect to see? (Required)

impor into can success

3. What did you see instead (Required)

import into failed when inject pdleader io delay 1s last for 1mins

the status of import job is not finished or running (now: 2023-11-05 05:48:04, jobId: 30011, step: importing, status: failed)
operatorLogs:
[2023-11-05 05:33:33] ###### start import into
import into user_data1 from 's3://brie/lightning-mhy-100G-csv/sysbench.user_data1.000001*.csv?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http://minio.pingcap.net:9001&force-path-style=true' WITH DETACHED,skip_rows=1
[2023-11-05 05:33:33] ###### wait for import job to finish
[2023-11-05 05:48:04] ###### wait for import job to finish failed
select id, step, status from mysql.tidb_import_jobs where start_time >= '2023-11-05 05:33:33'
jobId: 30011, step: importing, status: failed

4. What is your TiDB version? (Required)

git hash: aa93091

@Lily2025 Lily2025 added the type/bug The issue is confirmed as a bug. label Nov 6, 2023
@Lily2025
Copy link
Author

Lily2025 commented Nov 6, 2023

/assign D3Hunter

@D3Hunter
Copy link
Contributor

D3Hunter commented Nov 6, 2023

the body cannot be reused

[2023/11/05 05:47:34.890 +08:00] [ERROR] [backend.go:371] ["import failed"] [engineTag=`importtest`.`user_data1`:0] [engineUUID=9a09d9d7-e635-547b-86cb-e8564a7e4a95] [retryCnt=0] [takeTime=49.215683303s] [error="[400] {\n  \"code\": \"input\",\n  \"msg\": \"unexpected end of JSON input\",\n  \"data\": {\n    \"Offset\": 0\n  }\n}\n http://tc-pd:2379/pd/api/v1/config/region-label/rule: [BR:PD:ErrPDInvalidResponse]PD invalid response"]

req, err = http.NewRequestWithContext(ctx, method, reqURL, body)

@ti-chi-bot ti-chi-bot bot added may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 labels Nov 6, 2023
@D3Hunter D3Hunter added affects-6.1 affects-6.5 and removed may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.5 labels Nov 6, 2023
ti-chi-bot bot pushed a commit that referenced this issue Nov 6, 2023
@jebter jebter added the component/ddl This issue is related to DDL of TiDB. label Nov 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
affects-6.1 affects-6.5 affects-7.1 affects-7.5 component/ddl This issue is related to DDL of TiDB. severity/major type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants