Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

executor, server: load_data.go is changed and add unit test #23201

Merged
merged 7 commits into from
Mar 29, 2021
Merged

executor, server: load_data.go is changed and add unit test #23201

merged 7 commits into from
Mar 29, 2021

Conversation

zoujia-cm
Copy link
Contributor

What problem does this PR solve?

Problem Summary:
I use "load data" statement to load data from csv file. In csv file, fields are enclosed by "'" and are terminated by ",". lines are terminated by "|"(for example: '11','22'|'33','44'). I generated a csv file with 50000 items of record. When load it into db by "load data" statement, only items less than 50000 (often loss at least one item) were loaded.

What is changed and how it works?

What's Changed:
The way to locate the terminator (include field terminator and line terminator) was changed.
How it Works:
Line terminator can be located precisely in different situations.

Related changes

  • Need to cherry-pick to the release branch

Tests

  • Unit test

Side effects

  • Performance regression
    • Consumes more CPU
    • Consumes more MEM

Release note

  • fix load data bug that the number of items loaded by "load data" is less than the actual number

@zoujia-cm zoujia-cm requested a review from a team as a code owner March 9, 2021 08:49
@zoujia-cm zoujia-cm requested review from qw4990 and removed request for a team March 9, 2021 08:49
@ti-srebot ti-srebot added the first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. label Mar 9, 2021
@ti-chi-bot ti-chi-bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Mar 9, 2021
@github-actions github-actions bot added sig/execution SIG execution sig/sql-infra SIG: SQL Infra labels Mar 9, 2021
@zoujia-cm
Copy link
Contributor Author

/cc @tiancaiamao /cc @zimulala /cc @hi-rustin PTAL

@ti-chi-bot
Copy link
Member

@zoujia-cm: GitHub didn't allow me to request PR reviews from the following users: PTAL, /cc.

Note that only pingcap members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @tiancaiamao /cc @zimulala /cc @hi-rustin PTAL

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@zoujia-cm
Copy link
Contributor Author

/cc @zimulala PTAL

@ti-chi-bot
Copy link
Member

@zoujia-cm: GitHub didn't allow me to request PR reviews from the following users: PTAL.

Note that only pingcap members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @zimulala PTAL

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@Rustin170506
Copy link
Member

/uncc

Sorry, I am not a reviewer of these SIGs, I suggest you join the sig-sql-infra slack, and then join the SIG channel to request a review.

@ti-chi-bot ti-chi-bot removed the request for review from Rustin170506 March 10, 2021 10:31
@zoujia-cm
Copy link
Contributor Author

/cc @zimulala

@zoujia-cm
Copy link
Contributor Author

/cc @zimulala

@zoujia-cm
Copy link
Contributor Author

@zimulala please take a look

@zoujia-cm
Copy link
Contributor Author

/cc @zimulala

1 similar comment
@zoujia-cm
Copy link
Contributor Author

/cc @zimulala

@zoujia-cm
Copy link
Contributor Author

/cc @qw4990

2 similar comments
@zoujia-cm
Copy link
Contributor Author

/cc @qw4990

@zoujia-cm
Copy link
Contributor Author

/cc @qw4990

@zoujia-cm
Copy link
Contributor Author

/cc @qw4990

zz-jason
zz-jason reviewed Mar 27, 2021
View reviewed changes
server/server_test.go
@@ -575,6 +575,63 @@ func (cli *testServerClient) runTestLoadDataAutoRandom(c *C) {
})
}

func (cli *testServerClient) runTestLoadDataAutoRandomWithSpecialTerm(c *C) {
path := "/tmp/load_data_txn_error_term.csv"
Copy link
Member

@zz-jason zz-jason Mar 27, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to convert this test to an integration test.

zz-jason
zz-jason reviewed Mar 27, 2021
View reviewed changes
Copy link
Member

@zz-jason zz-jason left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about using executor, server: support " as field encloser for load data stmt as the PR title?

@tiancaiamao
Copy link
Contributor

The original PR is this one #22834

@tiancaiamao
Copy link
Contributor

/LGTM
@zimulala

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 29, 2021
@zimulala
Copy link
Contributor

/LGTM

@ti-chi-bot
Copy link
Member

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • tiancaiamao
  • zimulala

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

@ti-chi-bot ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Mar 29, 2021
@zimulala
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: ad00ffd

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Mar 29, 2021
@ti-chi-bot ti-chi-bot merged commit 76d48c1 into pingcap:master Mar 29, 2021
@xiongjiwei xiongjiwei mentioned this pull request Oct 29, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zz-jason zz-jason zz-jason left review comments

@qw4990 qw4990 Awaiting requested review from qw4990

@tiancaiamao tiancaiamao Awaiting requested review from tiancaiamao

@zimulala zimulala Awaiting requested review from zimulala

Assignees
No one assigned
Labels
first-time-contributor Indicates that the PR was contributed by an external member and is a first-time contributor. sig/execution SIG execution sig/sql-infra SIG: SQL Infra size/M Denotes a PR that changes 30-99 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@zoujia-cm @ti-chi-bot @Rustin170506 @tiancaiamao @zimulala @zz-jason @ti-srebot