Build dataset using COPY instead of multi-row inserts for TPROC-C #301
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The tcl files in this repo did not seem to adhere to any common indentation style. This made it quite hard for me to understand what the code was doing. To resolve this I used the reformat.tcl script that is provided on the tcl wiki: https://wiki.tcl-lang.org/page/Reformatting+Tcl+code+indentation I replaced the original reformat proc that was provided in the first message, with the reformat2 proc that was contributed by aplsimple in the last message on the thread. I did make one final change to this reformat2 script, such that does not exclude comments from indentation. I think this makes the code much more readable. I understand that this is a big PR, but it only changes whitespace. If you look at the diff by excluding whitespace, there's only one change. And this change is a change from file encoding from latin1 to utf-8.
JelteF
changed the title
The fastest way to bulk insert data in Postgres is by using COPY. This changes the dataset building to use that. I tried building a dataset for 1000 warehouses using 100 vusers. Without copy this took 100 minutes, with copy it only took 42 minutes. So it reduced the time it takes to build the dataset by ~58%. Docs on the usage of COPY in the tcl Postgres library can be found here: http://pgtclng.sourceforge.net/pgtcldocs/pgtcl-example-copy.html
|
I have tested this change on the same system with the changes for #295 already in place. |
sm-shaw
added a commit
that referenced
this pull request
Jan 12, 2022
Resolve Merge Conflicts for #301
|
Merging Pull Request as voted on by TPC-OSS subcommittee on 11th Jan 2022. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
NOTE: this PR includes the #292, to make it more readable and make merging them
both in the future easier. Adding comments to the diff is probably easier to do
on this equivalent PR with a different base branch on the Citus repo: citusdata#3
The fastest way to bulk insert data in Postgres is by using COPY. This
changes the dataset building to use that for TPROC-C. I tried building a dataset for
1000 warehouses using 100 vusers. Without copy this took 100 minutes, with
copy it only took 42 minutes. So it reduced the time it takes to build the
dataset by ~58%.
Docs on the usage of COPY in the tcl Postgres library can be found here:
http://pgtclng.sourceforge.net/pgtcldocs/pgtcl-example-copy.html
NOTE: A similar improvement could probably be done for postgres on TPROC-H