Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate Performance implications of disabling transactions for COPY operations. #7809

Closed
psudheer21 opened this issue Mar 25, 2021 · 1 comment
Closed

Investigate Performance implications of disabling transactions for COPY operations. #7809

psudheer21 opened this issue Mar 25, 2021 · 1 comment
Assignees
@emhna

Comments

@psudheer21
Copy link
Contributor

We use the distributed transactions to perform the writes. Explore if avoiding the transaction code path can speed up the whole operation.

If the whole operation succeeds it ideally should be much faster. If the operation fails we can truncate the whole table and restart the operation. This obviously should only be used when the table that is to copied over is new and we can truncate the table when failures occur

The logic is to use the non-transactional session instead of the transactional session in the PGgate layer to perform the writes.
@psudheer21 psudheer21 self-assigned this Mar 25, 2021
@psudheer21 psudheer21 changed the title Investigate disabling transactions for COPY operations. Investigate Performance implications of disabling transactions for COPY operations. Mar 25, 2021
@psudheer21
Copy link
Contributor Author

This new mode of writes can also be used for the LOAD stage of TPCC.

@m-iancu m-iancu assigned emhna and unassigned psudheer21 Oct 12, 2021
emhna added a commit that referenced this issue Nov 30, 2021
@emhna
[#7809][YSQL] Add session flag to enable non-transactional writes on …
f3b2e77 
…COPY

Summary:
Allowing faster writes on copy command by using session variable "yb_force_non_transactional_writes".
The default value for this flag is false.
To enable the flag,
```
SET yb_force_non_transactional_writes=true
```
To disable the flag,
```
SET yb_force_non_transactional_writes=false
```
Note, the session variable must be applied on non-DDL operations as non-transactional writes on DDL operations is not supported.

Test Plan:
Verified using non transactional flag improves latency on release build.
Added Java unit test.

Reviewers: mihnea, smishra

Reviewed By: smishra

Subscribers: smishra, yql

Differential Revision: https://phabricator.dev.yugabyte.com/D13703
@emhna emhna closed this as completed Nov 30, 2021
emhna added a commit that referenced this issue Dec 2, 2021
@emhna
[Backport 2.8][#7809][YSQL] Add session flag to enable non-transactio…
8305278 
…nal writes on COPY

Summary:
Allowing faster writes on copy command by using session variable "yb_force_non_transactional_writes".
The default value for this flag is false.
To enable the flag,
```
SET yb_force_non_transactional_writes=true
```
To disable the flag,
```
SET yb_force_non_transactional_writes=false
```
Note, the session variable must be applied on non-DDL operations as non-transactional writes on DDL operations is not supported.

Test Plan: Jenkins: rebase: 2.8

Reviewers: mihnea, smishra

Reviewed By: smishra

Subscribers: yql, smishra

Differential Revision: https://phabricator.dev.yugabyte.com/D14195
@ymahajan ymahajan mentioned this issue Mar 15, 2022
Open
@FranckPachot FranckPachot mentioned this issue Jul 8, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@emhna emhna

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@psudheer21 @emhna