Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pipeline: use Message value to reduce GC pressure #2415

Merged
merged 7 commits into from
Aug 2, 2021

Conversation

overvenus
Copy link
Member

@overvenus overvenus commented Jul 29, 2021

What problem does this PR solve?

Use Message value to reduce GC pressure

# master
goos: linux
goarch: amd64
pkg: github.com/pingcap/ticdc/pkg/pipeline
cpu: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
BenchmarkPipeline/BenchmarkPipeline/1_node(s)-40         	 1309424	       819.7 ns/op	      56 B/op	       2 allocs/op
BenchmarkPipeline/BenchmarkPipeline/2_node(s)-40         	  953646	      1214 ns/op	      80 B/op	       3 allocs/op
BenchmarkPipeline/BenchmarkPipeline/3_node(s)-40         	  739050	      1639 ns/op	     104 B/op	       4 allocs/op
BenchmarkPipeline/BenchmarkPipeline/4_node(s)-40         	  631210	      1953 ns/op	     128 B/op	       5 allocs/op
BenchmarkPipeline/BenchmarkPipeline/5_node(s)-40         	  535172	      2387 ns/op	     152 B/op	       6 allocs/op
BenchmarkPipeline/BenchmarkPipeline/6_node(s)-40         	  429428	      2819 ns/op	     176 B/op	       7 allocs/op
BenchmarkPipeline/BenchmarkPipeline/7_node(s)-40         	  404947	      3508 ns/op	     200 B/op	       8 allocs/op
BenchmarkPipeline/BenchmarkPipeline/8_node(s)-40         	  326889	      3714 ns/op	     224 B/op	       9 allocs/op
PASS
ok  	github.com/pingcap/ticdc/pkg/pipeline	11.001s

# this patch
goos: linux
goarch: amd64
pkg: github.com/pingcap/ticdc/pkg/pipeline
cpu: Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
BenchmarkPipeline/BenchmarkPipeline/1_node(s)-40         	 1643746	       732.4 ns/op	      48 B/op	       1 allocs/op
BenchmarkPipeline/BenchmarkPipeline/2_node(s)-40         	  972543	      1195 ns/op	      96 B/op	       2 allocs/op
BenchmarkPipeline/BenchmarkPipeline/3_node(s)-40         	  777621	      1634 ns/op	     144 B/op	       3 allocs/op
BenchmarkPipeline/BenchmarkPipeline/4_node(s)-40         	  648489	      1982 ns/op	     192 B/op	       4 allocs/op
BenchmarkPipeline/BenchmarkPipeline/5_node(s)-40         	  507319	      2488 ns/op	     240 B/op	       5 allocs/op
BenchmarkPipeline/BenchmarkPipeline/6_node(s)-40         	  395040	      2822 ns/op	     288 B/op	       6 allocs/op
BenchmarkPipeline/BenchmarkPipeline/7_node(s)-40         	  376586	      3316 ns/op	     336 B/op	       7 allocs/op
BenchmarkPipeline/BenchmarkPipeline/8_node(s)-40         	  318111	      3858 ns/op	     384 B/op	       8 allocs/op
PASS
ok  	github.com/pingcap/ticdc/pkg/pipeline	10.624s

Left: master
Right: this patch
image
image
image

Cc #2212

Check List

Tests

  • Unit test
  • Integration test

Related changes

  • Need to cherry-pick to the release branch

Release note

None

Signed-off-by: Neil Shen <overvenus@gmail.com>
@ti-chi-bot
Copy link
Member

ti-chi-bot commented Jul 29, 2021

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • amyangfei
  • liuzix

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@overvenus overvenus added subject/performance Denotes an issue or pull request is related to replication performance. needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. needs-cherry-pick-release-5.1 Should cherry pick this PR to release-5.1 branch. labels Jul 29, 2021
@ti-chi-bot ti-chi-bot added the release-note-none Denotes a PR that doesn't merit a release note. label Jul 29, 2021
@ti-chi-bot ti-chi-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 29, 2021
@overvenus overvenus requested review from Rustin170506 and amyangfei and removed request for zier-one July 29, 2021 08:16
Copy link
Contributor

@amyangfei amyangfei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a benchmark in the unit test to verify the optimization for pipeline memory allocation

Signed-off-by: Neil Shen <overvenus@gmail.com>
Signed-off-by: Neil Shen <overvenus@gmail.com>
Signed-off-by: Neil Shen <overvenus@gmail.com>
@overvenus
Copy link
Member Author

/run-all-tests

@overvenus
Copy link
Member Author

/hold

@ti-chi-bot ti-chi-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 30, 2021
@overvenus
Copy link
Member Author

/unhold

@ti-chi-bot ti-chi-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 30, 2021
@overvenus
Copy link
Member Author

/run-kafka-tests
/run-integration-tests

@ti-chi-bot ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Aug 2, 2021
@ti-chi-bot ti-chi-bot removed the status/LGT1 Indicates that a PR has LGTM 1. label Aug 2, 2021
@ti-chi-bot ti-chi-bot added the status/LGT2 Indicates that a PR has LGTM 2. label Aug 2, 2021
@amyangfei
Copy link
Contributor

/merge

@ti-chi-bot
Copy link
Member

This pull request has been accepted and is ready to merge.

Commit hash: 22d835f

@ti-chi-bot ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Aug 2, 2021
@ti-chi-bot ti-chi-bot merged commit c60bdb6 into pingcap:master Aug 2, 2021
ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Aug 2, 2021
Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #2439.

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #2440.

@ti-chi-bot
Copy link
Member

In response to a cherrypick label: new pull request created: #2441.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-cherry-pick-release-4.0 Should cherry pick this PR to release-4.0 branch. needs-cherry-pick-release-5.0 Should cherry pick this PR to release-5.0 branch. needs-cherry-pick-release-5.1 Should cherry pick this PR to release-5.1 branch. release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2. subject/performance Denotes an issue or pull request is related to replication performance.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants