Make MultiBucketConsumerService thread safe to use across slices during search #9047
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
neetikasinghal
requested review from
reta,
anasalkouz,
andrross,
Bukhtawar,
CEHENKLE,
dblock,
gbbafna,
setiah,
kartg,
kotwanikunal,
mch2,
nknize,
owaiskazi19,
Rishikesh1159,
ryanbogan,
saratvemulapalli,
shwetathareja,
dreamer-89,
tlfeng,
VachaShah,
dbwiddis,
sachinpkale and
sohami
as code owners
neetikasinghal
force-pushed
the
multi-bucket-3
branch
from
cd45582
to
2bafdef
Compare
Gradle Check (Jenkins) Run Completed with:
|
|
Compatibility status: |
|
Compatibility status: |
Gradle Check (Jenkins) Run Completed with:
|
sohami
reviewed
Aug 2, 2023
server/src/main/java/org/opensearch/search/aggregations/MultiBucketConsumerService.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/search/aggregations/MultiBucketConsumerService.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/search/aggregations/MultiBucketConsumerService.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/search/aggregations/MultiBucketConsumerService.java
Show resolved
Hide resolved
server/src/main/java/org/opensearch/search/aggregations/MultiBucketConsumerService.java
Outdated
Show resolved
Hide resolved
|
Compatibility status: |
sohami
approved these changes
Aug 3, 2023
@neetikasinghal There are some test failures. Can you please check those ? |
Gradle Check (Jenkins) Run Completed with:
|
reta
reviewed
Aug 3, 2023
server/src/main/java/org/opensearch/search/aggregations/MultiBucketConsumerService.java
Show resolved
Hide resolved
reta
approved these changes
Aug 3, 2023
Gradle Check (Jenkins) Run Completed with:
|
neetikasinghal
force-pushed
the
multi-bucket-3
branch
from
3d02f73
to
53bfcc2
Compare
@sohami thanks for calling this out. |
Gradle Check (Jenkins) Run Completed with:
|
Codecov Report
@@ Coverage Diff @@
## main #9047 +/- ##
============================================
+ Coverage 71.03% 71.10% +0.07%
+ Complexity 57314 57309 -5
============================================
Files 4765 4765
Lines 270357 270305 -52
Branches 39543 39534 -9
============================================
+ Hits 192043 192199 +156
+ Misses 62151 61966 -185
+ Partials 16163 16140 -23
|
|
@neetikasinghal Can you please rebase your branch ? |
…ng search Signed-off-by: Neetika Singhal <neetiks@amazon.com>
neetikasinghal
force-pushed
the
multi-bucket-3
branch
from
53bfcc2
to
4a20a20
Compare
@sohami Done |
Gradle Check (Jenkins) Run Completed with:
|
Known Flaky test: #9034 |
|
Compatibility status: |
|
The backport to To backport manually, run these commands in your terminal: # Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-9047-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 24595c9fbace3516e886137058f3a8b992937652
# Push it to GitHub
git push --set-upstream origin backport/backport-9047-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.xThen, create a pull request where the |
|
@neetikasinghal Backport failed. Can you please do it manually |
neetikasinghal
added a commit
to neetikasinghal/OpenSearch
that referenced
this pull request
Aug 4, 2023
…ng search (opensearch-project#9047) Signed-off-by: Neetika Singhal <neetiks@amazon.com> (cherry picked from commit 24595c9)
neetikasinghal
added a commit
to neetikasinghal/OpenSearch
that referenced
this pull request
Aug 4, 2023
…ng search (opensearch-project#9047) Signed-off-by: Neetika Singhal <neetiks@amazon.com> (cherry picked from commit 24595c9)
neetikasinghal
added a commit
to neetikasinghal/OpenSearch
that referenced
this pull request
Aug 4, 2023
…ng search (opensearch-project#9047) Signed-off-by: Neetika Singhal <neetiks@amazon.com>
6 tasks
kaushalmahi12
pushed a commit
to kaushalmahi12/OpenSearch
that referenced
this pull request
Sep 12, 2023
…ng search (opensearch-project#9047) Signed-off-by: Neetika Singhal <neetiks@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>
brusic
pushed a commit
to brusic/OpenSearch
that referenced
this pull request
Sep 25, 2023
…ng search (opensearch-project#9047) Signed-off-by: Neetika Singhal <neetiks@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>
shiv0408
pushed a commit
to Gaurav614/OpenSearch
that referenced
this pull request
Apr 25, 2024
…ng search (opensearch-project#9047) Signed-off-by: Neetika Singhal <neetiks@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
callCount variable needs to be made thread safe as there is a periodic check on the CircuitBreaker in the MultiConsumerService that checks if the CB has tripped. With Concurrent Search, for each shard, there are multiple slices running on different threads and having one instance of MultiBucketConsumer. Multiple threads within the same instance of MultiBucketConsumer will try to access the callCount variable, and as of current logic that checks for CB trip after every 1024 calls to the accept function, there is possibility that if the callCount is not made thread safe, the CB trip check might never happen.
Initializing callCount as a LongAdder variable making it thread safe and also having an additional volatile` variable that tries to trip the CB for the other threads in case for one of the threads CB has already tripped.
Related Issues
Resolves #7785
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.