Add support for wrapping CollectorManager with profiling during concurrent execution #9129
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ticheng-aws
requested review from
reta,
anasalkouz,
andrross,
Bukhtawar,
CEHENKLE,
dblock,
gbbafna,
setiah,
kartg,
kotwanikunal,
mch2,
nknize,
owaiskazi19,
Rishikesh1159,
ryanbogan,
saratvemulapalli,
shwetathareja,
dreamer-89,
tlfeng,
VachaShah,
dbwiddis,
sachinpkale and
sohami
as code owners
ticheng-aws
force-pushed
the
cs-col-mngr-wrap
branch
from
ee875a2
to
f2d70fc
Compare
ticheng-aws
changed the title
|
Compatibility status: |
|
Compatibility status: |
Gradle Check (Jenkins) Run Completed with:
|
Codecov Report
@@ Coverage Diff @@
## main #9129 +/- ##
============================================
- Coverage 71.19% 71.08% -0.11%
+ Complexity 57455 57390 -65
============================================
Files 4777 4777
Lines 270712 270747 +35
Branches 39566 39569 +3
============================================
- Hits 192729 192458 -271
- Misses 61782 62089 +307
+ Partials 16201 16200 -1
|
Gradle Check (Jenkins) Run Completed with:
|
reta
reviewed
Aug 14, 2023
server/src/main/java/org/opensearch/search/aggregations/AggregationCollectorManager.java
Outdated
Show resolved
Hide resolved
ticheng-aws
added a commit
to ticheng-aws/OpenSearch
that referenced
this pull request
Aug 14, 2023
…-project#9129) Signed-off-by: Ticheng Lin <ticheng@amazon.com>
ticheng-aws
force-pushed
the
cs-col-mngr-wrap
branch
from
2bf382d
to
7536cfc
Compare
Gradle Check (Jenkins) Run Completed with:
|
ticheng-aws
added a commit
to ticheng-aws/OpenSearch
that referenced
this pull request
Aug 14, 2023
…rrent execution (opensearch-project#9129) Signed-off-by: Ticheng Lin <ticheng@amazon.com>
ticheng-aws
added a commit
to ticheng-aws/OpenSearch
that referenced
this pull request
Aug 14, 2023
…-project#9129) Signed-off-by: Ticheng Lin <ticheng@amazon.com>
ticheng-aws
force-pushed
the
cs-col-mngr-wrap
branch
from
7536cfc
to
eb7555f
Compare
reta
approved these changes
Aug 14, 2023
Gradle Check (Jenkins) Run Completed with:
|
sohami
approved these changes
Aug 15, 2023
|
Compatibility status: |
…rrent execution (opensearch-project#9129) Signed-off-by: Ticheng Lin <ticheng@amazon.com>
…-project#9129) Signed-off-by: Ticheng Lin <ticheng@amazon.com>
ticheng-aws
force-pushed
the
cs-col-mngr-wrap
branch
from
eb7555f
to
11b24c3
Compare
Compatibility status:Checks if related components are compatible with change 0a7eade Incompatible componentsIncompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/security-analytics.git] Skipped componentsCompatible componentsCompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git] |
Gradle Check (Jenkins) Run Completed with:
|
|
The backport to To backport manually, run these commands in your terminal: # Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-9129-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 152321b3adaa06199d741472baca964f90aff068
# Push it to GitHub
git push --set-upstream origin backport/backport-9129-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.xThen, create a pull request where the |
ticheng-aws
added a commit
to ticheng-aws/OpenSearch
that referenced
this pull request
Aug 15, 2023
…rrent execution (opensearch-project#9129) * Add support for wrapping CollectorManager with profiling during concurrent execution (opensearch-project#9129) Signed-off-by: Ticheng Lin <ticheng@amazon.com> * Add more collectorResult test and work on the PR comments (opensearch-project#9129) Signed-off-by: Ticheng Lin <ticheng@amazon.com> --------- Signed-off-by: Ticheng Lin <ticheng@amazon.com>
6 tasks
ticheng-aws
added a commit
to ticheng-aws/OpenSearch
that referenced
this pull request
Aug 15, 2023
…rrent execution (opensearch-project#9129) * Add support for wrapping CollectorManager with profiling during concurrent execution (opensearch-project#9129) Signed-off-by: Ticheng Lin <ticheng@amazon.com> * Add more collectorResult test and work on the PR comments (opensearch-project#9129) Signed-off-by: Ticheng Lin <ticheng@amazon.com> --------- Signed-off-by: Ticheng Lin <ticheng@amazon.com>
reta
added a commit
that referenced
this pull request
Aug 16, 2023
…rrent execution (#9129) (#9373) * Add support for wrapping CollectorManager with profiling during concurrent execution (#9129) * Add more collectorResult test and work on the PR comments (#9129) --------- Signed-off-by: Ticheng Lin <ticheng@amazon.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Andriy Redko <andriy.redko@aiven.io>
kkmr
pushed a commit
to kkmr/OpenSearch
that referenced
this pull request
Aug 28, 2023
…rrent execution (opensearch-project#9129) * Add support for wrapping CollectorManager with profiling during concurrent execution (opensearch-project#9129) Signed-off-by: Ticheng Lin <ticheng@amazon.com> * Add more collectorResult test and work on the PR comments (opensearch-project#9129) Signed-off-by: Ticheng Lin <ticheng@amazon.com> --------- Signed-off-by: Ticheng Lin <ticheng@amazon.com> Signed-off-by: Kiran Reddy <kkreddy@amazon.com>
kaushalmahi12
pushed a commit
to kaushalmahi12/OpenSearch
that referenced
this pull request
Sep 12, 2023
…rrent execution (opensearch-project#9129) * Add support for wrapping CollectorManager with profiling during concurrent execution (opensearch-project#9129) Signed-off-by: Ticheng Lin <ticheng@amazon.com> * Add more collectorResult test and work on the PR comments (opensearch-project#9129) Signed-off-by: Ticheng Lin <ticheng@amazon.com> --------- Signed-off-by: Ticheng Lin <ticheng@amazon.com> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>
brusic
pushed a commit
to brusic/OpenSearch
that referenced
this pull request
Sep 25, 2023
…rrent execution (opensearch-project#9129) * Add support for wrapping CollectorManager with profiling during concurrent execution (opensearch-project#9129) Signed-off-by: Ticheng Lin <ticheng@amazon.com> * Add more collectorResult test and work on the PR comments (opensearch-project#9129) Signed-off-by: Ticheng Lin <ticheng@amazon.com> --------- Signed-off-by: Ticheng Lin <ticheng@amazon.com> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>
shiv0408
pushed a commit
to Gaurav614/OpenSearch
that referenced
this pull request
Apr 25, 2024
…rrent execution (opensearch-project#9129) * Add support for wrapping CollectorManager with profiling during concurrent execution (opensearch-project#9129) Signed-off-by: Ticheng Lin <ticheng@amazon.com> * Add more collectorResult test and work on the PR comments (opensearch-project#9129) Signed-off-by: Ticheng Lin <ticheng@amazon.com> --------- Signed-off-by: Ticheng Lin <ticheng@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
In the multi-level aggregation profile with concurrent execution, the response’s collectors miss profiling aggregator information. This happens because the AggCollectorManager is not wrapped by the InternalProfileCollectorManager when the profile is enabled and concurrent execution takes place. To resolve the issue, we need to wrap all the sub-collectorManagers with the InternalProfileCollectorManager.
Example of a valid profile response with concurrent search enabled:
{ ... "profile": { "shards": [ { "id": "[3Z04lnEDTq6XiPVosv-tgA][index][0]", "inbound_network_time_in_millis": 0, "outbound_network_time_in_millis": 0, "searches": [ { "query": [...], "rewrite_time": 11001, "collector": [ { "name": "QueryCollectorManager", "reason": "search_multi", "time_in_nanos": 3238624, "reduce_time_in_nanos": 3601167, "max_slice_time_in_nanos": 2177665, "min_slice_time_in_nanos": 342251, "avg_slice_time_in_nanos": 1712583, "slice_count": 5, "children": [ { "name": "SimpleTopDocsCollectorManager", "reason": "search_top_hits", "time_in_nanos": 1538416, "reduce_time_in_nanos": 684458, "max_slice_time_in_nanos": 326084, "min_slice_time_in_nanos": 184292, "avg_slice_time_in_nanos": 265216, "slice_count": 5 }, { "name": "NonGlobalAggCollectorManager: [histo]", "reason": "aggregation", "time_in_nanos": 2255000, "reduce_time_in_nanos": 2838750, "max_slice_time_in_nanos": 1045209, "min_slice_time_in_nanos": 152250, "avg_slice_time_in_nanos": 826808, "slice_count": 5 } ] } ] }, { "query": [...], "rewrite_time": 2833, "collector": [ { "name": "GlobalAggCollectorManager: [global]", "reason": "aggregation_global", "time_in_nanos": 657625, "reduce_time_in_nanos": 399416, "max_slice_time_in_nanos": 503000, "min_slice_time_in_nanos": 28999, "avg_slice_time_in_nanos": 217357, "slice_count": 5 } ] } ], "aggregations": [...] } ] } }Related Issues
Resolves #8332 #7354
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.