Add additional logging for Azure repository stats #12079

saratvemulapalli · 2024-01-30T00:30:57Z

Description

Adding additional logging for Azure repository testRequestStats to dump the requests ended up on the blob store.
These logs will only be printed when the tests fails.

The intention is to understand the number of calls received by Azure repo. The logger only dumps these when there is a failure in CI, shouldnot really bloat our runs.

Background: Ran this test 7k times and it successfully passed. See #9859 (comment). Looking at the errors we definitely need more logs for the number of calls being made by the SDK and what we really received.

Sample logs:

  1> [2024-01-30T11:00:23,635][INFO ][o.o.c.r.a.AllocationService] [node_s1] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[index-no-merges][0]]]).
  1> [2024-01-30T11:00:23,745][INFO ][o.o.s.SnapshotsService   ] [node_s1] deleting snapshots [snapshot] from repository [u]
  1> [2024-01-30T11:00:23,866][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] GET /container?restype=container&comp=list&prefix=&delimiter=/&include=metadata
  1> [2024-01-30T11:00:23,980][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] GET /container?restype=container&comp=list&prefix=indices/&delimiter=/&include=metadata
  1> [2024-01-30T11:00:24,172][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] HEAD /container/index-0
  1> [2024-01-30T11:00:24,278][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] PUT /container/index-1
  1> [2024-01-30T11:00:24,384][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] PUT /container/index.latest
  1> [2024-01-30T11:00:24,570][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/index-0
  1> [2024-01-30T11:00:24,681][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/snap-ArzD5w73RHe2k00ZVn3C9A.dat
  1> [2024-01-30T11:00:24,683][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/index-0
  1> [2024-01-30T11:00:24,723][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/meta-ArzD5w73RHe2k00ZVn3C9A.dat
  1> [2024-01-30T11:00:24,833][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] GET /container?restype=container&comp=list&prefix=indices/BncAwVfxS_e51fLqhzjIqQ/
  1> [2024-01-30T11:00:24,945][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__TmGel7s5RsSmLDAWLVukwA
  1> [2024-01-30T11:00:24,945][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__-kcqFsxNShu1srufxxJZEw
  1> [2024-01-30T11:00:24,970][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2Findex-VhLeLRKxRUOFxVm7WEnyeA
  1> [2024-01-30T11:00:24,970][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__bc5nSe_HQpijW94eZ5yyNw
  1> [2024-01-30T11:00:24,971][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__UToqxFPEQwynYRQTJYMVKw
  1> [2024-01-30T11:00:24,971][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__K-gkZJjATreJxkUzv7oGqw
  1> [2024-01-30T11:00:24,972][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__QOPjyMOET0-FXKfEamZAhw
  1> [2024-01-30T11:00:24,972][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2Fsnap-ArzD5w73RHe2k00ZVn3C9A.dat
  1> [2024-01-30T11:00:24,973][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__RPDdKkaFS0Kr0Fmq3MvpGQ
  1> [2024-01-30T11:00:24,974][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__g_AlZPLhShKAsXlGI61e2A
  1> [2024-01-30T11:00:24,974][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__UaTIRPQ2S36AAwqmoXLk1w
  1> [2024-01-30T11:00:24,975][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__T_zGbpNDTyiz3bIDaLbHGQ
  1> [2024-01-30T11:00:24,975][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__1ue9RuigTnOPOnI4J5-GnQ
  1> [2024-01-30T11:00:24,975][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__2GANRHt8S1qi36v4QKSPzA
  1> [2024-01-30T11:00:24,976][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__vTRXd000QsOHkYkxHtljiA
  1> [2024-01-30T11:00:24,976][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__nLwJInPfSmCG39AvVULpWg
  1> [2024-01-30T11:00:24,982][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__P2t0ZLvgSV6JwBnmTRTy0A
  1> [2024-01-30T11:00:24,991][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2F0%2F__-Y9Dudt8QWuiyEo4rMXNTQ
  1> [2024-01-30T11:00:24,991][INFO ][AzureStatsLogger         ] [[HTTP-Dispatcher]] DELETE /container/indices%2FBncAwVfxS_e51fLqhzjIqQ%2Fmeta-4CA8V40BKtO0Cfxc0NC3.dat
  1> [2024-01-30T11:00:24,992][INFO ][o.o.s.SnapshotsService   ] [node_s1] snapshots [snapshot/ArzD5w73RHe2k00ZVn3C9A] deleted

Related Issues

#9859

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com>

github-actions · 2024-01-30T00:34:44Z

❌ Gradle check result for 7679601: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-01-30T00:49:18Z

Compatibility status:

Checks if related components are compatible with change ed90c5d

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/cross-cluster-replication.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/sql.git]

...nternalClusterTest/java/org/opensearch/repositories/azure/AzureBlobStoreRepositoryTests.java

deshsidd · 2024-01-30T19:34:00Z

Building failing? Please fix or retrigger the build

Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com>

saratvemulapalli · 2024-01-30T20:06:29Z

Building failing? Please fix or retrigger the build

Yeah pretty weird, I haven't seen this. Looks like a network problem, not sure if its the tool server or jenkins CI.
Re-triggered with a push.

Build file '/var/jenkins/workspace/gradle-check/search/buildSrc/build.gradle' line: 279

* What went wrong:
A problem occurred configuring project ':build-tools'.
> Could not create task ':build-tools:spotlessJava'.
   > java.io.IOException: Failed to load eclipse jdt formatter: java.lang.RuntimeException: java.net.SocketTimeoutException: timeout

github-actions · 2024-01-30T20:56:17Z

✅ Gradle check result for 1f13238: SUCCESS

codecov · 2024-01-30T20:57:08Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (cfcb128) 71.43% compared to head (1f13238) 71.37%.

❗ Current head 1f13238 differs from pull request most recent head ed90c5d. Consider uploading reports for the commit ed90c5d to get more accurate results

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #12079      +/-   ##
============================================
- Coverage     71.43%   71.37%   -0.07%     
+ Complexity    59527    59499      -28     
============================================
  Files          4925     4925              
  Lines        279513   279513              
  Branches      40646    40643       -3     
============================================
- Hits         199674   199494     -180     
- Misses        63226    63473     +247     
+ Partials      16613    16546      -67

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

...nternalClusterTest/java/org/opensearch/repositories/azure/AzureBlobStoreRepositoryTests.java

Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>

github-actions · 2024-01-31T20:56:27Z

✅ Gradle check result for ed90c5d: SUCCESS

* Adding logging for AzureStats Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com> * Debugging Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com> * Removing debugging lines Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com> * Addressing comments Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com> * Addressing feedback 2 Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com> --------- Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com> Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com> (cherry picked from commit 16c5257) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Adding logging for AzureStats * Debugging * Removing debugging lines * Addressing comments * Addressing feedback 2 --------- (cherry picked from commit 16c5257) Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com> Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…#12079) * Adding logging for AzureStats Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com> * Debugging Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com> * Removing debugging lines Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com> * Addressing comments Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com> * Addressing feedback 2 Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com> --------- Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com> Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>

…#12079) * Adding logging for AzureStats Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com> * Debugging Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com> * Removing debugging lines Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com> * Addressing comments Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com> * Addressing feedback 2 Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com> --------- Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com> Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

saratvemulapalli added 3 commits January 29, 2024 13:45

Adding logging for AzureStats

6473dcf

Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com>

Debugging

476df65

Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com>

Removing debugging lines

7679601

Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com>

saratvemulapalli added the skip-changelog label Jan 30, 2024

saratvemulapalli mentioned this pull request Jan 30, 2024

[BUG] org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testRequestStats is flaky #9859

Closed

andrross reviewed Jan 30, 2024

View reviewed changes

...nternalClusterTest/java/org/opensearch/repositories/azure/AzureBlobStoreRepositoryTests.java Outdated Show resolved Hide resolved

deshsidd approved these changes Jan 30, 2024

View reviewed changes

saratvemulapalli added 2 commits January 30, 2024 11:54

Merge branch 'main' into azure-flaky-test

ad069de

Addressing comments

1f13238

Signed-off-by: Sarat Vemulapalli <vemsarat@amazon.com>

saratvemulapalli requested a review from andrross January 30, 2024 21:19

saratvemulapalli added the backport 2.x Backport to 2.x branch label Jan 30, 2024

kotwanikunal approved these changes Jan 31, 2024

View reviewed changes

...nternalClusterTest/java/org/opensearch/repositories/azure/AzureBlobStoreRepositoryTests.java Outdated Show resolved Hide resolved

saratvemulapalli added 2 commits January 31, 2024 12:01

Addressing feedback 2

e71f7e8

Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>

Merge branch 'main' into azure-flaky-test

ed90c5d

kotwanikunal approved these changes Jan 31, 2024

View reviewed changes

kotwanikunal merged commit 16c5257 into opensearch-project:main Jan 31, 2024
28 checks passed

opensearch-trigger-bot bot mentioned this pull request Jan 31, 2024

[Backport 2.x] Add additional logging for Azure repository stats #12115

Merged

saratvemulapalli deleted the azure-flaky-test branch January 31, 2024 20:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add additional logging for Azure repository stats #12079

Add additional logging for Azure repository stats #12079

saratvemulapalli commented Jan 30, 2024 •

edited

Loading

github-actions bot commented Jan 30, 2024

github-actions bot commented Jan 30, 2024 •

edited

Loading

deshsidd commented Jan 30, 2024

saratvemulapalli commented Jan 30, 2024

github-actions bot commented Jan 30, 2024

codecov bot commented Jan 30, 2024 •

edited

Loading

github-actions bot commented Jan 31, 2024

Add additional logging for Azure repository stats #12079

Add additional logging for Azure repository stats #12079

Conversation

saratvemulapalli commented Jan 30, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Jan 30, 2024

github-actions bot commented Jan 30, 2024 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

deshsidd commented Jan 30, 2024

saratvemulapalli commented Jan 30, 2024

github-actions bot commented Jan 30, 2024

codecov bot commented Jan 30, 2024 • edited Loading

Codecov Report

github-actions bot commented Jan 31, 2024

saratvemulapalli commented Jan 30, 2024 •

edited

Loading

github-actions bot commented Jan 30, 2024 •

edited

Loading

codecov bot commented Jan 30, 2024 •

edited

Loading