[CI] AzureSnapshotRepoTestKitIT testRepositoryAnalysis failing - credential rotation needed

[andreidan]

Build scan:
https://gradle-enterprise.elastic.co/s/ernaqchxzkbgu/tests/:x-pack:plugin:snapshot-repo-test-kit:qa:azure:javaRestTest/org.elasticsearch.repositories.blobstore.testkit.AzureSnapshotRepoTestKitIT/testRepositoryAnalysis

Reproduction line:

./gradlew ':x-pack:plugin:snapshot-repo-test-kit:qa:azure:javaRestTest' --tests "org.elasticsearch.repositories.blobstore.testkit.AzureSnapshotRepoTestKitIT.testRepositoryAnalysis" -Dtests.seed=8CA21C868891943A -Dtests.locale=is -Dtests.timezone=Australia/Brisbane -Druntime.java=20

Applicable branches:
main, 8.7, 8.6

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.repositories.blobstore.testkit.AzureSnapshotRepoTestKitIT&tests.test=testRepositoryAnalysis

Failure excerpt:

org.elasticsearch.client.ResponseException: method [PUT], host [http://[::1]:40239], URI [_snapshot/repository?verify=true], status line [HTTP/1.1 500 Internal Server Error]
{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[repository] path [main_repository_test_kit_tests_8CA21C868891943A] is not accessible on master node"}],"type":"repository_verification_exception","reason":"[repository] path [main_repository_test_kit_tests_8CA21C868891943A] is not accessible on master node","caused_by":{"type":"blob_storage_exception","reason":"If you are using a StorageSharedKeyCredential, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate method call.\nIf you are using a SAS token, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate generateSas method call.\nPlease remember to disable 'Azure-Storage-Log-String-To-Sign' before going to production as this string can potentially contain PII.\nStatus code 403, \"<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:a74702a9-001e-0076-4f32-62e45e000000\nTime:2023-03-29T11:34:40.4247708Z</Message><AuthenticationErrorDetail>Signed expiry time [Tue, 28 Mar 2023 16:26:00 GMT] must be after signed start time [Wed, 29 Mar 2023 11:34:40 GMT]</AuthenticationErrorDetail></Error>\"","suppressed":[{"type":"exception","reason":"#block terminated with an error"}]}},"status":500}

  at __randomizedtesting.SeedInfo.seed([8CA21C868891943A:759FBDE8560AA7D3]:0)
  at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347)
  at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313)
  at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
  at org.elasticsearch.test.rest.ESRestTestCase.registerRepository(ESRestTestCase.java:1723)
  at org.elasticsearch.test.rest.ESRestTestCase.registerRepository(ESRestTestCase.java:1714)
  at org.elasticsearch.repositories.blobstore.testkit.AbstractSnapshotRepoTestKitRestTestCase.testRepositoryAnalysis(AbstractSnapshotRepoTestKitRestTestCase.java:27)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
  at java.lang.reflect.Method.invoke(Method.java:578)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.elasticsearch.test.cluster.local.LocalElasticsearchCluster$1.evaluate(LocalElasticsearchCluster.java:39)
  at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1623)

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[andreidan]

Same for AzureSnapshotBasedRecoveryIT, AzureSearchableSnapshotsIT, etc https://gradle-enterprise.elastic.co/s/ernaqchxzkbgu

[craigtaverner]

Many tests in AzureSearchableSnapshotsIT failing with the same error, for example:

• AzureSearchableSnapshotsIT.testCloseAndReopen
• AzureRepositoriesMeteringIT.testStatsAreUpdatedAfterRepositoryOperations

And the failure rate is quite high:

[pxsalehi]

waiting for access so I can rotate the credentials used in the CI.

[pxsalehi]

A lot of tests have been muted due to this. We should unmute all of them once we can change the credentials. I've closed the other issues but the list of muted tests are in this PR: #95139.

[davidkyle]

RepositoryAzureClientYamlTestSuiteIT is suffering from the same problem

https://gradle-enterprise.elastic.co/s/gnhe252w435le