[CI] AzureSearchableSnapshotsIT multiple tests failing

[salvatore-campagna]

Build scan:
https://gradle-enterprise.elastic.co/s/ulfmptxt5kc34/tests/:x-pack:plugin:searchable-snapshots:qa:azure:javaRestTest/org.elasticsearch.xpack.searchablesnapshots.AzureSearchableSnapshotsIT/testSnapshotOfSearchableSnapshot

Reproduction line:

./gradlew ':x-pack:plugin:searchable-snapshots:qa:azure:javaRestTest' --tests "org.elasticsearch.xpack.searchablesnapshots.AzureSearchableSnapshotsIT.testSnapshotOfSearchableSnapshot" -Dtests.seed=CBF41A9FF6EDBA17 -Dtests.locale=fr -Dtests.timezone=MST -Druntime.java=20

Applicable branches:
main

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.searchablesnapshots.AzureSearchableSnapshotsIT&tests.test=testSnapshotOfSearchableSnapshot

Failure excerpt:

org.elasticsearch.client.ResponseException: method [PUT], host [http://[::1]:45397], URI [_snapshot/repository?verify=true], status line [HTTP/1.1 500 Internal Server Error]
{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[repository] path [main_searchable_snapshots_tests_CBF41A9FF6EDBA17] is not accessible on master node"}],"type":"repository_verification_exception","reason":"[repository] path [main_searchable_snapshots_tests_CBF41A9FF6EDBA17] is not accessible on master node","caused_by":{"type":"blob_storage_exception","reason":"If you are using a StorageSharedKeyCredential, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate method call.\nIf you are using a SAS token, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate generateSas method call.\nPlease remember to disable 'Azure-Storage-Log-String-To-Sign' before going to production as this string can potentially contain PII.\nStatus code 403, \"<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:b48772b6-601e-005f-0425-6cda2a000000\nTime:2023-04-11T03:30:23.0437807Z</Message><AuthenticationErrorDetail>Signed expiry time [Tue, 28 Mar 2023 16:26:00 GMT] must be after signed start time [Tue, 11 Apr 2023 03:30:23 GMT]</AuthenticationErrorDetail></Error>\"","suppressed":[{"type":"exception","reason":"#block terminated with an error"}]}},"status":500}

  at __randomizedtesting.SeedInfo.seed([CBF41A9FF6EDBA17:57736F521BF8AA39]:0)
  at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347)
  at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313)
  at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288)
  at org.elasticsearch.test.rest.ESRestTestCase.registerRepository(ESRestTestCase.java:1724)
  at org.elasticsearch.test.rest.ESRestTestCase.registerRepository(ESRestTestCase.java:1715)
  at org.elasticsearch.xpack.searchablesnapshots.AbstractSearchableSnapshotsRestTestCase.runSearchableSnapshotsTest(AbstractSearchableSnapshotsRestTestCase.java:102)
  at org.elasticsearch.xpack.searchablesnapshots.AbstractSearchableSnapshotsRestTestCase.runSearchableSnapshotsTest(AbstractSearchableSnapshotsRestTestCase.java:86)
  at org.elasticsearch.xpack.searchablesnapshots.AbstractSearchableSnapshotsRestTestCase.runSearchableSnapshotsTest(AbstractSearchableSnapshotsRestTestCase.java:82)
  at org.elasticsearch.xpack.searchablesnapshots.AbstractSearchableSnapshotsRestTestCase.testSnapshotOfSearchableSnapshot(AbstractSearchableSnapshotsRestTestCase.java:353)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
  at java.lang.reflect.Method.invoke(Method.java:578)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.elasticsearch.test.cluster.local.LocalElasticsearchCluster$1.evaluate(LocalElasticsearchCluster.java:39)
  at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1623)

[pxsalehi]

This is the same issue as #94868 I think. I'll close this.