searchable-snapshots:qa:azure various tests

[mayya-sharipova]

Not sure if this is just a temporary failure, or something must be done to fix it.

Affected tests:

• AzureSnapshotRepoTestKitIT
• AzureStorageCleanupThirdPartyTests
• AzureSearchableSnapshotsIT
• AzureRepositoriesMeteringIT

Build scan:
https://gradle-enterprise.elastic.co/s/zf3fsy7bw5wr4
https://gradle-enterprise.elastic.co/s/dkm4qbq6vccpq

Repro line:
./gradlew ':x-pack:plugin:searchable-snapshots:qa:azure:integTest' --tests "org.elasticsearch.xpack.searchablesnapshots.AzureSearchableSnapshotsIT.testClearCache" -Dtests.seed=2CD83CF739AF3A7C -Dtests.locale=nn-NO -Dtests.timezone=Asia/Srednekolymsk -Druntime.java=11

Reproduces locally?:
No

Applicable branches:
master, 7.13 and 7.14

Failure history:
12 times in the last 4 days

Failure excerpt:

{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[repository] path [master_searchable_snapshots_tests_2CD83CF739AF3A7C] is not accessible on master node"}],"type":"repository_verification_exception","reason":"[repository] path [master_searchable_snapshots_tests_2CD83CF739AF3A7C] is not accessible on master node","caused_by":{"type":"blob_storage_exception","reason":"Status code 403, "AuthenticationFailedServer failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:7b5c0312-d01e-00ac-0f2f-7f9b96000000\nTime:2021-07-22T19:30:03.1626424ZSigned expiry time [Tue, 20 Jul 2021 13:21:00 GMT] must be after signed start time [Thu, 22 Jul 2021 19:30:03 GMT]"","suppressed":[{"type":"exception","reason":"#block terminated with an error"}]}},"status":500}

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[fcofdez]

I think the SAS token we were using in these QA tests is expired.

[valeriy42]

Other failures

• 7.x https://gradle-enterprise.elastic.co/s/huk2u7q7g5era
• 7.14 https://gradle-enterprise.elastic.co/s/mksvlcvfwps3y

[mark-vieira]

I think the SAS token we were using in these QA tests is expired.

Who owns this? Do we need to request a new set of credentials from infra?

[fcofdez]

Who owns this? Do we need to request a new set of credentials from infra?

Yes, it's infra. I've opened an issue requesting a credentials rotation 👍

[fcofdez]

The credentials were refreshed last 29th of July and the issue should be solved now.