[CI] AzureStorageCleanupThirdPartyTests failing, possibly needs new set of credentials

[cbuescher]

CI Link

https://gradle-enterprise.elastic.co/s/sevrq5jszpe3w

Repro line

./gradlew ':plugins:repository-azure:azureThirdPartyTest' --tests "org.elasticsearch.repositories.azure.AzureStorageCleanupThirdPartyTests.testListChildren" \ -Dtests.seed=203404662522F345 \ -Dtests.locale=de-CH \ -Dtests.timezone=Europe/Podgorica \ -Druntime.java=8

Does it reproduce?

Didn't try

Applicable branches

7.17, 8.0, 8.1

Failure history

Several failures today

Failure excerpt

org.elasticsearch.repositories.azure.AzureStorageCleanupThirdPartyTests > testListChildren FAILED
RepositoryVerificationException[[test-repo] path [7.17_third_party_tests_203404662522F345] is not accessible on master node]; nested: BlobStorageException[Status code 403, "AuthorizationPermissionMismatchThis request is not authorized to perform this operation using this permission.
RequestId:3ebf5395-201e-0023-1ff7-27a758000000
Time:2022-02-22T14:22:00.3240871Z"];
at __randomizedtesting.SeedInfo.seed([203404662522F345:68BCA86A44A7FC4D]:0)

    Caused by:
    com.azure.storage.blob.models.BlobStorageException: Status code 403, "<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthorizationPermissionMismatch</Code><Message>This request is not authorized to perform this operation using this permission.
    RequestId:3ebf5395-201e-0023-1ff7-27a758000000
    Time:2022-02-22T14:22:00.3240871Z</Message></Error>"

[cbuescher]

This looks very similar to #75648, I'll see if I can open a similar issue for Infra to solve this.

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[cbuescher]

I opened https://github.com/elastic/infra/issues/34818 on the Infra side to see if they can check if this is indeed related to problems with old credentials.

[cbuescher]

These are probably also related, but the error looks slightly different: https://gradle-enterprise.elastic.co/s/euqoyry4qsqlq/console-log?task=:plugins:repository-azure:azureThirdPartyTest

org.elasticsearch.repositories.azure.AzureStorageCleanupThirdPartyTests > testCleanup FAILED
    RepositoryVerificationException[[test-repo] path [7.17_third_party_tests_3EE8C9A2B44BBD81] is not accessible on master node]; nested: BlobStorageException[Status code 404, "<?xml version="1.0" encoding="utf-8"?><Error><Code>ResourceNotFound</Code><Message>The specified resource does not exist.
    RequestId:96a48058-501e-004b-4bf7-27c1c8000000
    Time:2022-02-22T14:20:46.0735493Z</Message></Error>"];
        at __randomizedtesting.SeedInfo.seed([3EE8C9A2B44BBD81:6EC29A0CCDE62D34]:0)

        Caused by:
        com.azure.storage.blob.models.BlobStorageException: Status code 404, "<?xml version="1.0" encoding="utf-8"?><Error><Code>ResourceNotFound</Code><Message>The specified resource does not exist.
        RequestId:96a48058-501e-004b-4bf7-27c1c8000000
        Time:2022-02-22T14:20:46.0735493Z</Message></Error>"

The error code in the nested exception is slightly different and it doesn't mention 'AuthorizationPermissionMismatch'. The other build before is a "SAS" build, whatever that might mean to anybody.

[mark-vieira]

See @drewr's comment here.

@elastic/es-distributed I realize this is "delivery-esque" since it's infra but if any ES dev can update these credentials my preference is for the owning area team to be self-sufficient here.

[mark-vieira]

@fcofdez are you still having trouble updating these credentials? Do we need to reach back out to infra?

[fcofdez]

I've just updated the credentials, I'll keep an eye and close the issue once I verify that the suite is green again.

[dimitris-athanasiou]

There has been some more 403s here https://gradle-enterprise.elastic.co/s/jhuzprkhe7yau

[fcofdez]

That's strange, I'll take another look

[pugnascotia]

Another case here - https://gradle-enterprise.elastic.co/s/ekxpusn6yj2cg

[pugnascotia]

Possibly related failure in :x-pack:plugin:searchable-snapshots:qa:azure:integTest as well:

https://gradle-enterprise.elastic.co/s/5brgh22m64hrs/console-log?task=:x-pack:plugin:searchable-snapshots:qa:azure:integTest

The Azure response contains: <Code>AccountIsDisabled</Code>

[benwtrent]

Tests failed due to the missing repo, diggin in the logs I saw many instances of <Code>AccountIsDisabled</Code>.

Build: https://gradle-enterprise.elastic.co/s/blwy3eiqzpfuy

[fcofdez]

Sorry about the noise here, I thought that only sas tokens were expired. I renewed the regular keys too. The tests should pass now.

[benwtrent]

@fcofdez here is another failure. I am going to assume it doesn't have your commit fix yet in the build since that was merged so recently: https://gradle-enterprise.elastic.co/s/7s5v7fssadiam/console-log?task=:plugins:repository-azure:azureThirdPartyTest

[fcofdez]

Let me take another look @benwtrent, it's a credentials issue

[fcofdez]

I regenerated the SAS token and the tests pass locally, I'll keep an eye for failures on CI

[Luegg]

@fcofdez there is another failure that looks a lot like this issue: https://gradle-enterprise.elastic.co/s/qpf6oknyatqb2/console-log?task=:x-pack:plugin:repositories-metering-api:qa:azure:integTest