[CI] Multiple invocations of repository_s3/20_repository_permanent_credentials can interact

[droberts195]

repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials failed with the following error:

Failure at [repository_s3/20_repository_permanent_credentials:64]: expected [2xx] status code but api [snapshot.create] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"invalid_snapshot_name_exception","reason":"[repository_permanent:snapshot-one] Invalid snapshot name [snapshot-one], snapshot with the same name already exists","stack_trace":"InvalidSnapshotNameException[[repository_permanent:snapshot-one] Invalid snapshot name [snapshot-one], snapshot with the same name already exists]

This does not trivially reproduce, but it looks like it could happen if there were two test runs in CI at the same time for the same branch. CI sets the environment variable amazon_s3_base_path={branch} and this is what the test uses for its "permanent" repository.

The full failure log is https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+third-party-tests/35/console

The repro command is:

./gradlew :plugins:repository-s3:integTestRunner \
  -Dtests.seed=1FD0A748E26205B \
  -Dtests.class=org.elasticsearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT \
  -Dtests.method="test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials}" \
  -Dtests.security.manager=true \
  -Dtests.locale=sv-SE \
  -Dtests.timezone=America/Lower_Princes

[elasticmachine]

Pinging @elastic/es-distributed

[droberts195]

I think this was actually caused by files hanging around from a failed test - closing

[danielmitterdorfer]

This has failed again in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.5+third-party-tests-s3/75/consoleFull (see also attached build log and cluster logs):

reproduction line:

./gradlew :plugins:repository-s3:integTestRunner \
  -Dtests.seed=8A2C040608DE71A1 \
  -Dtests.class=org.elasticsearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT \
  -Dtests.method="test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials}" \
  -Dtests.security.manager=true \
  -Dtests.locale=id \
  -Dtests.timezone=America/Dominica \
  -Dcompiler.java=11 \
  -Druntime.java=8 \
  -Dtests.rest.blacklist=repository_s3/30_repository_temporary_credentials/*,repository_s3/40_repository_ec2_credentials/*,repository_s3/50_repository_ecs_credentials/*

The error message Invalid snapshot name [snapshot-one], snapshot with the same name already exists seems to indicate that the snapshot is still present due to a previous failure?

[danielmitterdorfer]

After it has also failed in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.5+third-party-tests-s3/76/console (no new relevant info compared to the previous test failure, hence not including more detail), I've muted the test 53f9f2d (only on branch 6.5).

[tlrx]

Closed by #36302 which cleans up snapshots before executing the test, just in case a previous test run failed and left some snapshots in the test repository.

Test unmuted in branch 6.5 (fe265ae)