Regression in S3 Compatible APIs for Snapshot Repositories in 8.13

[iveelsm]

Elasticsearch Version

8.13.3

Installed Plugins

No response

Java Version

bundled

OS Version

Linux 31-prd-iad-elasticsearch 6.1.0-21-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64 GNU/Linux

Problem Description

#105044 introduced a regression in S3 Compatible APIs like R2. R2 does not support the use of x- parameters. Given the extremely specific nature of the x-purpose search parameter, either the documentation needs to be marked as not supporting S3 compatible services, or there needs to be another approach to address the custom log format for AWS.

Cloudflare error message (and stack trace) as an example:

[2024-06-03T23:35:14,506][ERROR][o.e.x.i.IndexLifecycleRunner] [redacted] policy [redacted] for index [redacted] failed on step [{"phase":"delete","action":"delete","name":"cleanup-snapshot"}]. Moving to ERROR step
java.io.IOException: Exception when listing blobs by prefix [null]
	at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:356) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobs(S3BlobContainer.java:362) ~[?:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$createSnapshotsDeletion$12(BlobStoreRepository.java:887) ~[elasticsearch-8.13.3.jar:?]
	at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:52) ~[elasticsearch-8.13.3.jar:?]
	at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:49) ~[elasticsearch-8.13.3.jar:?]
	at org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:95) ~[elasticsearch-8.13.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.13.3.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.13.3.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: ListObjectsV1 search parameter x-purpose not implemented (Service: Amazon S3; Status Code: 501; Error Code: NotImplemented; Request ID: null; S3 Extended Request ID: null; Proxy: null)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5397) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:928) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeListing$15(S3BlobContainer.java:403) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:31) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobContainer.executeListing(S3BlobContainer.java:403) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobContainer.listBlobsByPrefix(S3BlobContainer.java:347) ~[?:?]
	... 10 more

Steps to Reproduce

1. Use a non-AWS S3 compatible storage.

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

The AWS S3 docs say specifically that S3 ignores all query-string parameters prefixed with an x-, so any service claiming to be S3-compatible must do the same.

See also these docs:

Please do not report Elasticsearch issues involving storage systems which claim to be S3-compatible unless you can demonstrate that the same issue exists when using a genuine AWS S3 repository.