[CI] RepositoryAzureClientYamlTestSuiteIT, AzureRepositoriesMeteringIT, AzureSearchableSnapshotsIT, AzureStorageCleanupThridPartyTests failing

[alex-spies]

Seems like authentication against Azure failed. Multiple tests and test suites are affected.

  Tests with failures:
   - org.elasticsearch.xpack.repositories.metering.azure.AzureRepositoriesMeteringIT.testDeleteThenAddRepositoryWithTheSameName
   - org.elasticsearch.xpack.repositories.metering.azure.AzureRepositoriesMeteringIT.testStatsAreTracked
   - org.elasticsearch.xpack.repositories.metering.azure.AzureRepositoriesMeteringIT.testStatsAreStoredIntoANewCounterInstanceAfterRepoConfigUpdate
   - org.elasticsearch.xpack.repositories.metering.azure.AzureRepositoriesMeteringIT.testClearRepositoriesStats
   - org.elasticsearch.xpack.repositories.metering.azure.AzureRepositoriesMeteringIT.testStatsAreUpdatedAfterRepositoryOperations
   - org.elasticsearch.xpack.repositories.metering.azure.AzureRepositoriesMeteringIT.testStatsAreArchivedAfterRepositoryDeletion
  
  7 tests completed, 6 failed
  
  Tests with failures:
   - org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT.test {yaml=repository_azure/20_repository/Register a repository with a non existing client}
   - org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT.test {yaml=repository_azure/20_repository/Register a repository with a non existing container}
   - org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT.test {yaml=repository_azure/20_repository/Restore a non existing snapshot}
   - org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT.test {yaml=repository_azure/20_repository/Delete a non existing snapshot}
   - org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT.test {yaml=repository_azure/20_repository/teardown}
   - org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT.test {yaml=repository_azure/20_repository/Get a non existing snapshot}
   - org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT.test {yaml=repository_azure/20_repository/Snapshot/Restore with repository-azure}
  
  8 tests completed, 7 failed
  
  Tests with failures:
   - org.elasticsearch.xpack.snapshotbasedrecoveries.recovery.AzureSnapshotBasedRecoveryIT.testRecoveryUsingSnapshots
   
  1 test completed, 1 failed
  
  Tests with failures:
   - org.elasticsearch.xpack.searchablesnapshots.AzureSearchableSnapshotsIT.testCloseAndReopen
   - org.elasticsearch.xpack.searchablesnapshots.AzureSearchableSnapshotsIT.testSourceOnlyRepository
   - org.elasticsearch.xpack.searchablesnapshots.AzureSearchableSnapshotsIT.testSnapshotOfSearchableSnapshot
   - org.elasticsearch.xpack.searchablesnapshots.AzureSearchableSnapshotsIT.testClearCache
   - org.elasticsearch.xpack.searchablesnapshots.AzureSearchableSnapshotsIT.testStats
   - org.elasticsearch.xpack.searchablesnapshots.AzureSearchableSnapshotsIT.testQueryScript
   - org.elasticsearch.xpack.searchablesnapshots.AzureSearchableSnapshotsIT.testSearchResults
  
  7 tests completed, 7 failed

Build scan:
https://gradle-enterprise.elastic.co/s/jl746rixjgqq4/tests/:modules:repository-azure:yamlRestTest/org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT/test%20%7Byaml=repository_azure%2F20_repository%2FRegister%20a%20repository%20with%20a%20non%20existing%20client%7D

Reproduction line:

./gradlew ':modules:repository-azure:yamlRestTest' --tests "org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT.test {yaml=repository_azure/20_repository/Register a repository with a non existing client}" -Dtests.seed=B5ED234071B5EBDF -Dtests.locale=sr -Dtests.timezone=Asia/Macao -Druntime.java=21

Applicable branches:
main, 8.14, 8.13, 7.17

Reproduces locally?:
Didn't try

Failure history:
Failure dashboard for org.elasticsearch.repositories.azure.RepositoryAzureClientYamlTestSuiteIT#test {yaml=repository_azure/20_repository/Register a repository with a non existing client}

Failure excerpt:

java.lang.AssertionError: Failure at [repository_azure/20_repository:6]: expected [2xx] status code but api [snapshot.create_repository] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[repository] path [main_integration_tests] is not accessible on master node","stack_trace":"org.elasticsearch.repositories.RepositoryVerificationException: [repository] path [main_integration_tests] is not accessible on master node\nCaused by: com.azure.storage.blob.models.BlobStorageException: If you are using a StorageSharedKeyCredential, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate method call.\nIf you are using a SAS token, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate generateSas method call.\nPlease remember to disable 'Azure-Storage-Log-String-To-Sign' before going to production as this string can potentially contain PII.\nStatus code 403, \"<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:10b94790-801e-001a-1f33-950fc9000000\nTime:2024-04-23T04:05:22.0161018Z</Message><AuthenticationErrorDetail>Signed expiry time [Thu, 18 Apr 2024 12:24:00 GMT] must be after signed start time [Tue, 23 Apr 2024 04:05:22 GMT]</AuthenticationErrorDetail></Error>\"\n\tat java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)\n\tat com.azure.core@1.34.0/com.azure.core.implementation.http.rest.ResponseExceptionConstructorCache.invoke(ResponseExceptionConstructorCache.java:56)\n\tat com.azure.core@1.34.0/com.azure.core.implementation.http.rest.RestProxyBase.instantiateUnexpectedException(RestProxyBase.java:378)\n\tat com.azure.core@1.34.0/com.azure.core.implementation.http.rest.AsyncRestProxy.lambda$ensureExpectedStatus$1(AsyncRestProxy.java:115)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2400)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2196)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2070)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)\n\tat reactor.core@3.4.34/reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:137)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:137)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)\n\tat reactor.core@3.4.34/reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onNext(FluxRetryWhen.java:174)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2666)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:180)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onComplete(MonoFlatMapMany.java:260)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onComplete(FluxContextWrite.java:126)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMap$MapConditionalSubscriber.onComplete(FluxMap.java:275)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:85)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:128)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxHandle$HandleSubscriber.onComplete(FluxHandle.java:220)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMap$MapConditionalSubscriber.onComplete(FluxMap.java:275)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:128)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onComplete(FluxHandleFuseable.java:236)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onComplete(FluxContextWrite.java:126)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1840)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.onComplete(MonoCollectList.java:129)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)\n\tat reactor.netty.core@1.0.39/reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:415)\n\tat reactor.netty.core@1.0.39/reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:439)\n\tat reactor.netty.core@1.0.39/reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:493)\n\tat reactor.netty.http@1.0.39/reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:768)\n\tat reactor.netty.core@1.0.39/reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:114)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)\n\tat io.netty.codec@4.1.107.Final/io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)\n\tat io.netty.codec@4.1.107.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler@4.1.107.Final/io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475)\n\tat io.netty.handler@4.1.107.Final/io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338)\n\tat io.netty.handler@4.1.107.Final/io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387)\n\tat io.netty.codec@4.1.107.Final/io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)\n\tat io.netty.codec@4.1.107.Final/io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)\n\tat io.netty.codec@4.1.107.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat io.netty.common@4.1.107.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat io.netty.common@4.1.107.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:571)\n\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45)\n\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.executors.PrivilegedExecutor.lambda$execute$0(PrivilegedExecutor.java:27)\n\tat org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:917)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n\tSuppressed: java.lang.Exception: #block terminated with an error\n\t\tat reactor.core@3.4.34/reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:100)\n\t\tat reactor.core@3.4.34/reactor.core.publisher.Mono.block(Mono.java:1742)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.AzureBlobStore.lambda$executeSingleUpload$19(AzureBlobStore.java:474)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46)\n\t\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:571)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.AzureBlobStore.executeSingleUpload(AzureBlobStore.java:462)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:387)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:110)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlobAtomic(AzureBlobContainer.java:105)\n\t\tat org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2023)\n\t\tat org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.repositories.RepositoriesService$4.doRun(RepositoriesService.java:507)\n\t\tat org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)\n\t\tat org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\t... 3 more\n"}],"type":"repository_verification_exception","reason":"[repository] path [main_integration_tests] is not accessible on master node","caused_by":{"type":"blob_storage_exception","reason":"If you are using a StorageSharedKeyCredential, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate method call.\nIf you are using a SAS token, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate generateSas method call.\nPlease remember to disable 'Azure-Storage-Log-String-To-Sign' before going to production as this string can potentially contain PII.\nStatus code 403, \"<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:10b94790-801e-001a-1f33-950fc9000000\nTime:2024-04-23T04:05:22.0161018Z</Message><AuthenticationErrorDetail>Signed expiry time [Thu, 18 Apr 2024 12:24:00 GMT] must be after signed start time [Tue, 23 Apr 2024 04:05:22 GMT]</AuthenticationErrorDetail></Error>\"","stack_trace":"com.azure.storage.blob.models.BlobStorageException: If you are using a StorageSharedKeyCredential, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate method call.\nIf you are using a SAS token, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate generateSas method call.\nPlease remember to disable 'Azure-Storage-Log-String-To-Sign' before going to production as this string can potentially contain PII.\nStatus code 403, \"<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:10b94790-801e-001a-1f33-950fc9000000\nTime:2024-04-23T04:05:22.0161018Z</Message><AuthenticationErrorDetail>Signed expiry time [Thu, 18 Apr 2024 12:24:00 GMT] must be after signed start time [Tue, 23 Apr 2024 04:05:22 GMT]</AuthenticationErrorDetail></Error>\"\n\tat java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)\n\tat com.azure.core@1.34.0/com.azure.core.implementation.http.rest.ResponseExceptionConstructorCache.invoke(ResponseExceptionConstructorCache.java:56)\n\tat com.azure.core@1.34.0/com.azure.core.implementation.http.rest.RestProxyBase.instantiateUnexpectedException(RestProxyBase.java:378)\n\tat com.azure.core@1.34.0/com.azure.core.implementation.http.rest.AsyncRestProxy.lambda$ensureExpectedStatus$1(AsyncRestProxy.java:115)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2400)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2196)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2070)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)\n\tat reactor.core@3.4.34/reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:137)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:137)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)\n\tat reactor.core@3.4.34/reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onNext(FluxRetryWhen.java:174)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2666)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:180)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onComplete(MonoFlatMapMany.java:260)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onComplete(FluxContextWrite.java:126)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMap$MapConditionalSubscriber.onComplete(FluxMap.java:275)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:85)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:128)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxHandle$HandleSubscriber.onComplete(FluxHandle.java:220)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMap$MapConditionalSubscriber.onComplete(FluxMap.java:275)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:128)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onComplete(FluxHandleFuseable.java:236)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onComplete(FluxContextWrite.java:126)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1840)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.onComplete(MonoCollectList.java:129)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)\n\tat reactor.netty.core@1.0.39/reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:415)\n\tat reactor.netty.core@1.0.39/reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:439)\n\tat reactor.netty.core@1.0.39/reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:493)\n\tat reactor.netty.http@1.0.39/reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:768)\n\tat reactor.netty.core@1.0.39/reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:114)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)\n\tat io.netty.codec@4.1.107.Final/io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)\n\tat io.netty.codec@4.1.107.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler@4.1.107.Final/io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475)\n\tat io.netty.handler@4.1.107.Final/io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338)\n\tat io.netty.handler@4.1.107.Final/io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387)\n\tat io.netty.codec@4.1.107.Final/io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)\n\tat io.netty.codec@4.1.107.Final/io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)\n\tat io.netty.codec@4.1.107.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat io.netty.common@4.1.107.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat io.netty.common@4.1.107.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:571)\n\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45)\n\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.executors.PrivilegedExecutor.lambda$execute$0(PrivilegedExecutor.java:27)\n\tat org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:917)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n\tSuppressed: java.lang.Exception: #block terminated with an error\n\t\tat reactor.core@3.4.34/reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:100)\n\t\tat reactor.core@3.4.34/reactor.core.publisher.Mono.block(Mono.java:1742)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.AzureBlobStore.lambda$executeSingleUpload$19(AzureBlobStore.java:474)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46)\n\t\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:571)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.AzureBlobStore.executeSingleUpload(AzureBlobStore.java:462)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:387)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:110)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlobAtomic(AzureBlobContainer.java:105)\n\t\tat org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2023)\n\t\tat org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.repositories.RepositoriesService$4.doRun(RepositoriesService.java:507)\n\t\tat org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)\n\t\tat org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\t... 3 more\n","suppressed":[{"type":"exception","reason":"#block terminated with an error","stack_trace":"java.lang.Exception: #block terminated with an error\n\tat reactor.core@3.4.34/reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:100)\n\tat reactor.core@3.4.34/reactor.core.publisher.Mono.block(Mono.java:1742)\n\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.AzureBlobStore.lambda$executeSingleUpload$19(AzureBlobStore.java:474)\n\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:571)\n\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45)\n\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.AzureBlobStore.executeSingleUpload(AzureBlobStore.java:462)\n\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:387)\n\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:110)\n\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlobAtomic(AzureBlobContainer.java:105)\n\tat org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2023)\n\tat org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.repositories.RepositoriesService$4.doRun(RepositoriesService.java:507)\n\tat org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)\n\tat org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n"}]},"stack_trace":"org.elasticsearch.repositories.RepositoryVerificationException: [repository] path [main_integration_tests] is not accessible on master node\nCaused by: com.azure.storage.blob.models.BlobStorageException: If you are using a StorageSharedKeyCredential, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate method call.\nIf you are using a SAS token, and the server returned an error message that says 'Signature did not match', you can compare the string to sign with the one generated by the SDK. To log the string to sign, pass in the context key value pair 'Azure-Storage-Log-String-To-Sign': true to the appropriate generateSas method call.\nPlease remember to disable 'Azure-Storage-Log-String-To-Sign' before going to production as this string can potentially contain PII.\nStatus code 403, \"<?xml version=\"1.0\" encoding=\"utf-8\"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:10b94790-801e-001a-1f33-950fc9000000\nTime:2024-04-23T04:05:22.0161018Z</Message><AuthenticationErrorDetail>Signed expiry time [Thu, 18 Apr 2024 12:24:00 GMT] must be after signed start time [Tue, 23 Apr 2024 04:05:22 GMT]</AuthenticationErrorDetail></Error>\"\n\tat java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)\n\tat com.azure.core@1.34.0/com.azure.core.implementation.http.rest.ResponseExceptionConstructorCache.invoke(ResponseExceptionConstructorCache.java:56)\n\tat com.azure.core@1.34.0/com.azure.core.implementation.http.rest.RestProxyBase.instantiateUnexpectedException(RestProxyBase.java:378)\n\tat com.azure.core@1.34.0/com.azure.core.implementation.http.rest.AsyncRestProxy.lambda$ensureExpectedStatus$1(AsyncRestProxy.java:115)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:113)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2400)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2196)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2070)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)\n\tat reactor.core@3.4.34/reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:137)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxHide$SuppressFuseableSubscriber.onNext(FluxHide.java:137)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:299)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1839)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)\n\tat reactor.core@3.4.34/reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onNext(FluxRetryWhen.java:174)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$MonoInnerProducerBase.complete(Operators.java:2666)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:180)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onComplete(MonoFlatMapMany.java:260)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onComplete(FluxContextWrite.java:126)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMap$MapConditionalSubscriber.onComplete(FluxMap.java:275)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:85)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:128)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxHandle$HandleSubscriber.onComplete(FluxHandle.java:220)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMap$MapConditionalSubscriber.onComplete(FluxMap.java:275)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:128)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onComplete(FluxHandleFuseable.java:236)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onComplete(FluxContextWrite.java:126)\n\tat reactor.core@3.4.34/reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1840)\n\tat reactor.core@3.4.34/reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.onComplete(MonoCollectList.java:129)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)\n\tat reactor.core@3.4.34/reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)\n\tat reactor.netty.core@1.0.39/reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:415)\n\tat reactor.netty.core@1.0.39/reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:439)\n\tat reactor.netty.core@1.0.39/reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:493)\n\tat reactor.netty.http@1.0.39/reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:768)\n\tat reactor.netty.core@1.0.39/reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:114)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)\n\tat io.netty.codec@4.1.107.Final/io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)\n\tat io.netty.codec@4.1.107.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler@4.1.107.Final/io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475)\n\tat io.netty.handler@4.1.107.Final/io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338)\n\tat io.netty.handler@4.1.107.Final/io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387)\n\tat io.netty.codec@4.1.107.Final/io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)\n\tat io.netty.codec@4.1.107.Final/io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)\n\tat io.netty.codec@4.1.107.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)\n\tat io.netty.transport@4.1.107.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat io.netty.common@4.1.107.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat io.netty.common@4.1.107.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:571)\n\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45)\n\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.executors.PrivilegedExecutor.lambda$execute$0(PrivilegedExecutor.java:27)\n\tat org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:917)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n\tSuppressed: java.lang.Exception: #block terminated with an error\n\t\tat reactor.core@3.4.34/reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:100)\n\t\tat reactor.core@3.4.34/reactor.core.publisher.Mono.block(Mono.java:1742)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.AzureBlobStore.lambda$executeSingleUpload$19(AzureBlobStore.java:474)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.SocketAccess.lambda$doPrivilegedVoidException$0(SocketAccess.java:46)\n\t\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:571)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.SocketAccess.doPrivilegedVoidException(SocketAccess.java:45)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.AzureBlobStore.executeSingleUpload(AzureBlobStore.java:462)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.AzureBlobStore.writeBlob(AzureBlobStore.java:387)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlob(AzureBlobContainer.java:110)\n\t\tat org.elasticsearch.repository.azure@8.15.0-SNAPSHOT/org.elasticsearch.repositories.azure.AzureBlobContainer.writeBlobAtomic(AzureBlobContainer.java:105)\n\t\tat org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2023)\n\t\tat org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.repositories.RepositoriesService$4.doRun(RepositoriesService.java:507)\n\t\tat org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)\n\t\tat org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\t\t... 3 more\n"},"status":500}]

  at __randomizedtesting.SeedInfo.seed([B5ED234071B5EBDF:3DB91C9ADF498627]:0)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:610)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:544)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.Random...

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[alex-spies]

And a similar failure for AzureStorageCleanupThridPartyTests on 8.14: https://gradle-enterprise.elastic.co/s/dalel5inn6nfi

[alex-spies]

8.13 also affected: https://gradle-enterprise.elastic.co/s/dsunqlvt5dxmc

[alex-spies]

And 7.17: https://gradle-enterprise.elastic.co/s/bm2j5r5xd6gxs

[alex-spies]

Got another on main: https://gradle-enterprise.elastic.co/s/bc6cvshdeim7k

[alex-spies]

And another on 8.14: https://gradle-enterprise.elastic.co/s/orhkaur67t7xw

[volodk85]

According to history, it started to fail quite recently

[ldematte]

Several more failures today:

• https://gradle-enterprise.elastic.co/s/nb2lukpfc7v2a
• https://gradle-enterprise.elastic.co/s/5zfmz6xxe4nqc
• https://gradle-enterprise.elastic.co/s/fdo54ogwg3i4i
• https://gradle-enterprise.elastic.co/s/6nvedezsd2t3c
• https://gradle-enterprise.elastic.co/s/6d6dvx2j4jzvk
• https://gradle-enterprise.elastic.co/s/kywgbgxp46svm
• https://gradle-enterprise.elastic.co/s/c5r6ayxveuf2c
• https://gradle-enterprise.elastic.co/s/oqtcjcufku6ic

I'm reluctant to mute whole test suites, but this is causing incredible noise and I see AzureStorageCleanupThridPartyTests has been already muted. I'll let @elastic/es-distributed decide what to do.

[fcofdez]

It looks like we need to rotate the SAS token used for integration tests