[CI] XPackRestIT.test {p0=ml/data_frame_analytics_crud/Test get given multiple analytics} failed

[imotov]

Build scan: https://gradle-enterprise.elastic.co/s/nsjgiug7abh6e

Repro line:

REPRODUCE WITH: ./gradlew ':x-pack:plugin:integTestRunner' --tests "org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=ml/data_frame_analytics_crud/Test get given multiple analytics}" \
  -Dtests.seed=CEAF7C47FCB79FC6 \
  -Dtests.security.manager=true \
  -Dtests.locale=el-CY \
  -Dtests.timezone=Jamaica \
  -Druntime.java=8 \
  -Dtests.rest.blacklist=getting_started/10_monitor_cluster_health/*

Reproduces locally?: No

Applicable branches: 7.9

Failure history: I couldn't find any other recent failures

Failure excerpt:

org.elasticsearch.client.ResponseException: method [DELETE], host [http://127.0.0.1:39947], URI [/_ml/data_frame/analytics/foo-2], status line [HTTP/1.1 503 Service Unavailable]
{"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503}
	at __randomizedtesting.SeedInfo.seed([CEAF7C47FCB79FC6:46FB439D524BF23E]:0)
	at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:302)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:272)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:278)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
	at org.elasticsearch.xpack.core.ml.integration.MlRestTestStateCleaner.deleteAllDataFrameAnalytics(MlRestTestStateCleaner.java:110)
	at org.elasticsearch.xpack.core.ml.integration.MlRestTestStateCleaner.clearMlMetadata(MlRestTestStateCleaner.java:32)
	at org.elasticsearch.xpack.test.rest.XPackRestIT.clearMlState(XPackRestIT.java:222)
	at org.elasticsearch.xpack.test.rest.XPackRestIT.cleanup(XPackRestIT.java:206)
	at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[benwtrent]

Looking over the test history, this only ever failed once

I do think that the .ml-stats index being created and allocated at the same time as test clean up could have caused this. The only jobs that created the .ml-stats index asynchronously were the inference tests. All the individual tests that did create the .ml-stats index asynchronously have since been moved out of the yaml tests.

closing