Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] DataStreamsClientYamlTestSuiteIT test {p0=data_stream/240_logs_ecs_mappings/Test Elastic Agent log ECS mappings} failing #97795

Closed
kingherc opened this issue Jul 19, 2023 · 6 comments · Fixed by #100407
Closed

[CI] DataStreamsClientYamlTestSuiteIT test {p0=data_stream/240_logs_ecs_mappings/Test Elastic Agent log ECS mappings} failing #97795

kingherc opened this issue Jul 19, 2023 · 6 comments · Fixed by #100407
Assignees
@gmarouli
Labels
:Data Management/Data streams Data streams and their lifecycles Team:Data Management Meta label for data/management team >test-failure Triaged test failures from CI

Comments

@kingherc
Copy link
Contributor

Build scan:
https://gradle-enterprise.elastic.co/s/56ydgs7cr5vyc/tests/:modules:data-streams:yamlRestTest/org.elasticsearch.datastreams.DataStreamsClientYamlTestSuiteIT/test%20%7Bp0=data_stream%2F240_logs_ecs_mappings%2FTest%20Elastic%20Agent%20log%20ECS%20mappings%7D

Reproduction line:

./gradlew ':modules:data-streams:yamlRestTest' --tests "org.elasticsearch.datastreams.DataStreamsClientYamlTestSuiteIT.test {p0=data_stream/240_logs_ecs_mappings/Test Elastic Agent log ECS mappings}" -Dtests.seed=C0FDAA9DB5113BE3 -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=es-UY -Dtests.timezone=AET -Druntime.java=20

Applicable branches:
8.9

Reproduces locally?:
Didn't try

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.datastreams.DataStreamsClientYamlTestSuiteIT&tests.test=test%20%7Bp0%3Ddata_stream/240_logs_ecs_mappings/Test%20Elastic%20Agent%20log%20ECS%20mappings%7D

Failure excerpt:

java.lang.AssertionError: Failure at [data_stream/240_logs_ecs_mappings:18]: expected [2xx] status code but api [indices.create_data_stream] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"no matching index template found for data stream [logs-generic-default]","stack_trace":"org.elasticsearch.ElasticsearchException$1: no matching index template found for data stream [logs-generic-default]\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:667)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:595)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.rest.RestResponse.build(RestResponse.java:176)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.rest.RestResponse.<init>(RestResponse.java:124)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.rest.RestResponse.<init>(RestResponse.java:103)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:55)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onFailure(NodeClient.java:170)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:217)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:60)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:72)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:60)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:72)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$2(TransportMasterNodeAction.java:231)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:127)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:60)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:132)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:60)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:72)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.lambda$notifyListenerFailed$1(AllocationActionListener.java:52)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.executeInContext(AllocationActionListener.java:58)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.notifyListenerFailed(AllocationActionListener.java:52)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener$1.onFailure(AllocationActionListener.java:84)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.AckedClusterStateUpdateTask.onFailure(AckedClusterStateUpdateTask.java:75)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$ExecutionResult.notifyFailure(MasterService.java:975)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$ExecutionResult.onClusterStateUnchanged(MasterService.java:938)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:245)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1622)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListener.run(ActionListener.java:382)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1619)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListener.run(ActionListener.java:382)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\nCaused by: java.lang.IllegalArgumentException: no matching index template found for data stream [logs-generic-default]\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.lookupTemplateForDataStream(MetadataCreateDataStreamService.java:307)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.createDataStream(MetadataCreateDataStreamService.java:221)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.createDataStream(MetadataCreateDataStreamService.java:168)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService$1.execute(MetadataCreateDataStreamService.java:90)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.execute(MasterService.java:550)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1039)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1004)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:232)\n\t... 11 more\n"}],"type":"illegal_argument_exception","reason":"no matching index template found for data stream [logs-generic-default]","stack_trace":"java.lang.IllegalArgumentException: no matching index template found for data stream [logs-generic-default]\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.lookupTemplateForDataStream(MetadataCreateDataStreamService.java:307)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.createDataStream(MetadataCreateDataStreamService.java:221)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.createDataStream(MetadataCreateDataStreamService.java:168)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService$1.execute(MetadataCreateDataStreamService.java:90)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.execute(MasterService.java:550)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1039)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1004)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:232)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1622)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListener.run(ActionListener.java:382)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1619)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListener.run(ActionListener.java:382)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\n"},"status":400}]

  at __randomizedtesting.SeedInfo.seed([C0FDAA9DB5113BE3:48A995471BED561B]:0)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:572)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:515)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
  at java.lang.reflect.Method.invoke(Method.java:578)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1623)

  Caused by: java.lang.AssertionError: expected [2xx] status code but api [indices.create_data_stream] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"no matching index template found for data stream [logs-generic-default]","stack_trace":"org.elasticsearch.ElasticsearchException$1: no matching index template found for data stream [logs-generic-default]\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:667)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:595)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.rest.RestResponse.build(RestResponse.java:176)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.rest.RestResponse.<init>(RestResponse.java:124)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.rest.RestResponse.<init>(RestResponse.java:103)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:55)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onFailure(NodeClient.java:170)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:217)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:60)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:72)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:60)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:72)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$2(TransportMasterNodeAction.java:231)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.acceptException(ActionListenerImplementations.java:127)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:60)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onFailure(ActionListenerImplementations.java:132)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:60)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:72)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.lambda$notifyListenerFailed$1(AllocationActionListener.java:52)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.executeInContext(AllocationActionListener.java:58)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener.notifyListenerFailed(AllocationActionListener.java:52)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.routing.allocation.allocator.AllocationActionListener$1.onFailure(AllocationActionListener.java:84)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.AckedClusterStateUpdateTask.onFailure(AckedClusterStateUpdateTask.java:75)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$ExecutionResult.notifyFailure(MasterService.java:975)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$ExecutionResult.onClusterStateUnchanged(MasterService.java:938)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:245)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1622)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListener.run(ActionListener.java:382)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1619)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListener.run(ActionListener.java:382)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\nCaused by: java.lang.IllegalArgumentException: no matching index template found for data stream [logs-generic-default]\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.lookupTemplateForDataStream(MetadataCreateDataStreamService.java:307)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.createDataStream(MetadataCreateDataStreamService.java:221)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.createDataStream(MetadataCreateDataStreamService.java:168)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService$1.execute(MetadataCreateDataStreamService.java:90)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.execute(MasterService.java:550)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1039)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1004)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:232)\n\t... 11 more\n"}],"type":"illegal_argument_exception","reason":"no matching index template found for data stream [logs-generic-default]","stack_trace":"java.lang.IllegalArgumentException: no matching index template found for data stream [logs-generic-default]\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.lookupTemplateForDataStream(MetadataCreateDataStreamService.java:307)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.createDataStream(MetadataCreateDataStreamService.java:221)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService.createDataStream(MetadataCreateDataStreamService.java:168)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.metadata.MetadataCreateDataStreamService$1.execute(MetadataCreateDataStreamService.java:90)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.execute(MasterService.java:550)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1039)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1004)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:232)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1622)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListener.run(ActionListener.java:382)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1619)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.action.ActionListener.run(ActionListener.java:382)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.server@8.9.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\n"},"status":400}]

    at org.junit.Assert.fail(Assert.java:88)
    at org.elasticsearch.test.rest.yaml.section.DoSection.execute(DoSection.java:372)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:552)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:515)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    at java.lang.reflect.Method.invoke(Method.java:578)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
    at java.lang.Thread.run(Thread.java:1623)
@kingherc kingherc added :Data Management/Data streams Data streams and their lifecycles Team:Data Management Meta label for data/management team >test-failure Triaged test failures from CI labels Jul 19, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@gmarouli gmarouli self-assigned this Jul 28, 2023
@gmarouli
Copy link
Contributor

This is a weird failure, I cannot reproduce it locally and I cannot figure out why would the template not be loaded byStackTemplateRegistry:

{
  "index_patterns": ["logs-*-*"],
  "priority": 100,
  "data_stream": {},
  "composed_of": [
    "logs-mappings",
    "logs-settings",
    "logs@custom",
    "ecs@dynamic_templates"
  ],
  "ignore_missing_component_templates": ["logs@custom"],
  "allow_auto_create": true,
  "_meta": {
    "description": "default logs template installed by x-pack",
    "managed": true
  },
  "version": ${xpack.stack.template.version}
}

I haven't found anything on the logs yet, but I will keep looking

@DaveCTurner
Copy link
Contributor

Two more like this today:

The problem is I think that the index templates get installed asynchronously after the node has started up, and we start running tests before that's all finished:

[2023-09-13T12:11:05,586][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding index template [metrics] for index patterns [metrics-*-*]
[2023-09-13T12:11:05,607][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding index template [synthetics] for index patterns [synthetics-*-*]
[2023-09-13T12:11:05,621][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding index template [.deprecation-indexing-template] for index patterns [.logs-deprecation.*]
[2023-09-13T12:11:05,677][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.monitoring-8-ilm-policy]
↓↓↓ TEST FAILS HERE ↓↓↓
[2023-09-13T12:11:05,687][INFO ][o.e.d.DataStreamsClientYamlTestSuiteIT] [test] Dump test yaml [/var/lib/jenkins/workspace/elastic+elasticsearch+main+multijob+platform-support-unix/modules/data-streams/build/resources/yamlRestTest/rest-api-spec/test/data_stream/200_logs_datastream_defaults.yml] on failure:
[2023-09-13T12:11:05,695][INFO ][o.e.d.DataStreamsClientYamlTestSuiteIT] [test] Stash dump on test failure [{
↑↑↑ TEST FAILS HERE ↑↑↑
[2023-09-13T12:11:05,767][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [behavioral_analytics-events-default_policy]
[2023-09-13T12:11:05,848][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [ml-size-based-ilm-policy]
[2023-09-13T12:11:05,909][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [metrics]
[2023-09-13T12:11:05,929][INFO ][o.e.x.m.MachineLearning  ] [test-cluster-0] Starting machine learning feature reset
[2023-09-13T12:11:05,950][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [7-days-default]
[2023-09-13T12:11:06,022][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [synthetics]
[2023-09-13T12:11:06,123][INFO ][o.e.x.c.t.IndexTemplateRegistry] [test-cluster-0] adding ingest pipeline behavioral_analytics-events-final_pipeline
[2023-09-13T12:11:06,123][INFO ][o.e.x.c.t.IndexTemplateRegistry] [test-cluster-0] adding ingest pipeline ent-search-generic-ingestion
[2023-09-13T12:11:06,124][INFO ][o.e.x.c.t.IndexTemplateRegistry] [test-cluster-0] adding ingest pipeline logs@json-message
[2023-09-13T12:11:06,130][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding component template [behavioral_analytics-events-settings]
[2023-09-13T12:11:06,197][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding index template [behavioral_analytics-events-default] for index patterns [behavioral_analytics-events-*]
[2023-09-13T12:11:06,239][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [30-days-default]
[2023-09-13T12:11:06,284][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [logs]
[2023-09-13T12:11:06,327][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [365-days-default]
[2023-09-13T12:11:06,364][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [180-days-default]
[2023-09-13T12:11:06,405][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [90-days-default]
[2023-09-13T12:11:06,447][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [slm-history-ilm-policy]
[2023-09-13T12:11:06,484][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [watch-history-ilm-policy-16]
[2023-09-13T12:11:06,525][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.fleet-file-tohost-meta-ilm-policy]
[2023-09-13T12:11:06,561][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.fleet-actions-results-ilm-policy]
[2023-09-13T12:11:06,593][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.deprecation-indexing-ilm-policy]
[2023-09-13T12:11:06,632][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.fleet-file-fromhost-data-ilm-policy]
[2023-09-13T12:11:06,667][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.fleet-file-fromhost-meta-ilm-policy]
[2023-09-13T12:11:06,703][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.fleet-file-tohost-data-ilm-policy]
[2023-09-13T12:11:06,828][INFO ][o.e.h.n.s.HealthNodeTaskExecutor] [test-cluster-0] Node [{test-cluster-0}{f-utzOFqQduw1rzY1XndlA}] is selected as the current health node.
[2023-09-13T12:11:06,951][INFO ][o.e.l.ClusterStateLicenseService] [test-cluster-0] license [85ae5aec-0054-4359-bf44-e2f047b533cb] mode [basic] - valid
[2023-09-13T12:11:06,953][INFO ][o.e.x.s.a.Realms         ] [test-cluster-0] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2023-09-13T12:11:07,097][INFO ][o.e.x.c.t.IndexTemplateRegistry] [test-cluster-0] adding ingest pipeline logs-default-pipeline
[2023-09-13T12:11:07,102][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding component template [logs-settings]
↓↓↓ INDEX TEMPLATE ADDED HERE ↓↓↓
[2023-09-13T12:11:07,157][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding index template [logs] for index patterns [logs-*-*]
↑↑↑ INDEX TEMPLATE ADDED HERE ↑↑↑
[2023-09-13T12:11:07,304][INFO ][o.e.x.m.MachineLearning  ] [test-cluster-0] Finished machine learning feature reset

@pgomulka
Copy link
Contributor

another one with the same exception mesage https://gradle-enterprise.elastic.co/s/2iawhjks6motq/tests/task/:modules:data-streams:yamlRestTest/details/org.elasticsearch.datastreams.DataStreamsClientYamlTestSuiteIT/test%20%7Bp0=data_stream%2F220_logs_default_pipeline%2FTest%20default%20logs-*-*%20pipeline%7D?top-execution=1

@luigidellaquila
Copy link
Contributor

One more failure today https://gradle-enterprise.elastic.co/s/igj572po64n6c/tests/task/:modules:data-streams:yamlRestTest/details/org.elasticsearch.datastreams.DataStreamsClientYamlTestSuiteIT/test%20%7Bp0=data_stream%2F240_logs_ecs_mappings%2FTest%20Elastic%20Agent%20log%20ECS%20mappings%7D?page=eyJvdXRwdXQiOnsiMCI6M319&top-execution=1

@danielmitterdorfer danielmitterdorfer mentioned this issue Sep 19, 2023
Merged
@masseyke
Copy link
Member

Is this a duplicate of #98163? That there's nothing that stops the server from coming up before all templates are ready?

@masseyke masseyke mentioned this issue Sep 19, 2023
Closed
@gmarouli gmarouli mentioned this issue Sep 20, 2023
Merged
@NEUpanning NEUpanning mentioned this issue Sep 21, 2023
Merged
This was referenced Sep 22, 2023
Closed
Closed
This was referenced Sep 26, 2023
Closed
Merged
@masseyke masseyke mentioned this issue Oct 2, 2023
Closed
@gmarouli gmarouli mentioned this issue Oct 6, 2023
Merged
elasticsearchmachine pushed a commit that referenced this issue Oct 11, 2023
@gmarouli
Fix remaining logs tests (#100407)
5dc7ccc 
In this PR we convert the remaining 3 yaml logs tests to java rest
tests. This should remove all the flaky tests from the data stream yaml
suite that's why we are reenabling it.

Closes: #99911 Closes:
#97795
gmarouli added a commit to gmarouli/elasticsearch that referenced this issue Oct 11, 2023
@gmarouli
Fix remaining logs tests (elastic#100407)
37a10ff 
In this PR we convert the remaining 3 yaml logs tests to java rest
tests. This should remove all the flaky tests from the data stream yaml
suite that's why we are reenabling it.

Closes: elastic#99911 Closes:
elastic#97795
elasticsearchmachine pushed a commit that referenced this issue Oct 11, 2023
@gmarouli
Fix remaining logs tests (#100407) (#100663)
6117368 
In this PR we convert the remaining 3 yaml logs tests to java rest
tests. This should remove all the flaky tests from the data stream yaml
suite that's why we are reenabling it.

Closes: #99911 Closes:
#97795
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gmarouli gmarouli

Labels
:Data Management/Data streams Data streams and their lifecycles Team:Data Management Meta label for data/management team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
7 participants
@masseyke @luigidellaquila @gmarouli @DaveCTurner @kingherc @pgomulka @elasticsearchmachine