Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TSDB][Downsampling] NullPointerException on downsampling a TSDB data stream #98792

Closed
dej611 opened this issue Aug 23, 2023 · 5 comments
Closed

[TSDB][Downsampling] NullPointerException on downsampling a TSDB data stream #98792

dej611 opened this issue Aug 23, 2023 · 5 comments
Assignees
@salvatore-campagna
Labels
>bug :StorageEngine/TSDB You know, for Metrics Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo)

Comments

@dej611
Copy link
Contributor

dej611 commented Aug 23, 2023

Elasticsearch Version

8.11.0

Installed Plugins

No response

Java Version

bundled

OS Version

MacOS

Problem Description

This problem surfaced since the beginning, but since #98587 it got much worst.

In Kibana there are some tests to check for TSDB downsampled data streams, which have been disabled before the bug that #98587 fixed, and now we're trying to restore them back: elastic/kibana#164410
The CI is reporting failures on the TSDB downsampling tests with errors related to a NullPointerException:

│ info Creating tsdb_index_2 data stream component template...
[00:17:48]               │ info [o.e.c.m.MetadataIndexTemplateService] [ftr] adding component template [tsdb_index_2_mapping]
[00:17:49]               │ info Updating tsdb_index_2 index template for TSDB...
[00:17:49]               │ info [o.e.c.m.MetadataIndexTemplateService] [ftr] adding index template [tsdb_index_2_index_template] for index patterns [tsdb_index_2]
[00:17:49]               │ info Creating tsdb_index_2 data stream index...
[00:17:49]               │ info [o.e.c.m.MetadataCreateIndexService] [ftr] [.ds-tsdb_index_2-2023.08.22-000001] creating index, cause [initialize_data_stream], templates [tsdb_index_2_index_template], shards [1]/[1]
[00:17:49]               │ info [o.e.c.m.MetadataCreateDataStreamService] [ftr] adding data stream [tsdb_index_2] with write index [.ds-tsdb_index_2-2023.08.22-000001], backing indices [], and aliases []
[00:17:49]               │ info [o.e.i.m.MapperService] [ftr] [.ds-tsdb_index_2-2023.08.22-000001] reloading search analyzers
[00:17:49]               │ info Adding 100 to tsdb_index_2 with starting time from Aug 22, 2023 @ 09:53:58.124 to Aug 22, 2023 @ 09:57:18.124
[00:17:49]               │ info Indexed 100 test data docs.
[00:17:49]               │ info Force a rollover for the data stream to get the backing "old_index"
[00:17:49]               │ info [o.e.c.m.MetadataCreateIndexService] [ftr] [.ds-tsdb_index_2-2023.08.22-000002] creating index, cause [rollover_data_stream], templates [tsdb_index_2_index_template], shards [1]/[1]
[00:17:49]               │ info [o.e.i.m.MapperService] [ftr] [.ds-tsdb_index_2-2023.08.22-000002] reloading search analyzers
[00:17:49]               │ info add write block to ".ds-tsdb_index_2-2023.08.22-000001" index...
[00:17:49]               │ info [o.e.c.m.MetadataIndexStateService] [ftr] adding [index.blocks.write] block to indices [[.ds-tsdb_index_2-2023.08.22-000001/AcqjMeeVR3mc8JPm-bV4dA]]
[00:17:49]               │ info [o.e.c.m.MetadataIndexStateService] [ftr] completed adding [index.blocks.write] block to indices [.ds-tsdb_index_2-2023.08.22-000001]
[00:17:49]               │ debg Wait 1s before running the downsampling to avoid a null_pointer_exception
[00:17:50]               │ info downsampling ".ds-tsdb_index_2-2023.08.22-000001" index...
[00:17:50]               │ info [r.suppressed] [ftr] path: /.ds-tsdb_index_2-2023.08.22-000001/_downsample/tsdb_index_2_downsampled, params: {target_index=tsdb_index_2_downsampled, index=.ds-tsdb_index_2-2023.08.22-000001} java.lang.NullPointerException: text
[00:17:50]               │      	at java.base/java.util.Objects.requireNonNull(Objects.java:259)
[00:17:50]               │      	at java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:2005)
[00:17:50]               │      	at java.base/java.time.OffsetDateTime.parse(OffsetDateTime.java:404)
[00:17:50]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.parseTimestamp(TransportDownsampleAction.java:490)
[00:17:50]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.createPersistentTaskParams(TransportDownsampleAction.java:481)
[00:17:50]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.performShardDownsampling(TransportDownsampleAction.java:375)
[00:17:50]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.lambda$masterOperation$4(TransportDownsampleAction.java:323)
[00:17:50]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:177)
[00:17:50]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction$1.taskSucceeded(TransportDownsampleAction.java:140)
[00:17:50]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction$1.taskSucceeded(TransportDownsampleAction.java:131)
[00:17:50]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.SimpleBatchedExecutor.lambda$execute$0(SimpleBatchedExecutor.java:73)
[00:17:50]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$ExecutionResult.onPublishSuccess(MasterService.java:929)
[00:17:50]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$4.onResponse(MasterService.java:374)
[00:17:50]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$4.onResponse(MasterService.java:369)
[00:17:50]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$RunAfterActionListener.onResponse(ActionListenerImplementations.java:260)
[00:17:50]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
[00:17:50]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.support.ThreadedActionListener$1.doRun(ThreadedActionListener.java:50)
[00:17:50]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
[00:17:50]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
[00:17:50]               │      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[00:17:50]               │      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[00:17:50]               │      	at java.base/java.lang.Thread.run(Thread.java:1623)
[00:17:50]               │ debg --- retry.tryForTime error: null_pointer_exception
[00:17:50]               │      	Root causes:
[00:17:50]               │      		null_pointer_exception: text
[00:17:50]               │ info [o.e.i.m.MapperService] [ftr] [tsdb_index_2_downsampled] reloading search analyzers
[00:17:50]               │ debg Data stream exists: true; old_index exists: true
[00:17:50]               │ debg Wait 1.5s before running the downsampling to avoid a null_pointer_exception
[00:17:52]               │ info downsampling ".ds-tsdb_index_2-2023.08.22-000001" index...
[00:17:52]               │ info [o.e.c.s.MasterService] [ftr] exception thrown by listener notifying of failure java.lang.NullPointerException: text
[00:17:52]               │      	at java.base/java.util.Objects.requireNonNull(Objects.java:259)
[00:17:52]               │      	at java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:2005)
[00:17:52]               │      	at java.base/java.time.OffsetDateTime.parse(OffsetDateTime.java:404)
[00:17:52]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.parseTimestamp(TransportDownsampleAction.java:490)
[00:17:52]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.createPersistentTaskParams(TransportDownsampleAction.java:481)
[00:17:52]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.performShardDownsampling(TransportDownsampleAction.java:375)
[00:17:52]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.lambda$masterOperation$5(TransportDownsampleAction.java:337)
[00:17:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
[00:17:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185)
[00:17:52]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction$DownsampleClusterStateUpdateTask.onFailure(TransportDownsampleAction.java:778)
[00:17:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$ExecutionResult.notifyFailure(MasterService.java:975)
[00:17:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$ExecutionResult.onClusterStateUnchanged(MasterService.java:938)
[00:17:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:245)
[00:17:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1626)
[00:17:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:368)
[00:17:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1623)
[00:17:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)
[00:17:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:368)
[00:17:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)
[00:17:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
[00:17:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
[00:17:52]               │      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[00:17:52]               │      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[00:17:52]               │      	at java.base/java.lang.Thread.run(Thread.java:1623)
[00:17:52]               │      	Suppressed: [tsdb_index_2_downsampled/3k6Lep8KRtKyjHGApvcH9g] org.elasticsearch.ResourceAlreadyExistsException: index [tsdb_index_2_downsampled/3k6Lep8KRtKyjHGApvcH9g] already exists
[00:17:52]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.validateIndexName(MetadataCreateIndexService.java:176)
[00:17:52]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.validate(MetadataCreateIndexService.java:1368)
[00:17:52]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:344)
[00:17:52]               │      		at org.elasticsearch.xpack.downsample.TransportDownsampleAction$3.execute(TransportDownsampleAction.java:751)
[00:17:52]               │      		at org.elasticsearch.xpack.downsample.TransportDownsampleAction$1.executeTask(TransportDownsampleAction.java:135)
[00:17:52]               │      		at org.elasticsearch.xpack.downsample.TransportDownsampleAction$1.executeTask(TransportDownsampleAction.java:131)
[00:17:52]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.SimpleBatchedExecutor.execute(SimpleBatchedExecutor.java:70)
[00:17:52]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1039)
[00:17:52]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1004)
[00:17:52]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:232)
[00:17:52]               │      		... 11 more
[00:17:52]               │      	Suppressed: [CIRCULAR REFERENCE: [tsdb_index_2_downsampled/3k6Lep8KRtKyjHGApvcH9g] org.elasticsearch.ResourceAlreadyExistsException: index [tsdb_index_2_downsampled/3k6Lep8KRtKyjHGApvcH9g] already exists]
[00:18:22]               │ info [o.e.c.s.MasterService] [ftr] exception thrown by listener notifying of failure java.lang.NullPointerException: text
[00:18:22]               │      	at java.base/java.util.Objects.requireNonNull(Objects.java:259)
[00:18:22]               │      	at java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:2005)
[00:18:22]               │      	at java.base/java.time.OffsetDateTime.parse(OffsetDateTime.java:404)
[00:18:22]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.parseTimestamp(TransportDownsampleAction.java:490)
[00:18:22]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.createPersistentTaskParams(TransportDownsampleAction.java:481)
[00:18:22]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.performShardDownsampling(TransportDownsampleAction.java:375)
[00:18:22]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.lambda$masterOperation$5(TransportDownsampleAction.java:337)
[00:18:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
[00:18:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185)
[00:18:22]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction$DownsampleClusterStateUpdateTask.onFailure(TransportDownsampleAction.java:778)
[00:18:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$ExecutionResult.notifyFailure(MasterService.java:975)
[00:18:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$ExecutionResult.onClusterStateUnchanged(MasterService.java:938)
[00:18:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:245)
[00:18:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1626)
[00:18:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:368)
[00:18:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1623)
[00:18:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)
[00:18:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:368)
[00:18:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)
[00:18:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
[00:18:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
[00:18:22]               │      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[00:18:22]               │      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[00:18:22]               │      	at java.base/java.lang.Thread.run(Thread.java:1623)
[00:18:22]               │      	Suppressed: [tsdb_index_2_downsampled/3k6Lep8KRtKyjHGApvcH9g] org.elasticsearch.ResourceAlreadyExistsException: index [tsdb_index_2_downsampled/3k6Lep8KRtKyjHGApvcH9g] already exists
[00:18:22]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.validateIndexName(MetadataCreateIndexService.java:176)
[00:18:22]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.validate(MetadataCreateIndexService.java:1368)
[00:18:22]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:344)
[00:18:22]               │      		at org.elasticsearch.xpack.downsample.TransportDownsampleAction$3.execute(TransportDownsampleAction.java:751)
[00:18:22]               │      		at org.elasticsearch.xpack.downsample.TransportDownsampleAction$1.executeTask(TransportDownsampleAction.java:135)
[00:18:22]               │      		at org.elasticsearch.xpack.downsample.TransportDownsampleAction$1.executeTask(TransportDownsampleAction.java:131)
[00:18:22]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.SimpleBatchedExecutor.execute(SimpleBatchedExecutor.java:70)
[00:18:22]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1039)
[00:18:22]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1004)
[00:18:22]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:232)
[00:18:22]               │      		... 11 more
[00:18:22]               │      	Suppressed: [CIRCULAR REFERENCE: [tsdb_index_2_downsampled/3k6Lep8KRtKyjHGApvcH9g] org.elasticsearch.ResourceAlreadyExistsException: index [tsdb_index_2_downsampled/3k6Lep8KRtKyjHGApvcH9g] already exists]
[00:18:52]               │ info [o.e.c.s.MasterService] [ftr] exception thrown by listener notifying of failure java.lang.NullPointerException: text
[00:18:52]               │      	at java.base/java.util.Objects.requireNonNull(Objects.java:259)
[00:18:52]               │      	at java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:2005)
[00:18:52]               │      	at java.base/java.time.OffsetDateTime.parse(OffsetDateTime.java:404)
[00:18:52]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.parseTimestamp(TransportDownsampleAction.java:490)
[00:18:52]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.createPersistentTaskParams(TransportDownsampleAction.java:481)
[00:18:52]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.performShardDownsampling(TransportDownsampleAction.java:375)
[00:18:52]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.lambda$masterOperation$5(TransportDownsampleAction.java:337)
[00:18:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
[00:18:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185)
[00:18:52]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction$DownsampleClusterStateUpdateTask.onFailure(TransportDownsampleAction.java:778)
[00:18:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$ExecutionResult.notifyFailure(MasterService.java:975)
[00:18:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$ExecutionResult.onClusterStateUnchanged(MasterService.java:938)
[00:18:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:245)
[00:18:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1626)
[00:18:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:368)
[00:18:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1623)
[00:18:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)
[00:18:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:368)
[00:18:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)
[00:18:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
[00:18:52]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
[00:18:52]               │      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[00:18:52]               │      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[00:18:52]               │      	at java.base/java.lang.Thread.run(Thread.java:1623)
[00:18:52]               │      	Suppressed: [tsdb_index_2_downsampled/3k6Lep8KRtKyjHGApvcH9g] org.elasticsearch.ResourceAlreadyExistsException: index [tsdb_index_2_downsampled/3k6Lep8KRtKyjHGApvcH9g] already exists
[00:18:52]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.validateIndexName(MetadataCreateIndexService.java:176)
[00:18:52]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.validate(MetadataCreateIndexService.java:1368)
[00:18:52]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:344)
[00:18:52]               │      		at org.elasticsearch.xpack.downsample.TransportDownsampleAction$3.execute(TransportDownsampleAction.java:751)
[00:18:52]               │      		at org.elasticsearch.xpack.downsample.TransportDownsampleAction$1.executeTask(TransportDownsampleAction.java:135)
[00:18:52]               │      		at org.elasticsearch.xpack.downsample.TransportDownsampleAction$1.executeTask(TransportDownsampleAction.java:131)
[00:18:52]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.SimpleBatchedExecutor.execute(SimpleBatchedExecutor.java:70)
[00:18:52]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1039)
[00:18:52]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1004)
[00:18:52]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:232)
[00:18:52]               │      		... 11 more
[00:18:52]               │      	Suppressed: [CIRCULAR REFERENCE: [tsdb_index_2_downsampled/3k6Lep8KRtKyjHGApvcH9g] org.elasticsearch.ResourceAlreadyExistsException: index [tsdb_index_2_downsampled/3k6Lep8KRtKyjHGApvcH9g] already exists]
[00:19:22]               │ info [o.e.c.s.MasterService] [ftr] exception thrown by listener notifying of failure java.lang.NullPointerException: text
[00:19:22]               │      	at java.base/java.util.Objects.requireNonNull(Objects.java:259)
[00:19:22]               │      	at java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:2005)
[00:19:22]               │      	at java.base/java.time.OffsetDateTime.parse(OffsetDateTime.java:404)
[00:19:22]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.parseTimestamp(TransportDownsampleAction.java:490)
[00:19:22]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.createPersistentTaskParams(TransportDownsampleAction.java:481)
[00:19:22]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.performShardDownsampling(TransportDownsampleAction.java:375)
[00:19:22]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.lambda$masterOperation$5(TransportDownsampleAction.java:337)
[00:19:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
[00:19:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185)
[00:19:22]               │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction$DownsampleClusterStateUpdateTask.onFailure(TransportDownsampleAction.java:778)
[00:19:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$ExecutionResult.notifyFailure(MasterService.java:975)
[00:19:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$ExecutionResult.onClusterStateUnchanged(MasterService.java:938)
[00:19:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:245)
[00:19:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1626)
[00:19:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:368)
[00:19:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1623)
[00:19:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)
[00:19:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:368)
[00:19:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)
[00:19:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
[00:19:22]               │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
[00:19:22]               │      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[00:19:22]               │      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[00:19:22]               │      	at java.base/java.lang.Thread.run(Thread.java:1623)
[00:19:22]               │      	Suppressed: [tsdb_index_2_downsampled/3k6Lep8KRtKyjHGApvcH9g] org.elasticsearch.ResourceAlreadyExistsException: index [tsdb_index_2_downsampled/3k6Lep8KRtKyjHGApvcH9g] already exists
[00:19:22]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.validateIndexName(MetadataCreateIndexService.java:176)
[00:19:22]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.validate(MetadataCreateIndexService.java:1368)
[00:19:22]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:344)
[00:19:22]               │      		at org.elasticsearch.xpack.downsample.TransportDownsampleAction$3.execute(TransportDownsampleAction.java:751)
[00:19:22]               │      		at org.elasticsearch.xpack.downsample.TransportDownsampleAction$1.executeTask(TransportDownsampleAction.java:135)
[00:19:22]               │      		at org.elasticsearch.xpack.downsample.TransportDownsampleAction$1.executeTask(TransportDownsampleAction.java:131)
[00:19:22]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.SimpleBatchedExecutor.execute(SimpleBatchedExecutor.java:70)
[00:19:22]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1039)
[00:19:22]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1004)
[00:19:22]               │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:232)
[00:19:22]               │      		... 11 more
[00:19:22]               │      	Suppressed: [CIRCULAR REFERENCE: [tsdb_index_2_downsampled/3k6Lep8KRtKyjHGApvcH9g] org.elasticsearch.ResourceAlreadyExistsException: index [tsdb_index_2_downsampled/3k6Lep8KRtKyjHGApvcH9g] already exists]
[00:19:31]               │ info [o.e.c.s.IndexScopedSettings] [ftr] [.ds-tsdb_index_2-2023.08.22-000001] updating [index.merge.policy.floor_segment] from [2mb] to [100mb]
[00:19:31]               │ info [o.e.c.s.IndexScopedSettings] [ftr] [.ds-tsdb_index_2-2023.08.22-000001] updating [index.merge.policy.merge_factor] from [32] to [16]
[00:19:31]               │ info [o.e.c.s.IndexScopedSettings] [ftr] [.ds-tsdb_index_2-2023.08.22-000001] updating [index.merge.policy.floor_segment] from [2mb] to [100mb]
[00:19:31]               │ info [o.e.c.s.IndexScopedSettings] [ftr] [.ds-tsdb_index_2-2023.08.22-000001] updating [index.merge.policy.merge_factor] from [32] to [16]
[00:19:31]               │ info [o.e.c.s.IndexScopedSettings] [ftr] [.ds-data_stream-2023.08.22-000001] updating [index.merge.policy.floor_segment] from [2mb] to [100mb]
[00:19:31]               │ info [o.e.c.s.IndexScopedSettings] [ftr] [.ds-data_stream-2023.08.22-000001] updating [index.merge.policy.merge_factor] from [32] to [16]
[00:19:31]               │ info [o.e.c.s.IndexScopedSettings] [ftr] [.ds-data_stream-2023.08.22-000001] updating [index.merge.policy.floor_segment] from [2mb] to [100mb]
[00:19:31]               │ info [o.e.c.s.IndexScopedSettings] [ftr] [.ds-data_stream-2023.08.22-000001] updating [index.merge.policy.merge_factor] from [32] to [16]
[00:19:31]               │ info [o.e.d.l.DataStreamLifecycleService] [ftr] Data stream lifecycle service successfully updated settings [[index.merge.policy.floor_segment, index.merge.policy.merge_factor]] for index index [.ds-tsdb_index_2-2023.08.22-000001]
[00:19:31]               │ info [o.e.d.l.DataStreamLifecycleService] [ftr] Data stream lifecycle service successfully updated settings [[index.merge.policy.floor_segment, index.merge.policy.merge_factor]] for index index [.ds-data_stream-2023.08.22-000001]
[00:19:52]               │ debg --- retry.tryForTime error: Request timed out

The log shows how the data stream being created, documents added, then rolled over to obtain an index to write block to perform a rollover before the downsampling: the first downsampling request fails with the NullPointerException: text error, then a new attempt is made with the same error, but this time the downsampled index seems to exists.

Steps to Reproduce

I've put together a simple script that performs the steps 10 times and it the error will randomly appear:

#! /bin/bash
for id in 001 002 003 004 005 006 007 008 009 010
do
    echo "DELETE all datastreams and configurations for tsdb_index_$id"
    curl --request DELETE --url http://elastic:changeme@localhost:9200/_data_stream/tsdb_index_$id?pretty
    curl --request DELETE --url http://elastic:changeme@localhost:9200/_data_stream/.ds-tsdb_index_$id-2023.08.23-000001-downsample
    curl --request DELETE --url http://elastic:changeme@localhost:9200/_index_template/tsdb_index_$id-index-template?pretty
    curl --request DELETE --url http://elastic:changeme@localhost:9200/_component_template/tsdb_index_$id-mappings?pretty

    echo "Create component template tsdb_index_$id"
    curl --request PUT \
            --url http://elastic:changeme@localhost:9200/_component_template/tsdb_index_$id-mappings?pretty \
            --header 'Content-Type: application/json' \
            --data "
        {
        \"template\": {
            \"settings\": {
                \"mode\": \"time_series\",
                \"routing_path\": [ \"sensor_id\", \"location\" ]
            },
            \"mappings\": {
                \"properties\": {
                    \"sensor_id\": {
                    \"type\": \"keyword\",
                    \"time_series_dimension\": true
                    },
                    \"location\": {
                    \"type\": \"keyword\",
                    \"time_series_dimension\": true
                    },
                    \"counter\": {
                    \"type\": \"double\",
                    \"time_series_metric\": \"counter\"
                    },
                    \"temperature\": {
                    \"type\": \"half_float\",
                    \"time_series_metric\": \"gauge\"
                    },
                    \"humidity\": {
                    \"type\": \"half_float\",
                    \"time_series_metric\": \"gauge\"
                    },
                    \"@timestamp\": {
                    \"type\": \"date\",
                    \"format\": \"strict_date_optional_time\"
                    }
                }
            }
        },
        \"_meta\": {
            \"description\": \"Testing TSDB stream component template\"
        }
    }
    "
    echo "Create index template tsdb_index_$id"
    curl --request PUT \
            --url http://elastic:changeme@localhost:9200/_index_template/tsdb_index_$id-index-template?pretty \
            --header 'Content-Type: application/json' \
            --data "
            {
            \"index_patterns\": [\"tsdb_index_$id\"],
            \"data_stream\": { },
            \"composed_of\": [ \"tsdb_index_$id-mappings\"],
            \"_meta\": {
                \"description\": \"Template for tsdb_index_$id\"
            }
            }
    "


    echo "Create tsdb_index_$id data stream"
    curl --request PUT \
            --url http://elastic:changeme@localhost:9200/_data_stream/tsdb_index_$id?pretty


    echo "Add documents to tsdb_index_$id"
    curl --request PUT \
            --url http://elastic:changeme@localhost:9200/tsdb_index_$id/_bulk?pretty \
            --header 'Content-Type: application/json' \
            --data "

        { \"create\":{ } }
        { \"@timestamp\": \"2023-08-23T11:21:17.000Z\", \"sensor_id\": \"HAL-000001\", \"location\": \"plains\", \"temperature\": 26.7,\"humidity\": 49.9, \"counter\": 1 }
        { \"create\":{ } }
        { \"@timestamp\": \"2023-08-23T11:21:18.000Z\", \"sensor_id\": \"SYKENET-000001\", \"location\": \"swamp\", \"temperature\": 32.4, \"humidity\": 88.9, \"counter\": 2 }
"

    echo "Rollover datastream tsdb_index_$id"
    curl --request POST \
            --url http://elastic:changeme@localhost:9200/tsdb_index_$id/_rollover/?pretty 


    echo "Block write index over .ds-tsdb_index_$id-2023.08.23-000001 index"
    curl --request PUT \
            --url http://elastic:changeme@localhost:9200/.ds-tsdb_index_$id-2023.08.23-000001/_block/write?pretty

    echo "Downsample .ds-tsdb_index_$id-2023.08.23-000001 into .ds-tsdb_index_$id-downsample"
    curl --request POST \
            --url http://elastic:changeme@localhost:9200/.ds-tsdb_index_$id-2023.08.23-000001/_downsample/.ds-tsdb_index_$id-downsample?pretty \
            --header 'Content-Type: application/json' \
            --data "
            {
            \"fixed_interval\": \"1h\"
            }
            "
done

Logs (if relevant)

No response
@dej611 dej611 added >bug needs:triage Requires assignment of a team area label labels Aug 23, 2023
@iverase iverase added :StorageEngine/TSDB You know, for Metrics and removed needs:triage Requires assignment of a team area label labels Aug 23, 2023
@elasticsearchmachine elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Aug 23, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytics-geo (Team:Analytics)

@martijnvg
Copy link
Member

Thanks for sharing this. This looks strange. The NPE suggest that the index doesn't the index.time_series.start_time or end_time setting, which is strange. Since this is required for tsdb indices.

@salvatore-campagna
Copy link
Contributor

For some reasons if I try to retrieve settings for the index before starting downsampling it is missing some time series required settings like start and end time

Retrieve index tsdb_index_001 settings
{
  ".ds-tsdb_index_001-2023.08.23-000001" : {
    "aliases" : { },
    "mappings" : {
      "_data_stream_timestamp" : {
        "enabled" : true
      },
      "properties" : {
        "@timestamp" : {
          "type" : "date",
          "format" : "strict_date_optional_time"
        },
        "counter" : {
          "type" : "double",
          "time_series_metric" : "counter"
        },
        "humidity" : {
          "type" : "half_float",
          "time_series_metric" : "gauge"
        },
        "location" : {
          "type" : "keyword",
          "time_series_dimension" : true
        },
        "sensor_id" : {
          "type" : "keyword",
          "time_series_dimension" : true
        },
        "temperature" : {
          "type" : "half_float",
          "time_series_metric" : "gauge"
        }
      }
    },
    "settings" : {
      "index" : {
        "mode" : "time_series",
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_hot"
            }
          }
        },
        "hidden" : "true",
        "number_of_shards" : "1",
        "blocks" : {
          "write" : "true"
        },
        "provided_name" : ".ds-tsdb_index_001-2023.08.23-000001",
        "creation_date" : "1692804896307",
        "number_of_replicas" : "1",
        "routing_path" : [
          "sensor_id",
          "location"
        ],
        "uuid" : "oq5hG6J5TROZXv2_cVcIgw",
        "version" : {
          "created" : "8110099"
        }
      }
    },
    "data_stream" : "tsdb_index_001"
  }
}

I used the following script

#!/bin/bash

TODAY=$(date '+%Y.%m.%d')
for ID in 001 002 003 004 005 006 007 008 009 010
do
    echo "DELETE all datastreams and configurations for tsdb_index_$ID"
    curl --request DELETE --url http://elastic-admin:elastic-password@localhost:9200/_data_stream/tsdb_index_$ID?pretty || true
    curl --request DELETE --url http://elastic-admin:elastic-password@localhost:9200/_data_stream/.ds-tsdb_index_$ID-$TODAY-000001-downsample || true
    curl --request DELETE --url http://elastic-admin:elastic-password@localhost:9200/_index_template/tsdb_index_$ID-index-template?pretty || true
    curl --request DELETE --url http://elastic-admin:elastic-password@localhost:9200/_component_template/tsdb_index_$ID-mappings?pretty || true

    echo "Create component template tsdb_index_$ID"
    curl --request PUT \
            --url http://elastic-admin:elastic-password@localhost:9200/_component_template/tsdb_index_$ID-mappings?pretty \
            --header 'Content-Type: application/json' \
            --data "
        {
        \"template\": {
            \"settings\": {
                \"mode\": \"time_series\",
                \"routing_path\": [ \"sensor_id\", \"location\" ]
            },
            \"mappings\": {
                \"properties\": {
                    \"sensor_id\": {
                    \"type\": \"keyword\",
                    \"time_series_dimension\": true
                    },
                    \"location\": {
                    \"type\": \"keyword\",
                    \"time_series_dimension\": true
                    },
                    \"counter\": {
                    \"type\": \"double\",
                    \"time_series_metric\": \"counter\"
                    },
                    \"temperature\": {
                    \"type\": \"half_float\",
                    \"time_series_metric\": \"gauge\"
                    },
                    \"humidity\": {
                    \"type\": \"half_float\",
                    \"time_series_metric\": \"gauge\"
                    },
                    \"@timestamp\": {
                    \"type\": \"date\",
                    \"format\": \"strict_date_optional_time\"
                    }
                }
            }
        },
        \"_meta\": {
            \"description\": \"Testing TSDB stream component template\"
        }
    }
    "
    echo "Create index template tsdb_index_$ID"
    curl --request PUT \
            --url http://elastic-admin:elastic-password@localhost:9200/_index_template/tsdb_index_$ID-index-template?pretty \
            --header 'Content-Type: application/json' \
            --data "
            {
            \"index_patterns\": [\"tsdb_index_$ID\"],
            \"data_stream\": { },
            \"composed_of\": [ \"tsdb_index_$ID-mappings\"],
            \"_meta\": {
                \"description\": \"Template for tsdb_index_$ID\"
            }
            }
    "


    echo "Create tsdb_index_$ID data stream"
    curl --request PUT \
            --url http://elastic-admin:elastic-password@localhost:9200/_data_stream/tsdb_index_$ID?pretty

    curl --request GET \
            --url http://elastic-admin:elastic-password@localhost:9200/_data_stream/tsdb_index_$ID?pretty


    echo "Add documents to tsdb_index_$ID"
    curl --request PUT \
            --url http://elastic-admin:elastic-password@localhost:9200/tsdb_index_$ID/_bulk?pretty \
            --header 'Content-Type: application/json' \
            --data "

        { \"create\":{ } }
        { \"@timestamp\": \"2023-08-23T11:21:17.000Z\", \"sensor_id\": \"HAL-000001\", \"location\": \"plains\", \"temperature\": 26.7,\"humidity\": 49.9, \"counter\": 1 }
        { \"create\":{ } }
        { \"@timestamp\": \"2023-08-23T11:21:18.000Z\", \"sensor_id\": \"SYKENET-000001\", \"location\": \"swamp\", \"temperature\": 32.4, \"humidity\": 88.9, \"counter\": 2 }
"

    echo "Rollover datastream tsdb_index_$ID"
    curl --request POST \
            --url http://elastic-admin:elastic-password@localhost:9200/tsdb_index_$ID/_rollover/?pretty


    echo "Block write index over .ds-tsdb_index_$ID-$TODAY-000001 index"
    curl --request PUT \
            --url http://elastic-admin:elastic-password@localhost:9200/.ds-tsdb_index_$ID-$TODAY-000001/_block/write?pretty

    echo "Retrieve index tsdb_index_$ID settings"
    curl --request GET \
            --url http://elastic-admin:elastic-password@localhost:9200/.ds-tsdb_index_$ID-$TODAY-000001?pretty

    echo "Downsample .ds-tsdb_index_$ID-$TODAY-000001 into .ds-tsdb_index_$ID-downsample"
    curl --request POST \
            --url http://elastic-admin:elastic-password@localhost:9200/.ds-tsdb_index_$ID-$TODAY-000001/_downsample/.ds-tsdb_index_$ID-downsample?pretty \
            --header 'Content-Type: application/json' \
            --data "
            {
            \"fixed_interval\": \"1h\"
            }
            "
done

This was referenced Aug 24, 2023
Open
Closed
@dej611
Copy link
Contributor Author

dej611 commented Aug 31, 2023

I've tried again with a recent ES snapshot to run the script above and now it hangs at first downsample task:

│ info [o.e.d.a.DeleteDataStreamTransportAction] [Marcos-MacBook-Pro.local] removing data stream [tsdb_index_001]
   │ info [o.e.c.m.MetadataDeleteIndexService] [Marcos-MacBook-Pro.local] [.ds-tsdb_index_001-2023.08.31-000001/KyDKsp3MTLack7dA_0LPfA] deleting index
   │ info [o.e.c.m.MetadataDeleteIndexService] [Marcos-MacBook-Pro.local] [.ds-tsdb_index_001-2023.08.31-000002/98_u2UvOQ4C2Xz2KiZixKA] deleting index
   │ info [o.e.c.m.MetadataIndexTemplateService] [Marcos-MacBook-Pro.local] removing index template [tsdb_index_001-index-template]
   │ info [o.e.c.m.MetadataIndexTemplateService] [Marcos-MacBook-Pro.local] removing component template [tsdb_index_001-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [Marcos-MacBook-Pro.local] adding component template [tsdb_index_001-mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [Marcos-MacBook-Pro.local] adding index template [tsdb_index_001-index-template] for index patterns [tsdb_index_001]
   │ info [o.e.c.m.MetadataCreateIndexService] [Marcos-MacBook-Pro.local] [.ds-tsdb_index_001-2023.08.31-000001] creating index, cause [initialize_data_stream], templates [tsdb_index_001-index-template], shards [1]/[1]
   │ info [o.e.c.m.MetadataCreateDataStreamService] [Marcos-MacBook-Pro.local] adding data stream [tsdb_index_001] with write index [.ds-tsdb_index_001-2023.08.31-000001], backing indices [], and aliases []
   │ info [o.e.i.m.MapperService] [Marcos-MacBook-Pro.local] [.ds-tsdb_index_001-2023.08.31-000001] reloading search analyzers
   │ info [o.e.c.m.MetadataCreateIndexService] [Marcos-MacBook-Pro.local] [.ds-tsdb_index_001-2023.08.31-000002] creating index, cause [rollover_data_stream], templates [tsdb_index_001-index-template], shards [1]/[1]
   │ info [o.e.i.m.MapperService] [Marcos-MacBook-Pro.local] [.ds-tsdb_index_001-2023.08.31-000002] reloading search analyzers
   │ info [o.e.c.m.MetadataIndexStateService] [Marcos-MacBook-Pro.local] adding [index.blocks.write] block to indices [[.ds-tsdb_index_001-2023.08.31-000001/h4yXRD9dQOumOoh-Ctff_Q]]
   │ info [o.e.c.m.MetadataIndexStateService] [Marcos-MacBook-Pro.local] completed adding [index.blocks.write] block to indices [.ds-tsdb_index_001-2023.08.31-000001]
   │ info [o.e.c.s.MasterService] [Marcos-MacBook-Pro.local] exception thrown by listener notifying of failure java.lang.NullPointerException: text
   │      	at java.base/java.util.Objects.requireNonNull(Objects.java:259)
   │      	at java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:2005)
   │      	at java.base/java.time.OffsetDateTime.parse(OffsetDateTime.java:404)
   │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.parseTimestamp(TransportDownsampleAction.java:490)
   │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.createPersistentTaskParams(TransportDownsampleAction.java:481)
   │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.performShardDownsampling(TransportDownsampleAction.java:375)
   │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction.lambda$masterOperation$5(TransportDownsampleAction.java:337)
   │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
   │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onFailure(ActionListener.java:185)
   │      	at org.elasticsearch.xpack.downsample.TransportDownsampleAction$DownsampleClusterStateUpdateTask.onFailure(TransportDownsampleAction.java:778)
   │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$ExecutionResult.notifyFailure(MasterService.java:975)
   │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$ExecutionResult.onClusterStateUnchanged(MasterService.java:938)
   │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:245)
   │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1626)
   │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:368)
   │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1623)
   │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)
   │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:368)
   │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)
   │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
   │      	at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
   │      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
   │      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
   │      	at java.base/java.lang.Thread.run(Thread.java:1623)
   │      	Suppressed: [.ds-tsdb_index_001-downsample/2CBVA_yxSqC_NXPDK2bWxA] org.elasticsearch.ResourceAlreadyExistsException: index [.ds-tsdb_index_001-downsample/2CBVA_yxSqC_NXPDK2bWxA] already exists
   │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.validateIndexName(MetadataCreateIndexService.java:176)
   │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.validate(MetadataCreateIndexService.java:1368)
   │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:344)
   │      		at org.elasticsearch.xpack.downsample.TransportDownsampleAction$3.execute(TransportDownsampleAction.java:751)
   │      		at org.elasticsearch.xpack.downsample.TransportDownsampleAction$1.executeTask(TransportDownsampleAction.java:135)
   │      		at org.elasticsearch.xpack.downsample.TransportDownsampleAction$1.executeTask(TransportDownsampleAction.java:131)
   │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.SimpleBatchedExecutor.execute(SimpleBatchedExecutor.java:70)
   │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1039)
   │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1004)
   │      		at org.elasticsearch.server@8.11.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:232)
   │      		... 11 more
   │      	Suppressed: [CIRCULAR REFERENCE: [.ds-tsdb_index_001-downsample/2CBVA_yxSqC_NXPDK2bWxA] org.elasticsearch.ResourceAlreadyExistsException: index [.ds-tsdb_index_001-downsample/2CBVA_yxSqC_NXPDK2bWxA] already exists]

On the script side I have this:

./tsdb-bug.sh
DELETE all datastreams and configurations for tsdb_index_001
{
  "acknowledged" : true
}
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [.ds-tsdb_index_001-2023.08.31-000001-downsample]","resource.type":"index_or_alias","resource.id":".ds-tsdb_index_001-2023.08.31-000001-downsample","index_uuid":"_na_","index":".ds-tsdb_index_001-2023.08.31-000001-downsample"}],"type":"index_not_found_exception","reason":"no such index [.ds-tsdb_index_001-2023.08.31-000001-downsample]","resource.type":"index_or_alias","resource.id":".ds-tsdb_index_001-2023.08.31-000001-downsample","index_uuid":"_na_","index":".ds-tsdb_index_001-2023.08.31-000001-downsample"},"status":404}{
  "acknowledged" : true
}
{
  "acknowledged" : true
}
Create component template tsdb_index_001
{
  "acknowledged" : true
}
Create index template tsdb_index_001
{
  "acknowledged" : true
}
Create tsdb_index_001 data stream
{
  "acknowledged" : true
}
Add documents to tsdb_index_001
{
  "errors" : false,
  "took" : 10,
  "items" : [
    {
      "create" : {
        "_index" : ".ds-tsdb_index_001-2023.08.31-000001",
        "_id" : "MjowXVEA-BHncbFFAAABikwvNcg",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 0,
        "_primary_term" : 1,
        "status" : 201
      }
    },
    {
      "create" : {
        "_index" : ".ds-tsdb_index_001-2023.08.31-000001",
        "_id" : "crxuhC8WO3aVdhvtAAABikwvObA",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 1,
        "_primary_term" : 1,
        "status" : 201
      }
    }
  ]
}
Rollover datastream tsdb_index_001
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "old_index" : ".ds-tsdb_index_001-2023.08.31-000001",
  "new_index" : ".ds-tsdb_index_001-2023.08.31-000002",
  "rolled_over" : true,
  "dry_run" : false,
  "conditions" : { }
}
Block write index over .ds-tsdb_index_001-2023.08.31-000001 index
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "indices" : [
    {
      "name" : ".ds-tsdb_index_001-2023.08.31-000001",
      "blocked" : true
    }
  ]
}
Downsample .ds-tsdb_index_001-2023.08.31-000001 into .ds-tsdb_index_001-downsample
...it hangs here...

@martijnvg
Copy link
Member

Fixed by #98840.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@salvatore-campagna salvatore-campagna

Labels
>bug :StorageEngine/TSDB You know, for Metrics Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@martijnvg @dej611 @iverase @elasticsearchmachine @salvatore-campagna