Promote the Data stream lifecycle feature to Technical Preview (#100187)

This releases the Data stream lifecycle feature as a Technical Preview feature. Data stream lifecycle, albeit in technical preview, will allow data streams to take advantage of a native simplified and resilient lifecycle implementation.
elastic · Oct 3, 2023 · 839afdc · 839afdc
1 parent ed69501
commit 839afdc
Show file tree

Hide file tree

Showing 15 changed files with 64 additions and 6 deletions.
diff --git a/docs/changelog/100187.yaml b/docs/changelog/100187.yaml
@@ -0,0 +1,17 @@
+pr: 100187
+summary: GA the data stream lifecycle
+area: Data streams
+type: "feature"
+issues: []
+highlight:
+  title: The data stream lifecycle is now in Technical Preview
+  body: "This marks the data stream lifecycle as available in Technical Preview.
+    Data streams will be able to take advantage of a built-in simplified and 
+    resilient lifecycle implementation. Data streams with a configured lifecycle will 
+    be automatically rolled over and tail merged (a forcemerge implementation that's
+    lightweight and only merges the long tail of small segments instead of the
+    whole shard). With the shard and index maintenance tasks being handled automatically
+    to ensure optimum performance, and trade-off between indexing and searching,
+    you'll be able to focus on the business related lifecycle aspects like data
+    retention."
+  notable: true
diff --git a/docs/reference/data-management.asciidoc b/docs/reference/data-management.asciidoc
@@ -26,6 +26,8 @@ To help you manage your data, {es} offers you:
 * <<data-stream-lifecycle, Data stream lifecycle>> which is the built-in lifecycle of data streams and addresses the most
 common lifecycle management needs.
 
+preview::["The built-in data stream lifecycle is in technical preview and may be changed or removed in a future release. Elastic will apply best effort to fix any issues, but this feature is not subject to the support SLA of official GA features."]
+
 **{ilm-init}** can be used to manage both indices and data streams and it allows you to:
 
 * Define the retention period of your data. The retention period is the minimum time your data will be stored in {es}.

diff --git a/docs/reference/data-streams/data-stream-apis.asciidoc b/docs/reference/data-streams/data-stream-apis.asciidoc
@@ -16,9 +16,13 @@ The following APIs are available for managing <<data-streams,data streams>>:
 The following APIs are available for managing the built-in lifecycle of data streams:
 
 * <<data-streams-put-lifecycle,Update data stream lifecycle>> 
+preview:[]
 * <<data-streams-get-lifecycle,Get data stream lifecycle>> 
+preview:[]
 * <<data-streams-delete-lifecycle,Delete data stream lifecycle>>
+preview:[]
 * <<data-streams-explain-lifecycle,Explain data stream lifecycle>>
+preview:[]
 
 The following API is available for <<tsds,time series data streams>>:
 

diff --git a/docs/reference/data-streams/lifecycle/apis/delete-lifecycle.asciidoc b/docs/reference/data-streams/lifecycle/apis/delete-lifecycle.asciidoc
@@ -4,6 +4,8 @@
 <titleabbrev>Delete Data Stream Lifecycle</titleabbrev>
 ++++
 
+preview:[]
+
 Deletes the lifecycle from a set of data streams.
 
 [[delete-lifecycle-api-prereqs]]

diff --git a/docs/reference/data-streams/lifecycle/apis/explain-lifecycle.asciidoc b/docs/reference/data-streams/lifecycle/apis/explain-lifecycle.asciidoc
@@ -4,6 +4,8 @@
 <titleabbrev>Explain Data Stream Lifecycle</titleabbrev>
 ++++
 
+preview:[]
+
 Retrieves the current data stream lifecycle status for one or more data stream backing indices.
 
 [[explain-lifecycle-api-prereqs]]

diff --git a/docs/reference/data-streams/lifecycle/apis/get-lifecycle.asciidoc b/docs/reference/data-streams/lifecycle/apis/get-lifecycle.asciidoc
@@ -4,6 +4,8 @@
 <titleabbrev>Get Data Stream Lifecycle</titleabbrev>
 ++++
 
+preview:[]
+
 Gets the lifecycle of a set of data streams.
 
 [[get-lifecycle-api-prereqs]]

diff --git a/docs/reference/data-streams/lifecycle/apis/put-lifecycle.asciidoc b/docs/reference/data-streams/lifecycle/apis/put-lifecycle.asciidoc
@@ -4,6 +4,8 @@
 <titleabbrev>Put Data Stream Lifecycle</titleabbrev>
 ++++
 
+preview:[]
+
 Configures the data stream lifecycle for the targeted data streams.
 
 [[put-lifecycle-api-prereqs]]

diff --git a/docs/reference/data-streams/lifecycle/index.asciidoc b/docs/reference/data-streams/lifecycle/index.asciidoc
@@ -2,6 +2,8 @@
 [[data-stream-lifecycle]]
 == Data stream lifecycle
 
+preview:[]
+
 A data stream lifecycle is the built-in mechanism data streams use to manage their lifecycle. It enables you to easily
 automate the management of your data streams according to your retention requirements. For example, you could configure
 the lifecycle to:

diff --git a/.../reference/data-streams/lifecycle/tutorial-manage-existing-data-stream.asciidoc b/.../reference/data-streams/lifecycle/tutorial-manage-existing-data-stream.asciidoc
@@ -2,6 +2,8 @@
 [[tutorial-manage-existing-data-stream]]
 === Tutorial: Update existing data stream
 
+preview:[]
+
 To update the lifecycle of an existing data stream you do the following actions:
 
 . <<set-lifecycle>>

diff --git a/docs/reference/data-streams/lifecycle/tutorial-manage-new-data-stream.asciidoc b/docs/reference/data-streams/lifecycle/tutorial-manage-new-data-stream.asciidoc
@@ -2,6 +2,8 @@
 [[tutorial-manage-new-data-stream]]
 === Tutorial: Create a data stream with a lifecycle
 
+preview:[]
+
 To create a data stream with a built-in lifecycle, follow these steps:
 
 . <<create-index-template-with-lifecycle>>
@@ -146,4 +148,4 @@ DELETE _index_template/my-index-template
 --------------------------------------------------
 // TEST[continued]
 
-//////////////////////////
+//////////////////////////
diff --git a/docs/reference/indices/get-data-stream.asciidoc b/docs/reference/indices/get-data-stream.asciidoc
@@ -153,6 +153,16 @@ Name of the backing index. For naming conventions, see
 `index_uuid`::
 (string)
 Universally unique identifier (UUID) for the index.
+
+`prefer_ilm`::
+(boolean)
+Functionality in preview:[]. Indicates if this index is configured to prefer {ilm} 
+when both {ilm-cap} and <<data-stream-lifecycle, Data stream lifecycle>> are configured to 
+manage this index.
+
+`managed_by`::
+(string)
+Functionality in preview:[]. Indicates the system that managed this index.
 =====
 
 `generation`::
@@ -206,6 +216,17 @@ NOTE: A data stream's backing indices may be assigned different lifecycle
 policies. To retrieve the lifecycle policy for individual backing indices,
 use the <<indices-get-settings,get index settings API>>.
 
+`next_generation_managed_by`::
+(string)
+Functionality in preview:[]. Indicates the system that will managed the next generation index
+(i.e. the next data stream write index).
+
+`prefer_ilm`::
+(boolean)
+Functionality in preview:[]. Indicates if the index template used to create the data 
+stream's backing indices is configured to prefer {ilm-cap} when both {ilm-cap} and 
+<<data-stream-lifecycle, Data stream lifecycle>> are configured to manage this index.
+
 `hidden`::
 (Boolean) If `true`, the data stream is <<multi-hidden,hidden>>.
 
@@ -225,7 +246,7 @@ cluster can not write into this data stream or change its mappings.
 
 `lifecycle`::
 (object)
-Contains the configuration for the data stream lifecycle management of this data stream.
+Functionality in preview:[]. Contains the configuration for the data stream lifecycle management of this data stream.
 +
 .Properties of `lifecycle`
 [%collapsible%open]

diff --git a/rest-api-spec/src/main/resources/rest-api-spec/api/indices.delete_data_lifecycle.json b/rest-api-spec/src/main/resources/rest-api-spec/api/indices.delete_data_lifecycle.json
@@ -4,7 +4,7 @@
       "url":"https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams-delete-lifecycle.html",
       "description":"Deletes the data stream lifecycle of the selected data streams."
     },
-    "stability":"stable",
+    "stability":"experimental",
     "visibility":"public",
     "headers":{
       "accept": [ "application/json"]

diff --git a/rest-api-spec/src/main/resources/rest-api-spec/api/indices.explain_data_lifecycle.json b/rest-api-spec/src/main/resources/rest-api-spec/api/indices.explain_data_lifecycle.json
@@ -4,7 +4,7 @@
       "url": "https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams-explain-lifecycle.html",
       "description": "Retrieves information about the index's current data stream lifecycle, such as any potential encountered error, time since creation etc."
     },
-    "stability": "stable",
+    "stability": "experimental",
     "visibility": "public",
     "headers": {
       "accept": [

diff --git a/rest-api-spec/src/main/resources/rest-api-spec/api/indices.get_data_lifecycle.json b/rest-api-spec/src/main/resources/rest-api-spec/api/indices.get_data_lifecycle.json
@@ -4,7 +4,7 @@
       "url":"https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams-get-lifecycle.html",
       "description":"Returns the data stream lifecycle of the selected data streams."
     },
-    "stability":"stable",
+    "stability":"experimental",
     "visibility":"public",
     "headers":{
       "accept": [ "application/json"]

diff --git a/rest-api-spec/src/main/resources/rest-api-spec/api/indices.put_data_lifecycle.json b/rest-api-spec/src/main/resources/rest-api-spec/api/indices.put_data_lifecycle.json
@@ -4,7 +4,7 @@
       "url":"https://www.elastic.co/guide/en/elasticsearch/reference/master/data-streams-put-lifecycle.html",
       "description":"Updates the data stream lifecycle of the selected data streams."
     },
-    "stability":"stable",
+    "stability":"experimental",
     "visibility":"public",
     "headers":{
       "accept": [ "application/json"]