From 170cc7b8406c84f4c887dcc61071ad8944fa1b93 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Tue, 26 Oct 2021 17:38:45 -0400 Subject: [PATCH] [DOCS] Add docs for restoring to new cluster (#79683) (#79875) When restoring a snapshot to a new cluster, users may expect the cluster to not contain any conflicting indices or data streams. However, some features, such as the GeoIP processor, automatically create indices at startup. This adds and updates related procedures in the restore a snapshot tutorial. I plan to improve other documentation related to feature states in snapshots in a separate PR(s). This PR also updates the restore snapshot API's example to include the `indices` and `feature_states` parameters. Relates to #79675 --- .../apis/restore-snapshot-api.asciidoc | 6 + .../restore-snapshot.asciidoc | 198 ++++++++++++++---- 2 files changed, 168 insertions(+), 36 deletions(-) diff --git a/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc b/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc index af1fa20c9c53a..e077bf16a3765 100644 --- a/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc +++ b/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc @@ -53,8 +53,14 @@ POST /index_4/_close?wait_for_active_shards=0 [source,console] ---- POST /_snapshot/my_repository/my_snapshot/_restore +{ + "indices": "*,-.*", + "feature_states": [ "geoip" ] +} ---- // TEST[s/_restore/_restore?wait_for_completion=true/] +// TEST[s/",/"/] +// TEST[s/"feature_states": \[ "geoip" \]//] [[restore-snapshot-api-request]] ==== {api-request-title} diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 16177d011ceba..dd233f9201c7f 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -10,6 +10,7 @@ In this guide, you'll learn how to: * Get a list of available snapshots * Restore an index or data stream from a snapshot +* Restore a feature state * Restore an entire cluster * Monitor the restore operation * Cancel an ongoing restore @@ -33,7 +34,7 @@ When restoring data from a snapshot, keep the following in mind: * If you restore a data stream, you also restore its backing indices. * You can only restore an existing index if it's <> and -the index in the snapshot has the same number of primary shards. +the index in the snapshot has the same number of primary shards. * You can't restore an existing open index. This includes backing indices for a data stream. @@ -79,15 +80,21 @@ GET _snapshot/my_repository/*?verbose=false You can restore a snapshot using {kib}'s *Snapshot and Restore* feature or the <>. -In most cases, you only need to restore a specific index or data stream from a -snapshot. However, you can't restore an existing open index. +By default, a restore request attempts to restore all indices and data streams +in a snapshot, including <>. In most cases, you only need to restore a specific index or data +stream from a snapshot. However, you can't restore an existing open index. -To avoid conflicts with existing indices and data streams, use one of the -following methods: +If you're restoring data to a pre-existing cluster, use one of the +following methods to avoid conflicts with existing indices and data streams: * <> * <> +Some {es} features automatically create system indices on cluster startup. To +avoid conflicts with these system indices when restoring data to a new cluster, +see <>. + [discrete] [[delete-restore]] ==== Delete and restore @@ -113,18 +120,12 @@ DELETE _data_stream/logs-my_app-default ---- // TEST[setup:setup-snapshots] -By default, a restore request attempts to restore all indices and data -streams in the snapshot, including system indices. If your cluster already -contains one or more of these system indices, the request will return an error. - -To avoid this error, specify the indices and data streams to restore. To exclude -system indices and other index names that begin with a dot (`.`), append the -`-.*` wildcard pattern. To restore all indices and data streams except dot -indices, use `*,-.*`. +In the restore request, explicitly specify any indices and data streams to +restore. [source,console] ---- -POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore +POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore { "indices": "my-index,logs-my_app-default" } @@ -149,7 +150,7 @@ any restored index or data stream. [source,console] ---- -POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore +POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore { "indices": "my-index,logs-my_app-default", "rename_pattern": "(.+)", @@ -205,17 +206,94 @@ POST _reindex ---- // TEST[continued] +[discrete] +[[restore-exclude-system-indices]] +==== Exclude system indices + +Some {es} features, such as the <>, +automatically create system indices at startup. To avoid naming conflicts with +these indices, use the `-.*` wildcard pattern to exclude system indices and +other dot (`.`) indices from your restore request. + +For example, the following request uses the `*,-.*` wildcard pattern to restore +all indices and data streams except dot indices. + +[source,console] +---- +POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore +{ + "indices": "*,-.*" +} +---- +// TEST[setup:setup-snapshots] +// TEST[s/^/DELETE my-index\nDELETE _data_stream\/logs-my_app-default\n/] +// TEST[s/_restore/_restore?wait_for_completion=true/] + +[discrete] +[[restore-feature-state]] +=== Restore a feature state + +You can restore a feature state to recover system indices, system data streams, +and other configuration data for a feature from a snapshot. Restoring a feature +state is the preferred way to restore system indices and system data streams. + +If you restore a snapshot's cluster state, the operation restores all feature +states in the snapshot by default. Similarly, if you don't restore a snapshot's +cluster state, the operation doesn't restore any feature states by default. You +can also choose to restore only specific feature states from a snapshot, +regardless of the cluster state. + +To view a snapshot's feature states, use the get snapshot API. + +[source,console] +---- +GET _snapshot/my_repository/my_snapshot_2099.05.06 +---- +// TEST[setup:setup-snapshots] + +The response's `feature_states` property contains a list of features in the +snapshot as well as each feature's indices. + +To restore a specific feature state from the snapshot, specify the +`feature_name` from the response in the restore snapshot API's +<> parameter. When you +restore a feature state, {es} closes and overwrites the feature's existing +indices. + +WARNING: Restoring the `security` feature state overwrites system indices +used for authentication. If you use {ess}, ensure you have access to the {ess} +Console before restoring the `security` feature state. If you run {es} on your +own hardware, <> to ensure you'll still be able to access your cluster. + +To avoid accidentally restoring system indices, system data streams, and other +dot indices, append the `-.*` wildcard pattern to the `indices` value. + +[source,console] +---- +POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore +{ + "indices": "*,-.*", + "feature_states": [ "geoip" ] +} +---- +// TEST[setup:setup-snapshots] +// TEST[s/^/DELETE my-index\nDELETE _data_stream\/logs-my_app-default\n/] +// TEST[s/_restore/_restore?wait_for_completion=true/] +// TEST[s/",/"/] +// TEST[s/"feature_states": \[ "geoip" \]//] + [discrete] [[restore-entire-cluster]] === Restore an entire cluster In some cases, you need to restore an entire cluster from a snapshot, including -the cluster state and all system indices. These cases should be rare, such as in +the cluster state and all feature states. These cases should be rare, such as in the event of a catastrophic failure. -Restoring an entire cluster involves deleting important system indices, including -those used for authentication. Consider whether you can restore specific indices -or data streams instead. +Restoring an entire cluster involves deleting important system indices, +including those used for authentication. Consider whether you can restore +specific indices or data streams instead. If you're restoring to a different cluster, see <> before you start. @@ -235,6 +313,18 @@ the cluster. . Temporarily stop indexing and turn off the following features: + -- +* GeoIP database downloader ++ +[source,console] +---- +PUT _cluster/settings +{ + "persistent": { + "ingest.geoip.downloader.enabled": false + } +} +---- + * ILM + [source,console] @@ -265,19 +355,18 @@ POST _ml/set_upgrade_mode?enabled=false // TEST[continued] //// -//// +* Monitoring ++ [source,console] ---- PUT _cluster/settings { "persistent": { - "xpack.monitoring.collection.enabled": true + "xpack.monitoring.collection.enabled": false } } ---- // TEST[warning:[xpack.monitoring.collection.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.] -// TEST[continued] -//// * Watcher + @@ -295,23 +384,29 @@ POST _watcher/_start //// -- -. If you use {es} security features, log in to a node host, navigate to the {es} + +. {blank} ++ +-- +[[restore-create-file-realm-user]] +If you use {es} security features, log in to a node host, navigate to the {es} installation directory, and add a user with the `superuser` role to the file realm using the <> tool. -+ + For example, the following command creates a user named `restore_user`. -+ + [source,sh] ---- ./bin/elasticsearch-users useradd restore_user -p my_password -r superuser ---- -+ + Use this file realm user to authenticate requests until the restore operation is complete. +-- . Use the <> to set <> to -`false`. This lets you delete indices and data streams using wildcards. +`false`. This lets you delete data streams and indices using wildcards. + [source,console] ---- @@ -324,24 +419,28 @@ PUT _cluster/settings ---- // TEST[setup:setup-snapshots] -. Delete existing indices and data streams on the cluster. +. Delete all existing data streams on the cluster. + [source,console] ---- -# Delete all indices -DELETE * +DELETE _data_stream/*?expand_wildcards=all +---- +// TEST[continued] -# Delete all data streams -DELETE _data_stream/* +. Delete all existing indices on the cluster. ++ +[source,console] +---- +DELETE *?expand_wildcards=all ---- // TEST[continued] -. Restore the entire snapshot, including the cluster state. This also restores -any system indices in the snapshot. +. Restore the entire snapshot, including the cluster state. By default, +restoring the cluster state also restores any feature states in the snapshot. + [source,console] ---- -POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore +POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore { "indices": "*", "include_global_state": true @@ -354,6 +453,19 @@ POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore features you stopped: + -- +* GeoIP database downloader ++ +[source,console] +---- +PUT _cluster/settings +{ + "persistent": { + "ingest.geoip.downloader.enabled": true + } +} +---- +//TEST[s/true/false/] + * ILM + [source,console] @@ -368,6 +480,20 @@ POST _ilm/start POST _ml/set_upgrade_mode?enabled=false ---- +* Monitoring ++ +[source,console] +---- +PUT _cluster/settings +{ + "persistent": { + "xpack.monitoring.collection.enabled": true + } +} +---- +// TEST[warning:[xpack.monitoring.collection.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.] +// TEST[s/true/false/] + * Watcher + [source,console]