Improve documentation around system feature snapshot and restore

[williamrandolph]

Our documentation around snapshots, system indices, and system features is unclear. The technically correct information exists in the API documentation, but mostly under the information about the feature_states parameter. I don't believe there is a high-level summary about what a system feature is, what it means to take a snapshot of one, and how to include or exclude it from a snapshot restoration.

For example, here is what we currently have in the "Create a snapshot" page:

Besides creating a copy of each data stream and index, the snapshot process can also store global cluster metadata, which includes persistent cluster settings, templates, and data stored in system indices, such as Watches and task records, regardless of whether those system indices are named in the indices section of the request. You can also use the create snapshot API’s feature_states parameter to include only a subset of system indices in the snapshot. Snapshots do not store transient settings or registered snapshot repositories.

User stories to document

First, a few definitions from the developer's perspective:

• In the codebase, a "system feature" is a component that defines one or more system indices, associated indices, or system data streams, alongside code for various management operations for those indices and data streams.
• "Feature state" means the system indices, associated indices, and system data streams of a feature at a given time, for example, all of the backing data that Kibana stores in Elasticsearch.
• A "system index" is an index that is meant to be hidden from users. In 7.x, this means that you get deprecation warnings when you access them. In 8.0, you have to have special permissions to access them directly. Eventually, we don't want there to be any direct access. (Due to a quirk of development, there is no direct access to the GeoIP system index in 7.x)
• A "system data stream" is a data stream that is hidden from users but used by a system component. Currently, only Fleet has a system data stream.
• An "associated index" is an index that the feature uses, but that doesn't need the protections provided to system indices. Often this is because it contains information that we want end-users to be able to see and search. Such indices are also included in snapshots of "feature state."

We hope that "feature" and "feature state" are the main concepts with which users needs to concern themselves.

Snapshot

1. A user wants to make a snapshot that includes the system configuration and data for one or more system components, for example, Elasticsearch security or Kibana.
2. The user calls the GET features api to see which system features are present in the system. (Under the hood, a "system feature" is usually defined in an x-pack plugin, but that is an implementation detail which already has one exception and is subject to further change.)
3. The user decides to include a feature state in their snapshot request using the feature_states parameter.
4. Once the snapshot is taken, the user can see the included system indices using the GET snapshot API.

When the snapshot has include_global_state set to true, all feature states (meaning, all system indices, associated indices, and system data streams) are included in the snapshot. Snapshots with global state have already proved tricky to restore into new clusters, often because a system index in the global state clashes with something already named in the cluster.

Restore

1. A user wants to reset a feature to a previous state, or restore a feature's settings into a new cluster.
2. The user calls the GET snapshot API to see which features are included in the snapshot of interest.
3. The user issues a restore request with the names of those features in the feature_states request parameter.

Where users are running into trouble at the moment is when system indices they don't really care about clash with an index that is already in the cluster. Many users want to know how to exclude a particular system index from the restore operation. Our desired solution is that the user could exclude the feature that owns the system index. Unfortunately, we don't have an excluded_feature_states request parameter; the only way to exclude a feature state right now is to put all the feature states except that one in the feature_states parameter.

Tricky cases

• In 7.x, snapshotting and restoring all indices includes system indices for backwards compatibility. Users might not expect this. The documented behavior is as follows:
  ** Request snapshots or restores all indices, with no feature_states or include_global_state parameter: all feature states included
  ** Request snapshots or restores one specific index, with include_global_state set to true: all feature states included
  ** Request snapshots or restores all indices, with include_global_state set to false: no feature states included
  ** Request snapshots or restores all indices, with feature_states: []: no feature states included
  ** Request snapshots or restores all indices, with feature_states: ["none"]: no feature states included
• I honestly can't remember what happens if a snapshot has a feature state that the cluster we are restoring to lacks. I don't think this can happen in practice, but it could in the feature if users have custom plugins that define system indices.
• For normal indices, a user must explicitly close the index before it can be restored from a snapshot. System indices are different; they are automatically closed and overwritten during the restore operation. In hindsight, I can see that this might have unintended consequences.

cc: @gwbrown, @lockewritesdocs, @debadair

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[elasticmachine]

Pinging @elastic/es-docs (Team:Docs)

[jrodewig]

Hi @williamrandolph,

I'm currently working on an overhaul of the snapshot/restore documentation with #79081. That work doesn't touch on feature states in depth but does mention system indices.

I also recently updated the restore a snapshot tutorial with #76929. Again, that work doesn't go into feature states in depth but does cover some methods for excluding system indices.

I think these updates will work best as a follow-on to those efforts, but let me know what you think. I see that you assigned @lockewritesdocs. If you've already discussed these changes with him, let me know.

[jrodewig]

Where users are running into trouble at the moment is when system indices they don't really care about clash with an index that is already in the cluster. Many users want to know how to exclude a particular system index from the restore operation.

I've opened #79683 to specifically address this user pain point. I tentatively plan to incorporate the other changes into #79081.

Our desired solution is that the user could exclude the feature that owns the system index. Unfortunately, we don't have an excluded_feature_states request parameter; the only way to exclude a feature state right now is to put all the feature states except that one in the feature_states parameter.

I may be missing something, but I don't think this works. Even if you exclude the feature state, the request will still return an error if the indices parameter is permissive enough to try to restore an existing system index. Unfortunately, it's that permissive by default.

The only solutions I found were:

1. Explicitly excluding dot indices from indices using the -.* wildcard pattern.
2. Restoring the system index via the feature_states parameter. However, this only works if the snapshot contains the feature state for the system index. It may also not actually be desired by the user.

Since option 1 works in all cases, I documented that in #79683. However, I could be wrong. Let me know!

[williamrandolph]

Thanks, James!

When it comes to the terminology around "system indices" and "feature states," our thinking is that eventually we want "system indices" to be an implementation detail, and "system features" to be the user-facing catch-all term for system components that use Elasticsearch to store configuration data, metadata, or any other "state."

I think you are right about the unfortunate permissiveness of the indices parameter. @gwbrown is working on changing the behavior for 8.0 to something that makes more sense to everyone, and he might be able to add some useful context here.

If you and @lockewritesdocs think that this issue is already covered by what you are doing in your overhaul, it is okay with me if you close this issue.

[jrodewig]

Thanks @williamrandolph. I'll leave this open for now.

Here's how I've spread out the related docs work:

• [DOCS] Add docs for restoring to new cluster #79683 covers how to restore a feature state and how to exclude system indices from a restore.
• Commit c16f74e98fc34d47ccc6a8ea5fcbce1e7adbfec of [DOCS] Overhaul snapshot and restore docs #79081 covers the feature state concept and how to include feature states in a snapshot. That PR is still a draft, but feel free to leave reviews on the feature state bits if wanted.

Both of those reflect how things work as of this writing. We'll need to do another update if we merge #79670 or make other changes that affect the snapshot/restore defaults.

[jrodewig]

With #79683 and #79081 merged, I'm going to close out this issue.

We'll need to update the docs when #79670 or a similar change merges, but I think we can handle that with our usual docs update process for new development. Feel free to re-open, etc. if that makes it easier for tracking.