Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce "Feature States" for managing snapshots of system indices #63513

Merged
merged 157 commits into from
Feb 11, 2021

Conversation

gwbrown
Copy link
Contributor

@gwbrown gwbrown commented Oct 8, 2020

This PR expands the meaning of include_global_state for snapshots to include system indices. If include_global_state is true on creation, system indices will be included in the snapshot regardless of the contents of the indices field. If include_global_state is true on restoration, system indices will be restored (if included in the snapshot), regardless of the contents of the indices field. Index renaming is not applied to system indices, as system indices rely on their names matching certain patterns. If restored system indices are already present, they are automatically deleted prior to restoration from the snapshot to avoid conflicts.

This behavior can be overridden to an extent by including a new field in the snapshot creation or restoration call, feature_states, which contains an array of strings indicating the "feature" for which system indices should be snapshotted or restored. For example, this call will only restore the watcher and security system indices (in addition to index_1):

POST /_snapshot/my_repository/snapshot_2/_restore
{
  "indices": "index_1",
  "include_global_state": true,
  "feature_states": ["watcher", "security"]
}

If feature_states is present, the system indices associated with those features will be snapshotted or restored regardless of the value of include_global_state. All system indices can be omitted by providing a special value of none ("feature_states": ["none"]), or included by omitting the field or explicitly providing an empty array ("feature_states": []), similar to the indices field.

The list of currently available features can be retrieved via a new "Get Snapshottable Features" API:

GET /_snapshottable_features

which returns a response of the form:

{
    "features": [
        {
            "name": "tasks",
            "description": "Manages task results"
        },
        {
            "name": "kibana",
            "description": "Manages Kibana configuration and reports"
        }
    ]
}

Features currently map one-to-one with SystemIndexPlugins, but this should be considered an implementation detail. The Get Snapshottable Features API and snapshot creation rely upon all relevant plugins being installed on the master node.

Further, the list of feature states included in a given snapshot is exposed by the Get Snapshot API, which now includes a new field, feature_states, which contains a list of the feature states and their associated system indices which are included in the snapshot. All system indices in feature states are also included in the indices array for backwards compatibility, although explicitly requesting system indices included in a feature state is deprecated. For example, an excerpt from the Get Snapshot API showing feature_states:

"feature_states": [
    {
        "feature_name": "tasks",
        "indices": [
            ".tasks"
        ]
    }
],
"indices": [
    ".tasks",
    "test1",
    "test2"
]

Relates #61657

@gwbrown gwbrown added >enhancement WIP :Core/Infra/Core Core issues without another label v8.0.0 labels Oct 8, 2020
@gwbrown
Copy link
Contributor Author

gwbrown commented Feb 11, 2021

I talked with @original-brownbear elsewhere and confirmed he's okay with merging this once the test is working properly, which it now is. So I'll be merging this once we get a green CI run.

@gwbrown gwbrown merged commit 3f6472d into elastic:master Feb 11, 2021
gwbrown added a commit to gwbrown/elasticsearch that referenced this pull request Feb 11, 2021
…lastic#63513)

This PR expands the meaning of `include_global_state` for snapshots to include system indices. If `include_global_state` is `true` on creation, system indices will be included in the snapshot regardless of the contents of the `indices` field. If `include_global_state` is `true` on restoration, system indices will be restored (if included in the snapshot), regardless of the contents of the `indices` field. Index renaming is not applied to system indices, as system indices rely on their names matching certain patterns. If restored system indices are already present, they are automatically deleted prior to restoration from the snapshot to avoid conflicts.

This behavior can be overridden to an extent by including a new field in the snapshot creation or restoration call, `feature_states`, which contains an array of strings indicating the "feature" for which system indices should be snapshotted or restored. For example, this call will only restore the `watcher` and `security` system indices (in addition to `index_1`):

```
POST /_snapshot/my_repository/snapshot_2/_restore
{
  "indices": "index_1",
  "include_global_state": true,
  "feature_states": ["watcher", "security"]
}
```

If `feature_states` is present, the system indices associated with those features will be snapshotted or restored regardless of the value of `include_global_state`. All system indices can be omitted by providing a special value of `none` (`"feature_states": ["none"]`), or included by omitting the field or explicitly providing an empty array (`"feature_states": []`), similar to the `indices` field.

The list of currently available features can be retrieved via a new "Get Snapshottable Features" API:
```
GET /_snapshottable_features
```

which returns a response of the form:
```
{
    "features": [
        {
            "name": "tasks",
            "description": "Manages task results"
        },
        {
            "name": "kibana",
            "description": "Manages Kibana configuration and reports"
        }
    ]
}
```

Features currently map one-to-one with `SystemIndexPlugin`s, but this should be considered an implementation detail. The Get Snapshottable Features API and snapshot creation rely upon all relevant plugins being installed on the master node.

Further, the list of feature states included in a given snapshot is exposed by the Get Snapshot API, which now includes a new field, `feature_states`, which contains a list of the feature states and their associated system indices which are included in the snapshot. All system indices in feature states are also included in the `indices` array for backwards compatibility, although explicitly requesting system indices included in a feature state is deprecated. For example, an excerpt from the Get Snapshot API showing `feature_states`:
```
"feature_states": [
    {
        "feature_name": "tasks",
        "indices": [
            ".tasks"
        ]
    }
],
"indices": [
    ".tasks",
    "test1",
    "test2"
]
```

Co-authored-by: William Brafford <william.brafford@elastic.co>
gwbrown added a commit that referenced this pull request Feb 11, 2021
…ices (#63513)

This PR expands the meaning of `include_global_state` for snapshots to include system indices. If `include_global_state` is `true` on creation, system indices will be included in the snapshot regardless of the contents of the `indices` field. If `include_global_state` is `true` on restoration, system indices will be restored (if included in the snapshot), regardless of the contents of the `indices` field. Index renaming is not applied to system indices, as system indices rely on their names matching certain patterns. If restored system indices are already present, they are automatically deleted prior to restoration from the snapshot to avoid conflicts.

This behavior can be overridden to an extent by including a new field in the snapshot creation or restoration call, `feature_states`, which contains an array of strings indicating the "feature" for which system indices should be snapshotted or restored. For example, this call will only restore the `watcher` and `security` system indices (in addition to `index_1`):

```
POST /_snapshot/my_repository/snapshot_2/_restore
{
  "indices": "index_1",
  "include_global_state": true,
  "feature_states": ["watcher", "security"]
}
```

If `feature_states` is present, the system indices associated with those features will be snapshotted or restored regardless of the value of `include_global_state`. All system indices can be omitted by providing a special value of `none` (`"feature_states": ["none"]`), or included by omitting the field or explicitly providing an empty array (`"feature_states": []`), similar to the `indices` field.

The list of currently available features can be retrieved via a new "Get Snapshottable Features" API:
```
GET /_snapshottable_features
```

which returns a response of the form:
```
{
    "features": [
        {
            "name": "tasks",
            "description": "Manages task results"
        },
        {
            "name": "kibana",
            "description": "Manages Kibana configuration and reports"
        }
    ]
}
```

Features currently map one-to-one with `SystemIndexPlugin`s, but this should be considered an implementation detail. The Get Snapshottable Features API and snapshot creation rely upon all relevant plugins being installed on the master node.

Further, the list of feature states included in a given snapshot is exposed by the Get Snapshot API, which now includes a new field, `feature_states`, which contains a list of the feature states and their associated system indices which are included in the snapshot. All system indices in feature states are also included in the `indices` array for backwards compatibility, although explicitly requesting system indices included in a feature state is deprecated. For example, an excerpt from the Get Snapshot API showing `feature_states`:
```
"feature_states": [
    {
        "feature_name": "tasks",
        "indices": [
            ".tasks"
        ]
    }
],
"indices": [
    ".tasks",
    "test1",
    "test2"
]
```

Co-authored-by: William Brafford <william.brafford@elastic.co>
gwbrown added a commit that referenced this pull request Feb 11, 2021
This PR adjusts the feature states compatibility version following the backport of #63513
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Core/Infra/Core Core issues without another label >enhancement Team:Core/Infra Meta label for core/infra team v7.12.0 v8.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants