Deprecate REST access to System Indices (#63274) (Original #60945)

This PR adds deprecation warnings when accessing System Indices via the REST layer. At this time, these warnings are only enabled for Snapshot builds by default, to allow projects external to Elasticsearch additional time to adjust their access patterns.

Deprecation warnings will be triggered by all REST requests which access registered System Indices, except for purpose-specific APIs which access System Indices as an implementation detail a few specific APIs which will continue to allow access to system indices by default:

- `GET _cluster/health`
- `GET {index}/_recovery`
- `GET _cluster/allocation/explain`
- `GET _cluster/state`
- `POST _cluster/reroute`
- `GET {index}/_stats`
- `GET {index}/_segments`
- `GET {index}/_shard_stores`
- `GET _cat/[indices,aliases,health,recovery,shards,segments]`

Deprecation warnings for accessing system indices take the form:
```
this request accesses system indices: [.some_system_index], but in a future major version, direct access to system indices will be prevented by default
```

docs/reference/api-conventions.asciidoc

@@ -42,6 +42,14 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=expand-wildcards]
 
 The defaults settings for the above parameters depend on the API being used.
 
+Some indices (hereafter "system indices") are used by various system
+modules and/or plugins to store state or configuration. These indices
+are not intended to be accessed directly, and accessing them directly is
+deprecated. In the next major version, access to these indices will no longer be
+allowed to prevent accidental operations that may cause problems with
+Elasticsearch features which depend on the consistency of data in these
+indices.
+
 Some multi-target APIs that can target indices also support the following query
 string parameter:
 

docs/reference/docs/delete-by-query.asciidoc

@@ -53,13 +53,13 @@ POST /my-index-000001/_delete_by_query
 ==== {api-description-title}
 
 You can specify the query criteria in the request URI or the request body
-using the same syntax as the  <<search-search,Search API>>. 
+using the same syntax as the  <<search-search,Search API>>.
 
 When you submit a delete by query request, {es} gets a snapshot of the data stream or index
 when it begins processing the request and deletes matching documents using
 `internal` versioning. If a document changes between the time that the
 snapshot is taken and the delete operation is processed, it results in a version
-conflict and the delete operation fails. 
+conflict and the delete operation fails.
 
 NOTE: Documents with a version equal to 0 cannot be deleted using delete by
 query because `internal` versioning does not support 0 as a valid
@@ -70,18 +70,18 @@ requests sequentially to find all of the matching documents to delete. A bulk
 delete request is performed for each batch of matching documents. If a
 search or bulk request is rejected, the requests are retried up to 10 times, with
 exponential back off. If the maximum retry limit is reached, processing halts
-and all failed requests are returned in the response. Any delete requests that 
-completed successfully still stick, they are not rolled back. 
+and all failed requests are returned in the response. Any delete requests that
+completed successfully still stick, they are not rolled back.
 
-You can opt to count version conflicts instead of halting and returning by 
-setting `conflicts` to `proceed`. 
+You can opt to count version conflicts instead of halting and returning by
+setting `conflicts` to `proceed`.
 
 ===== Refreshing shards
 
 Specifying the `refresh` parameter refreshes all shards involved in the delete
-by query once the request completes. This is different than the delete API's 
-`refresh` parameter, which causes just the shard that received the delete 
-request to be refreshed. Unlike the delete API, it does not support 
+by query once the request completes. This is different than the delete API's
+`refresh` parameter, which causes just the shard that received the delete
+request to be refreshed. Unlike the delete API, it does not support
 `wait_for`.
 
 [[docs-delete-by-query-task-api]]
@@ -90,7 +90,7 @@ request to be refreshed. Unlike the delete API, it does not support
 If the request contains `wait_for_completion=false`, {es}
 performs some preflight checks, launches the request, and returns a
 <<tasks,`task`>> you can use to cancel or get the status of the task. {es} creates a
-record of this task as a document at `.tasks/task/${taskId}`. When you are 
+record of this task as a document at `.tasks/task/${taskId}`. When you are
 done with a task, you should delete the task document so {es} can reclaim the
 space.
 
@@ -101,20 +101,20 @@ before proceeding with the request. See <<index-wait-for-active-shards>>
 for details. `timeout` controls how long each write request waits for unavailable
 shards to become available. Both work exactly the way they work in the
 <<docs-bulk,Bulk API>>. Delete by query uses scrolled searches, so you can also
-specify the `scroll` parameter to control how long it keeps the search context 
+specify the `scroll` parameter to control how long it keeps the search context
 alive, for example `?scroll=10m`. The default is 5 minutes.
 
 ===== Throttling delete requests
 
 To control the rate at which delete by query issues batches of delete operations,
 you can set `requests_per_second` to any positive decimal number. This pads each
-batch with a wait time to throttle the rate. Set `requests_per_second` to `-1` 
+batch with a wait time to throttle the rate. Set `requests_per_second` to `-1`
 to disable throttling.
 
-Throttling uses a wait time between batches so that the internal scroll requests 
-can be given a timeout that takes the request padding into account. The padding 
-time is the difference between the batch size divided by the 
-`requests_per_second` and the time spent writing.  By default the batch size is 
+Throttling uses a wait time between batches so that the internal scroll requests
+can be given a timeout that takes the request padding into account. The padding
+time is the difference between the batch size divided by the
+`requests_per_second` and the time spent writing.  By default the batch size is
 `1000`, so if `requests_per_second` is set to `500`:
 
 [source,txt]
@@ -123,9 +123,9 @@ target_time = 1000 / 500 per second = 2 seconds
 wait_time = target_time - write_time = 2 seconds - .5 seconds = 1.5 seconds
 --------------------------------------------------
 
-Since the batch is issued as a single `_bulk` request, large batch sizes 
-cause {es} to create many requests and wait before starting the next set. 
-This is "bursty" instead of "smooth". 
+Since the batch is issued as a single `_bulk` request, large batch sizes
+cause {es} to create many requests and wait before starting the next set.
+This is "bursty" instead of "smooth".
 
 [[docs-delete-by-query-slice]]
 ===== Slicing
@@ -134,11 +134,11 @@ Delete by query supports <<slice-scroll, sliced scroll>> to parallelize the
 delete process. This can improve efficiency and provide a
 convenient way to break the request down into smaller parts.
 
-Setting `slices` to `auto` chooses a reasonable number for most data streams and indices. 
-If you're slicing manually or otherwise tuning automatic slicing, keep in mind 
+Setting `slices` to `auto` chooses a reasonable number for most data streams and indices.
+If you're slicing manually or otherwise tuning automatic slicing, keep in mind
 that:
 
-* Query performance is most efficient when the number of `slices` is equal to 
+* Query performance is most efficient when the number of `slices` is equal to
 the number of shards in the index or backing index. If that number is large (for example,
 500), choose a lower number as too many `slices` hurts performance. Setting
 `slices` higher than the number of shards generally does not improve efficiency
@@ -171,25 +171,25 @@ Defaults to `true`.
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=analyzer]
 
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=analyze_wildcard]
-  
+
 `conflicts`::
-  (Optional, string) What to do if delete by query hits version conflicts: 
+  (Optional, string) What to do if delete by query hits version conflicts:
   `abort` or `proceed`. Defaults to `abort`.
 
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=default_operator]
 
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=df]
-  
+
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=expand-wildcards]
 +
 Defaults to `open`.
 
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=from]
 
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=index-ignore-unavailable]
-  
+
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=lenient]
-  
+
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=max_docs]
 
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=preference]
@@ -214,9 +214,9 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=scroll_size]
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=search_type]
 
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=search_timeout]
-  
+
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=slices]
-  
+
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=sort]
 
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=source]
@@ -226,7 +226,7 @@ include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=source_excludes]
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=source_includes]
 
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=stats]
-  
+
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=terminate_after]
 
 include::{es-repo-dir}/rest-api/common-parms.asciidoc[tag=timeout]
@@ -239,9 +239,9 @@ include::{docdir}/rest-api/common-parms.asciidoc[tag=wait_for_active_shards]
 ==== {api-request-body-title}
 
 `query`::
-  (Optional, <<query-dsl,query object>>) Specifies the documents to delete 
+  (Optional, <<query-dsl,query object>>) Specifies the documents to delete
   using the  <<query-dsl,Query DSL>>.
-  
+
 
 [[docs-delete-by-query-api-response-body]]
 ==== Response body
@@ -345,7 +345,7 @@ this is non-empty then the request aborted because of those failures.
 Delete by query is implemented using batches, and any failure causes the entire
 process to abort but all failures in the current batch are collected into the
 array. You can use the `conflicts` option to prevent reindex from aborting on
-version conflicts.  
+version conflicts.
 
 [[docs-delete-by-query-api-example]]
 ==== {api-examples-title}
@@ -377,7 +377,7 @@ POST /my-index-000001,my-index-000002/_delete_by_query
 // TEST[s/^/PUT my-index-000001\nPUT my-index-000002\n/]
 
 Limit the delete by query operation to shards that a particular routing
-value: 
+value:
 
 [source,console]
 --------------------------------------------------
@@ -571,7 +571,7 @@ though these are all taken at approximately the same time.
 
 The value of `requests_per_second` can be changed on a running delete by query
 using the `_rethrottle` API. Rethrottling that speeds up the
-query takes effect immediately but rethrotting that slows down the query 
+query takes effect immediately but rethrotting that slows down the query
 takes effect after completing the current batch to prevent scroll
 timeouts.
 
@@ -670,6 +670,6 @@ POST _tasks/r1A2WoRbTwKZ516z6NEs5A:36619/_cancel
 
 The task ID can be found using the <<tasks,tasks API>>.
 
-Cancellation should happen quickly but might take a few seconds. The task status 
-API above will continue to list the delete by query task until this task checks that it 
+Cancellation should happen quickly but might take a few seconds. The task status
+API above will continue to list the delete by query task until this task checks that it
 has been cancelled and terminates itself.