[ResponseOps][Alerting] Set .alerts-* auto-expand replicas to 0-1

[doakalexi]

Resolves #139864

Summary

Updated .alerts-* indices to have auto_expand_replicas: 0-1

Before the change, after adding an Anomaly Rule I saw the following:

GET /_cluster/health
{
  "cluster_name": "elasticsearch",
  "status": "yellow",
  "timed_out": false,
  "number_of_nodes": 1,
  "number_of_data_nodes": 1,
  "active_primary_shards": 11,
  "active_shards": 11,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 1,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 91.66666666666666
}

GET .alerts-observability.apm.alerts-default/_settings
{
  ".internal.alerts-observability.apm.alerts-default-000001": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": ".alerts-ilm-policy",
          "rollover_alias": ".alerts-observability.apm.alerts-default"
        },
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_content"
            }
          }
        },
        "mapping": {
          "total_fields": {
            "limit": "1700"
          }
        },
        "hidden": "true",
        "number_of_shards": "1",
        "provided_name": ".internal.alerts-observability.apm.alerts-default-000001",
        "creation_date": "1662662584486",
        "number_of_replicas": "1",
        "uuid": "GoSrZY47QMWd4qviKMZvZg",
        "version": {
          "created": "8050099"
        }
      }
    }
  }
}

After the change, after adding an Anomaly Rule I saw the following:

GET /_cluster/health
{
  "cluster_name": "elasticsearch",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 1,
  "number_of_data_nodes": 1,
  "active_primary_shards": 11,
  "active_shards": 11,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100
}

GET .alerts-observability.apm.alerts-default/_settings
{
  ".internal.alerts-observability.apm.alerts-default-000001": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": ".alerts-ilm-policy",
          "rollover_alias": ".alerts-observability.apm.alerts-default"
        },
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_content"
            }
          }
        },
        "mapping": {
          "total_fields": {
            "limit": "1700"
          }
        },
        "hidden": "true",
        "number_of_shards": "1",
        "auto_expand_replicas": "0-1",
        "provided_name": ".internal.alerts-observability.apm.alerts-default-000001",
        "creation_date": "1662663348954",
        "number_of_replicas": "0",
        "uuid": "slSS70gJQlSJolrZhOeGwg",
        "version": {
          "created": "8050099"
        }
      }
    }
  }
}

To Verify

• Checkout main and create an Anomaly rule or any rule that uses the Rule Registry
• Run the following commands to verify that the cluster is unhealthy

GET .alerts-observability.apm.alerts-default/_settings
GET /_cluster/health

• Checkout this pr branch and create a rule that uses the Rule Registry
• Run the above commands to verify that the cluster is healthy and that you see "auto_expand_replicas": "0-1"

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[ymao1]

LGTM! Verified that new .alerts indices are green after initial creation. Verified no effect on existing indices.

[maryam-saeidi]

LGTM! Tested it locally and the number of replicas is as expected.

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: bf56c2e

Metrics [docs]

✅ unchanged

History

• 💚 Build #70818 succeeded 0425eed
• 💚 Build #70618 succeeded 2092b68

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kibanamachine]

💔 Some backports could not be created

Status	Branch	Result
❌	7.17	Backport failed because of merge conflicts
✅	8.4

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 140335

Questions ?

Please refer to the Backport tool documentation

[doakalexi]

💚 All backports created successfully

Status	Branch	Result
✅	7.17

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation