[Rule Registry] Default index settings and ILM policy for all indices

[banderror]

Parent ticket: #101016

Summary

We never really talked about which index settings and ILM policy would be best for all RAC indices by default. What we have in the code is more like a draft and an issue was found with the default policy: #111029

kibana/x-pack/plugins/rule_registry/common/assets/lifecycle_policies/default_lifecycle_policy.ts

Lines 8 to 26 in 1205ba4

	export const defaultLifecyclePolicy = {
	policy: {
	phases: {
	hot: {
	actions: {
	rollover: {
	max_age: '90d',
	max_size: '50gb',
	},
	},
	},
	delete: {
	actions: {
	delete: {},
	},
	},
	},
	},
	};

Index settings are minimalistic and defined in the technical component template:

kibana/x-pack/plugins/rule_registry/common/assets/component_templates/technical_component_template.ts

Lines 14 to 16 in 0d55d30

	settings: {
	number_of_shards: 1,
	},

Ideas for the policy

We could use the .siem-signals policy as a base because it was in production for some time:

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d"
          }
        }
      }
    }
  }
}

30d and 50gb are the recommended defaults, and max_size is deprecated in favour of max_primary_shard_size:

So it could be

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb",
            "max_age": "30d"
          }
        }
      }
    }
  }
}

Some default policies also include a _meta object with managed: true and a description.

Any other settings that would make sense? Data tiers, priorities, etc?

Ideas for the settings

I was thinking about something like that:

  settings: {
    number_of_shards: 1,
    auto_expand_replicas: '0-1',
    'mapping.total_fields.limit': 10000,
    'sort.field': '@timestamp',
    'sort.order': 'desc',
  },

• auto_expand_replicas: '0-1' can be useful in simple 1-node setups (otherwise indices won’t be healthy); it’s used in event_log plugin for example
• ‘mapping.total_fields.limit’: 10000 is used in Security Solution for .siem-signals indices
• default sorting could be probably useful to make all queries faster

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[banderror]

cc @weltenwort

[weltenwort]

ℹ️ I'm implementing the removal of the delete phase as a fix for 7.15.0 in #111139. I also align the rollover action thresholds with the default you mentioned above.

[banderror]

Hey everyone, I removed this ticket from the backlog of the Detection Rules area.

We (@elastic/security-detections-response-rules) are not the owners anymore (however feel free to still ping us if you have any tech questions about the ticket). Ownership of this ticket and other tickets related to rule_registry (like #101016) now goes to the Detection Alerts area (Team:Detection Alerts label). Please ping @peluja1012 and @marshallmain if you have any questions.

[marshallmain]

Transferring again to @elastic/response-ops as they now own the rule registry implementation.

[mikecote]

Linking with #141146 for researching.

[ymao1]

Closing as default index settings and ILM policy already exists for RAC indices.