Add persistent event log for actions and alerting

[pmuellr]

Actions and alerting need a persistent event log to record activities that have occurred with these objects, for general exploratory in general, and for the alerting / action UI specifically.

Current tack is to create a new ES index for this.

This is an umbrella task, with the following sub-tasks:

in progress PR: #45081

• RFC (see version in PR)
• basic ES structures; ilm, index template, alias, initial index, adding events to index
  • but OFF by default config
• write events to log for action execution
• basic query support via http endpoint removed because not needed ATM and security concerns

before shipping:

• new platform plugin access to kibana.index config; issue [alerting event log] remove hard-coded ".kibana" prefix on event log ES resources #55629
• enable ILM access for .kibana* resources; issue need kibana internal user to have read/write access to .kibana* ILM resources elasticsearch#46894
• add Kibana server uuid to all logged events; issue [alerting event log] add Kibana server uuid to all logged events #55631
• set default config for using ES from false to true (will be done when index logging works)
• additional security bits; feature controls, read access (not needed as queries will be gated via requiring saved object ids)
• add query support - [alerting event log] add query support #55633
• buffer events instead of writing directly (efficiency and to handle short-ish ES partition); issue: [alerting event log] buffer events being written instead of writing when logged #55634
• write events to log for other action and alert activities; issue: [alerting event log] add event log for alert execution and alerts scheduling actions #55636
• deal with migrating event log mapping across Kibana version upgrades that adds more fields; issue: [alerting event log] deal with migrating event log mappings over releases #55639
• change SO references in event from nested to single; issue [alerting event log] change SO references in event log entries to single from nested #55640
• tests for all runtime code; issue: [alerting event log] add more tests for existing but unused code #55642
• remove es/context.ts::callEs() and make ES calls directly; see comment; issue: [alerting event log] change ES access functions to use callCluster types #55644

dreaming:

• tests for schema generation
• support logging to different cluster (not sure how)
• implement file store when es calls error, to be replayed into es when it's working again
• support for ack'ing (via new types in usual Kibana SO's?)
• support for annotations (via new types in usual Kibana SO's?)

Quick synopsis; the event log will be a new ES index, like other indices created by Kibana plugins. ILM, rollover, templates, alias, needs all the goodies. Any Kibana plugin can depend on this plugin to get an access function to write log entries to the index. The documents indexed are a subset of ECS properties extended with some Kibana-specific properties.

[elasticmachine]

Pinging @elastic/kibana-stack-services

[epixa]

I definitely think using a dedicated index makes sense for alerting event logs. There's a lot of value in treating them like userland data rather than system information.

Have you considered using the Elastic Common Schema: https://www.elastic.co/guide/en/ecs/current/ecs-event.html ? If we could make that work, you'd get seamless integrations between alerting events and other functionality in Kibana (like SIEM).