[Metrics UI][R&D] Profile event loop delays for Metric Threshold alert using high cardinality data

[simianhacker]

We need to profile the Metric Threshold Alert with High Cardinality data and figure out where some of the bottlenecks are happening. There seems to be two problems with the high cardinality data:

1. There is a large amount of time spent making round trips between ES and Kibana to collect all the data via composite aggregations.
2. Processing all this data in-memory for the alert executor

We would like to test:

1. The current impact on the event loop for 1000, 10K, 100K (cardinality)
2. Reducing the impact by changing the approach from "gather all data then process" to "gather and process in chunks"
3. Play around with adding process.nextTick() to the processing so we don't block the event loop when processing large amounts of data in Node.

Related Issues:

• [Metrics UI] Increase composite.size on our alerting rule queries #119501
• [RAC] Investigate Performance Testing for o11y's use of rule registry #119673

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Infra Monitoring UI)