Log a warning for event loop delay > threshold

[mshustov]

Nodejs is async by nature: it's been created to handle I/O operations quickly, not perform expensive computations.
Read more https://nodejs.org/en/docs/guides/dont-block-the-event-loop/
It means Kibana starts choking when it has to deal with an enormous amount of computations. That can lead to barely debuggable problems: users can observe occasional failures in different unrelated modules that cannot be explained sometimes.

To simplify the diagnosis of problems related to delays in the event loop Core could enhance the existing metrics service logging to notify users when the system is under heavy load. This way, the users can diagnose a problem and keep the whole system running, potentially by disabling some plugins or changing the parameters (e.g. reducing the task manager poll interval).

Scope:

• There is no "standard" value for nodejs event loop delay: we might need to rely on gathered Kibana APM metrics to find an optimal value for helpfulness and noize level ratio.
• Add logs that are rate-limited to avoid spamming
• Add telemetry when threshold violated (TBD on which numbers to report max, min, median, etc.)

@watson can you provide any recommendations, maybe?

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[watson]

Now that we're on a newer Node.js version, we have access to APIs from Node.js core to monitor the event loop:

• perf_hooks.monitorEventLoopDelay()
• perf_hooks.performance.eventLoopUtilization()

If I recall correctly these are the APIs that APM also uses the collect its metrics, so we could just use those as you also mention.