Bundle Metricbeat with Elasticsearch

[jasontedor]

For the 8.0.0 release of Elasticsearch, we are aiming to remove monitoring internal collectors/exporters (#47214, #47215). While Metricbeat can now monitor Elasticsearch, the removal of internal collection leaves our users in a position where they have to take extra steps to do such monitoring via Metricbeat. We want to keep the experience of monitoring Elasticsearch simple. To that end, we are aiming to bundle Metricbeat with Elasticsearch and have its lifecycle managed by Elasticsearch. This saves the user from having to install Metricbeat, and also manage its lifecycle.

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Packaging)

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Monitoring)

[cachedout]

Hi @jasontedor. Thanks for filing this.

Given that Stack Monitoring can also consume ES logs, do you think it would be worth discussing whether or not to also bundle Filebeat in addition to Metricbeat? It is not required for the monitoring application to function, but provides additional functionality on top of the default metrics. Thoughts?

[beiske]

we are aiming to bundle Metricbeat with Elasticsearch and have its lifecycle managed by Elasticsearch

@jasontedor Does this mean enabling and disabling the bundled Metricbeat be a dynamic setting or will it potentially require a reboot of Elasticsearch?

[jasontedor]

It will be a restart of Elasticsearch. Elasticsearch can't fork a native process after startup is complete, by the security manager, and by our seccomp policy and we are never going to relax that.

[beiske]

Thanks, is the corollary then that Elasticsearch will exit if Metricbeat exits so that both processes may be rebooted?

[jasontedor]

@beiske That seems too harsh to me. I think that should be orchestrated externally.

[beiske]

@jasontedor I agree, but just asking to understand the division of responsibilities. Will the user then have to detect such a failure and handle it or will Elasticsearch bundle an init process that acts as a watchdog for both? Will that perhaps depend on the package? Linux distroes have an init process, but docker images don't unless they run multiple processes. I guess we would want a similar experience for the zip package also.

[cachedout]

Hi @jasontedor Did you see my question about Filebeat above? :) I would also like to ask if you've considered possibly doing this bundling prior to 8.0 to ease the story for people who are ready to migrate? For example, in this issue @jakelandis mentions that we might want to do this prior to introducing a deprecation warning but I wanted to see what you think. Thanks!

[jasontedor]

@cachedout I think that Filebeat needs to be considered separately and presents some interesting challenges. I think that we can integrate Metricbeat in before 8.0.

[jasontedor]

@beiske I think we can expose in Elasticsearch if the Metricbeat process has died. I think the experience would be the same for all the packages.

[pugnascotia]

I'm not clear whether it's necessary for Metricbeat, but FYI #50277 will introduce the tini init process to the Elasticsearch image to ensure correct clean-up of ML processes.

[jasontedor]

I'm closing this since there's doubt at this time that we're going to move forward with this. We can re-open if plans resurface.