[Metrics UI] Set up APM on our Metrics UI performance cluster

[jasonrhodes]

We have a private Metrics UI performance cluster where we need to run APM so we can evaluate performance.

AC:

• APM node agent is running in this branch
• APM server is running in a cloud instance (start with our performance cluster, but we can have a separate APM cluster if that ends up making sense)

Other tickets

• APM RUM agent is running in this branch
• Investigate and add custom spans
• Analyze waffle map query using APM

Note: it would also be good to understand what, if any, custom spans we should consider introducing and make sure we understand how to do that in our own APM set up. If we add custom spans to our code, can we leave them there even if users don't use APM?

Note: We should check with the Synthetics team about writing synthetics tests that could test our UI and time things. We can spin that work into a new ticket.

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[jasonrhodes]

@elastic/apm can someone help us understand our questions about instrumenting our code with APM spans? @elastic/apm-agent-node-js may also want to get involved here :)

[dgieselaar]

@jasonrhodes perhaps useful for you as well: #90403

[vigneshshanmugam]

Let us know how we can help here, I can provide help with both RUM and Synthetics.

[jasonrhodes]

@dgieselaar @vigneshshanmugam thanks! When one of us picks this up in the next week or two, we'll reach out!

[estermv]

Kibana already comes with APM. To enable it, we need to add a config/apm.dev.js with the following content:

module.exports = {
  active: true,
  // Use if APM Server requires a token
  secretToken: <secret_token>,
  // Set custom APM Server URL (default: http://localhost:8200)
  serverUrl: '<server_url>',
};

I was able to setup a local kibana connected to the performance cluster and send some data to APM:

[smith]

You can also specify elastic.apm.active: true in your kibana.yml, but due to a bug (#79490), it won't pick up the setting unless you pass --config config/kibana.dev.yml to the yarn start command. This is nice because you may want to enable or disable it depending on which cluster you're connecting to.

[dgieselaar]

If this is a cloud cluster (and it looks like it is), I'm not sure you can enable APM there. @sqren might know more. If you connect a local Kibana instance to it, that will collect APM data, but not if you go to the Kibana instance on cloud.

[estermv]

@dgieselaar you are right, it's a cloud cluster, but we enabled APM in a local kibana. We collect data from a local kibana instance, but that's what we want (for now) since we want to do some changes in our code and see how this affects performance.

[sorenlouv]

@dgieselaar That's also my understanding, that it's still not possible to instrument Kibana on cloud (coming soon 🤞 )
From @estermv's comment it sounds like they are instrumenting a local kibana and sending it to an APM Server instance on cloud which should be fine.