update APM nodejs agent to a version usable without APM-server

[mshustov]

Kibana is going to use the APM nodejs agent for log correlation. For this, we need a mode when the APM nodejs agent can be used without sending data to the APM server.

Blocked by elastic/apm-agent-nodejs#2101 Update to a version containing the enhancement.

After the agent version is updated, investigate whether we can (and should) use APM (see setCustomContext) for context propagation instead of relying on a custom AsyncLocalStorage. With this API, customContext will be available in APM UI as well.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[trentm]

elastic/apm-agent-nodejs#2101 has been included in elastic-apm-node@3.17.0 now, so Kibana could be updated to use that and the new disableSend: true config option.

[mshustov]

After the agent version is updated, investigate whether we can (and should) use APM (see setCustomContext) for context propagation instead of relying on a custom AsyncLocalStorage. With this API, customContext will be available in APM UI as well.

I played around a bit with using the APM agent for storing execution context and it doesn't quite match the APM data model (for example, a transaction doesn't have access to a parent transaction instance). I talked to @trentm and we agreed to wait until the Baggage header API is supported by the agent. Once Baggage support is implemented, we can use it instead of relying upon AsyncLocalStorage mechanism, see examples here #97934 (comment)

In the current task, I will update the APM agent version anyway as it's a prerequisite for adding tracing information to the Kibana logs #102699

In a follow-up PR, I can add execution_context information to an Elasticsearch request transaction to allow users to find this information in APM UI. @trentm what is the correct way to attach meta information to the Elasticseach request transaction?
I tried both:

  client.on('request', (e, event) => {
    agent.setCustomContext({ kbnContextReq: 'kbnContextReq' });
    agent.setLabel('kbnContextReq', 'kbnContextReq');
  });
  client.on('response', (error, event) => {
    agent.setCustomContext({ kbnContextRes: 'kbnContextRes' });
    agent.setLabel('kbnContextRes', 'kbnContextRes');
});

But I don't see them in APM UI. Should I configure anything else?

[mshustov]

If we are okay to keep the current execution_context implementation based on AyncLocalStorage, I can close the issue since I updated the APM agent to a version with necessary functionality in #109877

[trentm]

for example, a transaction doesn't have access to a parent transaction instance

First, I'm worried there is some miscommunication on terminology. In APM terminology for a single incoming request to some service (e.g. the Kibana API), there is a single "transaction". Under that transaction there is a hierarchy of zero or more "spans" for interesting events -- either manually added via apm.startSpan() et al, or automatically added for things like outgoing Elasticsearch client requests. For a distributed trace (e.g. the Kibana front end making a call to the Kibana backend API) there is a separate transaction for each separate service/process (e.g. one for the front end click or task that resulted in the call to the Kibana backend, and one for the backend incoming API call).

The screenshot above is a fly out for a span.

Transaction: id=7c4cf941a22fe98c, name="POST /internal/bsearch"
    ...
        Span: id=df4122dce7a04ea9, type=db, subtype=elasticsearch, name="Elasticsearch: GET /.kibana_security_session_1/..."

At runtime, when that Span: id=df4122dce7a04ea9 is the "currentSpan", it has access to the top-level transaction, but does not have a pointer to a possible parent span (represented by the ...).

@trentm what is the correct way to attach meta information to the Elasticseach request transaction?

I assume you mean the Elasticsearch request span.

The agent.setCustomContext(...) calls shown above are setting data on the current transaction, so I believe you should see kbnContextReq and kbnContextRes data on the fly out for the transaction. However, that is problematic in the typical case where a transaction (i.e. a single call to the Kibana backend API) has multiple Elasticsearch client requests: they will overwrite each other.

Perhaps what you want is to set data on the span via: apm.setLabel('kbnExecutionContext', value) or apm.addLabels(...).

client.on('request', function (err, event) {
//   apm.setCustomContext({kbnExecutionContext: getExecutionContext()})
  apm.currentSpan.setLabel('kbnExecutionContext', JSON.stringify(getExecutionContext()))
}

Note that labels are indexed, as opposed to custom context not being indexed. That is probably fine, though.

Also labels can only be string, number, or boolean, so you'd have to stringify the KibanaExecutionContext somehow -- perhaps JSON.stringify() or the same encoding being used for x-opaque-id.

---

Does that answer your question? I think I am sometimes getting mixed up between (a) the effort to track execution context without APM UI and (b) adding info to APM UI about this tracked execution context.

[mshustov]

I think I am sometimes getting mixed up between (a) the effort to track execution context without APM UI and (b) adding info to APM UI about this tracked execution context.

My bad, you are absolutely right. I shouldn't have hijacked the initial topic.

adding info to APM UI about this tracked execution context.

This is a potential follow-up task and shouldn't be considered as part of the current issue.