[Agents]: "rpc error: code = Canceled desc" error logs on setting logging level to debug.

[amolnater-qasource]

Kibana version: 7.14.0 BC-4 Kibana cloud environment

Host OS and Browser version: All, All

Build details:

Build: 42656
Commit: 82a4f6a7fa23946667599787cdb1b2d82c1d0dfb
Artifact Link: https://staging.elastic.co/7.14.0-b3779639/summary-7.14.0.html

Preconditions:

1. 7.14.0 BC-4 Kibana cloud environment should be available.
2. Agents must be installed using Default policy having System integration.

Steps to reproduce:

1. Login to Kibana environment.
2. Navigate to Agent Logs tab.
3. Select elastic-agent.filebeat and elastic-agent.metricbeat from dataset dropdown filter.
4. Apply error filter from log level dropdown.
5. Observe below errors:

12:31:41.786
elastic_agent.metricbeatmetricbeat-json.log
[
elastic_agent.metricbeatmetricbeat-json.log
][error] elastic-agent-client got error: rpc error: code = Canceled desc = context canceled
12:31:42.275
elastic_agent.filebeatfilebeat-json.log
[
elastic_agent.filebeatfilebeat-json.log
][error] elastic-agent-client got error: rpc error: code = Canceled desc = context canceled

Windows agent Logs:
Win-logs.zip

MAC agent Logs:
Mac-logs.zip

Linux .tar agent Logs:
Linux-tar-logs.zip

Policy:
elastic-agent (2).zip

Note:

• These logs are also observed for MAC, Linux tar, Linux .rpm, Linux .deb and Linux ARMx64 agents.
• No impact on Agent functionality is observed.
• These logs are generated at regular intervals of time.

Expected Result:
"rpc error: code = Canceled desc" error logs should not be generated on setting logging level to debug.

Screenshots:

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[amolnater-qasource]

@manishgupta-qasource Please review.

[manishgupta-qasource]

Reviewed & assigned to @EricDavisX

[EricDavisX]

@ruflin @michalpristas I'll ask you both, if you think this warrants any research for why Agent is throwing errors. Ideally, Agent wouldn't, but as it happens only once and only on start up and we observe no other malfunctioning behaviors we could close it out. Or leave it open for if someone wants to look at it as a tech-debt type item.

[ruflin]

It might be related to timing. I marked is a bug I think we should investigate this. It might hide another more sever issue.

[michalpristas]

looked into it.
it reports context cancelled on beats shutdown/restart. so not a bug which would affect behaviour it just produces this error log line which may be misleading.
i will add a filter for context cancel error after stop is called, also i can add it to 7.14 if we make it in time but i don't see it as that burning

[amolnater-qasource]

Hi @EricDavisX
We have revalidated these error logs on 7.15.0 BC-2 Kibana cloud environment.
We haven't observed exactly same error logs on any of our agents[Windows, MAC and Linux .tar].

However, just on installing MAC and linux .tar agents we have observed similar below error logs.
Screenshot:

Note:

• These logs are NOT OBSERVED FOR WINDOWS agent.

Build details:

Build: 43742
Commit: 113d5f2a745ab9be7d567d13ff906dfd0f5b7a9a
Artifact Link: https://staging.elastic.co/7.15.0-642a73fa/summary-7.15.0.html

Hence we are reopening this issue.
cc: @michalpristas
Thanks

[ruflin]

@amolnater-qasource To reproduce this, is still enabling debug logging required or this just happens on enrollment on OS X. Does the Elastic Agent go afterwards back to green or it stays in error state.

Definitively something we must investigate further.

[amolnater-qasource]

@ruflin These errors just shows up on just after installing the elastic-agent on mac and linux .tar. So just to avoid any duplicate issues and as this has no impact on functionality we have re-opened this pre-existing issue related to error logs.
Further agent remains Healthy throughout.
Thanks

[michalpristas]

these are coming from somewhere else, i will address this path as well

[amolnater-qasource]

Hi @EricDavisX
We have revalidated the similar second reported issue at #27062 (comment) on 7.15.0 BC-5.
We have found it fixed now.

Observations:

• No error logs observed for Linux and MAC agent after installation with policy having only System integration.

Build details:
Build: 43957
Commit: 0239ff6864dd9930cfe9bcd9a679272f2b7465c2
Artifact Link: https://staging.elastic.co/7.15.0-edd09abf/summary-7.15.0.html

Screenshot:

Thanks