[Agent Upgrade]: Agent upgrade fails on upgrade from 8.2.0>8.3.0-Snapshot and goes Unhealthy on all OS's.

[ghost]

Description:
Agent upgrade fails on upgrade from 8.2.0>8.3.0-Snapshot and goes Unhealthy on all OS's.

Build Details:
VERSION: 8.3.0-Snapshot
BUILD: 53295
COMMIT: 6ef457053adce432a3b4d5b016997f92488d857d
ARTIFACT LINK OF 8.2.0 : https://www.elastic.co/downloads/past-releases/elastic-agent-8-2-0

Preconditions:

1. Elastic 8.3.0-Snapshot environment should be deployed.

Steps to Reproduce:

1. Login to Kibana environment.
2. Navigate to Fleet>Agents tab.
3. Upgrade lower version agents from the Agents tab.
4. Observe that Agent upgrade fails on upgrade from 8.2.0>8.3.0-Snapshot and goes Unhealthy on all OS's.

Actual Result:
Agent upgrade fails on upgrade from 8.2.0>8.3.0-Snapshot and goes Unhealthy on all OS's.
Also, If agent is upgraded from 8.2.0>8.2.1 the agent gets upgraded to v8.2.1.
Further, if agent is upgraded from 8.2.1>8.3.0-Snapshot the agent fails to upgrade and rolls back to v8.2.0 and goes to unhealthy state.

Expected Result:
Agent upgrade should not fail on upgrade from 8.2.0>8.3.0-Snapshot and should not go to unhealthy state.

Screenshot:

Logs:
logs.zip

[manishgupta-qasource]

Secondary review for this ticket is Done

[joshdover]

@samratbhadra-qasource are there any logs available on these unhealthy agents?

[ghost]

Hi @joshdover
Thanks for looking into this.
Please find below the logs of these unhealthy agents:

• Windows
  logs.zip
• Linux
  Linux-elastic-agent-diagnostics-2022-06-09T05-07-23Z-00.zip
• MAC
  MAC-elastic-agent-diagnostics-2022-06-09T05-13-39Z-00.zip

Please do let us know if anything else is required from our end.

[wearmg]

Not sure if this is related to a problem I am having. I'm running 8.2.2 with agents at 8.1.2 and when I attempt to upgrade them they go unhealthy. I did a procmon and found that after they download the zip file they look for c:\program files\elastic\agent\data\elastic-agent-[version code]\downloads\elastic-agent-8.2.2-windows-x86_64.zip.asc. I found that if I manually download and place that .asc file in the directory on a client then I can remotely upgrade them ok from fleet.

[joshdover]

@wearmg You're running into a known issue with agent upgrades on 8.1.2: https://www.elastic.co/guide/en/fleet/8.1/release-notes-8.1.2.html

This feature is currently in beta, so we apologize for the bug here. You'll need to manually reinstall the agent on affected hosts (or use the workaround you discovered).

[joshdover]

This does appear to be a similar issue to #275, here are the logs from the diags provided above:

failed to dispatch actions, error: failed verification of agent binary: 3 errors occurred:
	* checksum for "elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz" was not found in "/opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.sha512"
	* checksum for "elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz" was not found in "/opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.sha512"
	* checksum for "elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz" was not found in "/opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.sha512"

View JSON logs

{"log.level":"info","@timestamp":"2022-06-08T08:55:12.480Z","log.origin":{"file.name":"cmd/run.go","file.line":153},"message":"APM instrumentation disabled","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T08:55:12.482Z","log.origin":{"file.name":"application/application.go","file.line":78},"message":"Detecting execution mode","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T08:55:12.483Z","log.origin":{"file.name":"application/application.go","file.line":102},"message":"Agent is managed by Fleet","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T08:55:12.484Z","log.origin":{"file.name":"capabilities/capabilities.go","file.line":59},"message":"capabilities file not found in /opt/Elastic/Agent/capabilities.yml","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T08:55:12.858Z","log.logger":"composable.providers.docker","log.origin":{"file.name":"docker/docker.go","file.line":43},"message":"Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T08:55:12.859Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":63},"message":"Starting stats endpoint","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T08:55:12.859Z","log.origin":{"file.name":"application/managed_mode.go","file.line":299},"message":"Agent is starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T08:55:12.859Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":65},"message":"Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/elastic-agent.sock (configured: unix:///opt/Elastic/Agent/data/tmp/elastic-agent.sock)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T08:55:14.522Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"New State ID is 2mkuxoUH","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T08:55:14.522Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":49},"message":"Converging state requires execution of 3 step(s)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T08:55:21.330Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T04:55:21-04:00 - message: Application: metricbeat--8.2.0[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T08:55:26.434Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T04:55:26-04:00 - message: Application: metricbeat--8.2.0[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T08:55:28.505Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T04:55:28-04:00 - message: Application: filebeat--8.2.0[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T08:55:29.215Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-install' skipped for filebeat.8.2.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T08:55:29.408Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T04:55:29-04:00 - message: Application: filebeat--8.2.0--36643631373035623733363936343635[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T08:55:30.347Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-install' skipped for metricbeat.8.2.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T08:55:30.465Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T04:55:30-04:00 - message: Application: metricbeat--8.2.0--36643631373035623733363936343635[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T08:55:30.469Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":66},"message":"Updating internal state","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T08:55:33.277Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T04:55:33-04:00 - message: Application: filebeat--8.2.0[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T08:55:33.816Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T04:55:33-04:00 - message: Application: filebeat--8.2.0--36643631373035623733363936343635[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T08:55:35.140Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T04:55:35-04:00 - message: Application: metricbeat--8.2.0--36643631373035623733363936343635[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:35.865Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"New State ID is rxQt2Qvo","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:35.866Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":49},"message":"Converging state requires execution of 4 step(s)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:36.793Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-install' skipped for metricbeat.8.2.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:36.793Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-start' skipped for metricbeat.8.2.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:36.807Z","log.origin":{"file.name":"process/configure.go","file.line":50},"message":"initiating restart of 'metricbeat' due to config change","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:37.324Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T05:08:37-04:00 - message: Application: metricbeat--8.2.0[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:37.720Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T05:08:37-04:00 - message: Application: metricbeat--8.2.0[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:42.150Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T05:08:42-04:00 - message: Application: metricbeat--8.2.0[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:43.247Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T05:08:43-04:00 - message: Application: endpoint-security--8.2.0[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:44.355Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-install' skipped for filebeat.8.2.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:44.355Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-start' skipped for filebeat.8.2.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:44.357Z","log.origin":{"file.name":"process/configure.go","file.line":50},"message":"initiating restart of 'filebeat' due to config change","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:44.874Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T05:08:44-04:00 - message: Application: filebeat--8.2.0[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:45.353Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T05:08:45-04:00 - message: Application: filebeat--8.2.0[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:46.068Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-install' skipped for filebeat.8.2.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:46.068Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-start' skipped for filebeat.8.2.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:46.070Z","log.origin":{"file.name":"process/configure.go","file.line":50},"message":"initiating restart of 'filebeat_monitoring' due to config change","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:46.580Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T05:08:46-04:00 - message: Application: filebeat--8.2.0--36643631373035623733363936343635[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:46.873Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T05:08:46-04:00 - message: Application: filebeat--8.2.0--36643631373035623733363936343635[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:47.744Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-install' skipped for metricbeat.8.2.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:47.744Z","log.origin":{"file.name":"operation/operator.go","file.line":307},"message":"operation 'operation-start' skipped for metricbeat.8.2.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:47.747Z","log.origin":{"file.name":"process/configure.go","file.line":50},"message":"initiating restart of 'metricbeat_monitoring' due to config change","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:48.257Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T05:08:48-04:00 - message: Application: metricbeat--8.2.0--36643631373035623733363936343635[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:48.663Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T05:08:48-04:00 - message: Application: metricbeat--8.2.0--36643631373035623733363936343635[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:48.664Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":66},"message":"Updating internal state","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:49.731Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T05:08:49-04:00 - message: Application: filebeat--8.2.0[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:51.136Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T05:08:51-04:00 - message: Application: filebeat--8.2.0--36643631373035623733363936343635[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:08:53.177Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T05:08:53-04:00 - message: Application: metricbeat--8.2.0--36643631373035623733363936343635[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:09:25.406Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T05:09:25-04:00 - message: Application: endpoint-security--8.2.0[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to CONFIG:  - type: 'STATE' - sub_type: 'CONFIG'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:09:26.361Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T05:09:26-04:00 - message: Application: endpoint-security--8.2.0[d8ba05f2-d291-4628-b449-c921638baa40]: State changed to RUNNING: Protecting with policy {bf774c68-8025-4ce3-9b54-2b38e9dacc08} - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:31:19.410Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T05:31:19-04:00 - message: Application: [d8ba05f2-d291-4628-b449-c921638baa40]: State changed to UPDATING: Update to version '8.3.0-SNAPSHOT' started - type: 'STATE' - sub_type: 'UPDATING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-06-08T09:35:43.619Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2022-06-08T05:35:43-04:00 - message: Application: [d8ba05f2-d291-4628-b449-c921638baa40]: State changed to FAILED: failed verification of agent binary: 3 errors occurred:\n\t* fetching asc file from '/opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.asc': open /opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.asc: no such file or directory\n\t* fetching asc file from https://snapshots.elastic.co/8.3.0-f13a89b4/downloads/beats/elastic-agent/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.asc: failed loading public key: Get \"https://snapshots.elastic.co/8.3.0-f13a89b4/downloads/beats/elastic-agent/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.asc\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\n\t* fetching asc file from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.asc: failed loading public key: Get \"https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.asc\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n\n - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-06-08T09:35:43.625Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":179},"message":"failed to dispatch actions, error: failed verification of agent binary: 3 errors occurred:\n\t* fetching asc file from '/opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.asc': open /opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.asc: no such file or directory\n\t* fetching asc file from https://snapshots.elastic.co/8.3.0-f13a89b4/downloads/beats/elastic-agent/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.asc: failed loading public key: Get \"https://snapshots.elastic.co/8.3.0-f13a89b4/downloads/beats/elastic-agent/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.asc\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)\n\t* fetching asc file from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.asc: failed loading public key: Get \"https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.asc\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n\n","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-06-08T09:35:43.626Z","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'error'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:40:19.186Z","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'online'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:43:03.599Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T05:43:03-04:00 - message: Application: [d8ba05f2-d291-4628-b449-c921638baa40]: State changed to UPDATING: Update to version '8.2.1' started - type: 'STATE' - sub_type: 'UPDATING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-06-08T09:45:19.363Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2022-06-08T05:45:19-04:00 - message: Application: [d8ba05f2-d291-4628-b449-c921638baa40]: State changed to FAILED: failed verification of agent binary: 2 errors occurred:\n\t* fetching asc file from '/opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.2.1-linux-x86_64.tar.gz.asc': open /opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.2.1-linux-x86_64.tar.gz.asc: no such file or directory\n\t* fetching asc file from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.2.1-linux-x86_64.tar.gz.asc: failed loading public key: Get \"https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.2.1-linux-x86_64.tar.gz.asc\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n\n - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-06-08T09:45:19.364Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":179},"message":"failed to dispatch actions, error: failed verification of agent binary: 2 errors occurred:\n\t* fetching asc file from '/opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.2.1-linux-x86_64.tar.gz.asc': open /opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.2.1-linux-x86_64.tar.gz.asc: no such file or directory\n\t* fetching asc file from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.2.1-linux-x86_64.tar.gz.asc: failed loading public key: Get \"https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.2.1-linux-x86_64.tar.gz.asc\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\n\n","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-06-08T09:45:19.364Z","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'error'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:49:51.163Z","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'online'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T09:56:20.732Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T05:56:20-04:00 - message: Application: [d8ba05f2-d291-4628-b449-c921638baa40]: State changed to UPDATING: Update to version '8.3.0-SNAPSHOT' started - type: 'STATE' - sub_type: 'UPDATING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-06-08T09:56:27.793Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2022-06-08T05:56:27-04:00 - message: Application: [d8ba05f2-d291-4628-b449-c921638baa40]: State changed to FAILED: failed verification of agent binary: 3 errors occurred:\n\t* checksum for \"elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz\" was not found in \"/opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.sha512\"\n\t* checksum for \"elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz\" was not found in \"/opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.sha512\"\n\t* checksum for \"elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz\" was not found in \"/opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.sha512\"\n\n - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-06-08T09:56:27.795Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":179},"message":"failed to dispatch actions, error: failed verification of agent binary: 3 errors occurred:\n\t* checksum for \"elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz\" was not found in \"/opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.sha512\"\n\t* checksum for \"elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz\" was not found in \"/opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.sha512\"\n\t* checksum for \"elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz\" was not found in \"/opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.sha512\"\n\n","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-06-08T09:56:27.795Z","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'error'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T10:01:11.755Z","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'online'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T10:54:57.780Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-06-08T06:54:57-04:00 - message: Application: [d8ba05f2-d291-4628-b449-c921638baa40]: State changed to UPDATING: Update to version '8.3.0-SNAPSHOT' started - type: 'STATE' - sub_type: 'UPDATING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-06-08T10:55:00.456Z","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2022-06-08T06:55:00-04:00 - message: Application: [d8ba05f2-d291-4628-b449-c921638baa40]: State changed to FAILED: failed verification of agent binary: 3 errors occurred:\n\t* checksum for \"elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz\" was not found in \"/opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.sha512\"\n\t* checksum for \"elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz\" was not found in \"/opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.sha512\"\n\t* checksum for \"elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz\" was not found in \"/opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.sha512\"\n\n - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-06-08T10:55:00.456Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":179},"message":"failed to dispatch actions, error: failed verification of agent binary: 3 errors occurred:\n\t* checksum for \"elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz\" was not found in \"/opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.sha512\"\n\t* checksum for \"elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz\" was not found in \"/opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.sha512\"\n\t* checksum for \"elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz\" was not found in \"/opt/Elastic/Agent/data/elastic-agent-b9a28a/downloads/elastic-agent-8.3.0-SNAPSHOT-linux-x86_64.tar.gz.sha512\"\n\n","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-06-08T10:55:00.457Z","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'error'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-06-08T10:59:54.806Z","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'online'","ecs.version":"1.6.0"}

@michel-laterman could you take a look?

[michel-laterman]

I'll try to recreate and investigate

[wearmg]

@wearmg You're running into a known issue with agent upgrades on 8.1.2: https://www.elastic.co/guide/en/fleet/8.1/release-notes-8.1.2.html

This feature is currently in beta, so we apologize for the bug here. You'll need to manually reinstall the agent on affected hosts (or use the workaround you discovered).

Thanks Josh, I have ~2,300 agents sitting like this. Just wrote a powershell script to drop the .asc file on them all and so far they are looking good and i'll just have to handle the handful of macs manually.

[michel-laterman]

I'm unable to recreate the same issue on an Ubuntu 20.04 VM.
When I try to update it, the UI gets stuck with an "updating" status.

On the vm, the 8.3.0-SNAPSHOT version is present (and the symlink points to it) but is unable to start correctly.

Jun 14 16:43:39 ubuntu-focal systemd[1]: Started Elastic Agent is a unified agent to observe, monitor and protect your system..
Jun 14 16:43:39 ubuntu-focal elastic-agent[8344]: Error: fleet configuration is invalid: empty access token
Jun 14 16:43:39 ubuntu-focal elastic-agent[8344]: For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.3/fleet-troubleshooting.html
Jun 14 16:43:39 ubuntu-focal systemd[1]: elastic-agent.service: Main process exited, code=exited, status=1/FAILURE
Jun 14 16:43:39 ubuntu-focal systemd[1]: elastic-agent.service: Killing process 8371 (elastic-agent) with signal SIGKILL.
Jun 14 16:43:39 ubuntu-focal systemd[1]: elastic-agent.service: Failed with result 'exit-code'.

There are no other agents or beats running on the instance.
IIRC we had an update issue similar to this in the past

[aleksmaus]

@michel-laterman I'll dig into this, this might be related to new upgrade changes that suppose to migrate all the agent configurations into encrypted format.

[ghost]

Hi @aleksmaus
We have tested this issue on the latest 8.3.0-SNAPSHOT build and had below observations:

• 8.2.0 released Windows and MAC agent upgraded to 8.3.0-Snapshot successfully.
• 8.2.0 released Linux agent failed to upgrade and goes to Unhealthy state.

We have shared detailed observation at #173

Screenshot:

Build details:
VERSION: 8.3.0-SNAPSHOT
BUILD: 53487
COMMIT: b6dbc7712cc5c383bdd2f8392a2423f724068a2a

Thanks!