[Fleet]: Hosted fleet server gets unhealthy on 8.16 Snapshot.

[harshitgupta-qasource]

Deployment Links:

• https://cloud.elastic.co/home

Description:
Hosted fleet server gets unhealthy on 8.16 Snapshot and we have observed APM integration shouws error.

Build details:
VERSION: 8.16.0 SNAPSHOT
BUILD: 78494
COMMIT: 156a76cb03e60a89792f905642817405002099a1

Screenshot

[harshitgupta-qasource]

@amolnater-qasource Kindly review

[amolnater-qasource]

Secondary Review for this ticket is Done.

[ycombinator]

I'm able to reproduce this simply by creating an 8.16.0-SNAPSHOT deployment on ESS production in the CFT region. I downloaded the diagnostic and checked for errors in the logs. Here's what I see:

$ cat elastic-agent-20240924-1.ndjson | grep error | jq '.message'
...
"Component state changed apm-es-containerhost (STARTING->FAILED): Failed: pid '11978' exited with code '1'"
"Unit state changed apm-es-containerhost (STARTING->FAILED): Failed: pid '11978' exited with code '1'"
"Unit state changed apm-es-containerhost-elastic-cloud-apm (STARTING->FAILED): Failed: pid '11978' exited with code '1'"
"Error: error loading config file: stat apm-server.yml: no such file or directory"
"Usage:"
"apm-server [flags]"
"apm-server [command]"
"Available Commands:"
"apikey      Manage API Keys for communication between APM agents and server (deprecated)"
"export      Export current config"
"help        Help about any command"
"keystore    Manage secrets keystore"
"run         Run APM Server"
"test        Test config"
"version     Show current version info"
"Flags:"
"-E, --E setting=value      Configuration overwrite"
"-N, --N                    Disable actual publishing for testing"
"-c, --c string             Configuration file, relative to path.config (default \"apm-server.yml\")"
"--cpuprofile string    Write cpu profile to file"
"-d, --d stringArray        Enable certain debug selectors"
"-e, --e                    Log to stderr and disable syslog/file output"
"--environment string   Set the environment in which the process is running (default \"default\")"
"-h, --help                 help for apm-server"
"--httpprof string      Start pprof http server"
"--memprofile string    Write memory profile to this file"
"--path.config string   Configuration path"
"--path.data string     Data path"
"--path.home string     Home path"
...

So it seems like the apm-server.yml file is missing.

[cmacknz]

Pinged APM server team in Slack for ideas.

[cmacknz]

From Slack discussion looks related to the group membership change in the Wolfi container, where the agent user is no long in gid user. The permissions of the apm-server.yml file make it unreadable to the elastic-agent processes:

apm-server binary is 1000:1000 while apm-server.yml is 0:0

[ycombinator]

Thanks @cmacknz. I'm guessing the fix here is on the APM Server end where the apm-server.yml file (and any others) need to be readable by the Elastic Agent user?