Enrolling on Windows is flaky

[adam-stokes]

There seems to be a necessary wait period after fleet is setup according to Kibana before we can enroll an agent. For now we will work around it with a utils.Sleep but we should investigate <fleet-server>:8220/api/status to see what we can use there for determining when Fleet server is actually ready

[adam-stokes]

Linking elastic/beats#27836

[adam-stokes]

I think we need to make sure we're capturing /usr/share/elastic-agent/state/data/logs/ on the fleet server

[adam-stokes]

The current flakyiness is due to a 400 BadRequest error:

[2021-09-28T06:23:32.430Z] time="2021-09-28T06:23:31Z" level=error msg="Error executing command" args="[install -e -v --force --insecure --enrollment-token=WS1JUUszd0J0TzRsX0ppXzBMVXE6N3JGVjFGOUxSd2lHVktfdG84T3ZjUQ== --url http://10.224.0.24:8220]" baseDir=. command="C:\\elastic-agent\\elastic-agent.exe" error="exit status 1" stderr="2021-09-28T06:23:27.512Z\tINFO\t[composable.providers.docker]\tdocker/docker.go:43\tDocker provider skipped, unable to connect: protocol not available\n2021-09-28T06:23:27.612Z\tINFO\tcapabilities/capabilities.go:59\tcapabilities file not found in C:\\elastic-agent\\capabilities.yml\n2021-09-28T06:23:30.035Z\tWARN\t[tls]\ttlscommon/tls_config.go:98\tSSL/TLS verifications disabled.\n2021-09-28T06:23:30.178Z\tINFO\tcmd/enroll_cmd.go:432\tStarting enrollment to URL: http://10.224.0.24:8220/\nError: fail to enroll: fail to execute request to fleet-server: status code: 400, fleet-server returned an error: BadRequest\nFor help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.0/fleet-troubleshooting.html\n2021-09-28T06:23:31.126Z\tINFO\t[composable.providers.docker]\tdocker/docker.go:43\tDocker provider skipped, unable to connect: protocol not available\n2021-09-28T06:23:31.226Z\tINFO\tcapabilities/capabilities.go:59\tcapabilities file not found in C:\\elastic-agent\\capabilities.yml\n2021-09-28T06:23:31.475Z\tINFO\t[composable.providers.docker]\tdocker/docker.go:43\tDocker provider skipped, unable to connect: protocol not available\n2021-09-28T06:23:31.578Z\tINFO\tcapabilities/capabilities.go:59\tcapabilities file not found in C:\\elastic-agent\\capabilities.yml\nError: enroll command failed with exit code: 1\nFor help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.0/fleet-troubleshooting.html\n"

It is not easy to reproduce and we are instrumenting some additional data collection in the pipeline to help determine what's going on.

[cachedout]

@adam-stokes Hi! I'm searching for a bit of clarity on this issue. It looks like we moved it out of the backlog on September 27 but it hasn't seen much activity since then.

You wrote:

we are instrumenting some additional data collection in the pipeline to help determine what's going on.

Has that instrumentation been added? I'm just trying to get a sense of what specifically we're trying to accomplish with this issue. Until we can articulate that clearly, I'm not sure that this is ready to be added to the list of near-term work.

[adam-stokes]

This is a hard problem to track down, it doesn't happen consistently, right now this is on the back burner until I can finish out #1740

[cachedout]

Thanks, @adam-stokes . I think we should probably move this into the backlog until we have a clear plan about what to do here. WDYT?

[mdelapenya]

I think we can close this one, as the Windows machine on AWS is working on a daily basis