Allow setting ES api key via env variable in container mode

[swiatekm]

What does this PR do?

Remove defaults for username and password when running in container mode, and also allow api_key to be set in that mode via the ELASTICSEARCH_API_KEY environment variable.

Note that the first of these changes is breaking. I've added two separate changelog entries to make this clear.

Why is it important?

It's very useful for users to be able to use an ES api key for authentication without needing to override the whole config file.

Checklist

• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

This is a breaking change for users who rely on the default username and password in container mode. These users will need to explicitly set these values after this change is rolled out. After the upgrade, the agent will simply fail to start, and will complain about lack of authentication credentials.

How to test this PR locally

DEV=true SNAPSHOT=true EXTERNAL=true PACKAGES=docker mage package
docker run -e ELASTICSEARCH_API_KEY="***" -e ELASTICSEARCH_HOSTS="***" --rm docker.elastic.co/beats/elastic-agent-complete:9.0.0-SNAPSHOT

This should successfully start if the host and api key are valid.

Related issues

Closes #97.

[mergify]

This pull request does not have a backport label. Could you fix it @swiatekm? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-v8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[jlind23]

@ycombinator @nimarezainia if this is considered a breaking shouldn't make it land only in 9.x?

[swiatekm]

@jlind23 that was my intent, though I believe @cmacknz mentioned wanting to consider it for 8.x as well.

FYI, I'm waiting for E2E tests to be successfully runnable on this change before submitting it properly.

[jlind23]

Thanks @swiatekm! @cmacknz do we really need it for 8.x?

FYI, I'm waiting for E2E tests to be successfully runnable on this change before submitting it properly.

This should be shortly solved probably tomorrow EOD the latest.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[jlind23]

@swiatekm Are we good to merge this in 9.0?
cc @ycombinator

[swiatekm]

@swiatekm Are we good to merge this in 9.0? cc @ycombinator

I think so. I considered adding an E2E test for it, but it ended up being a lot of setup for such a trivial check.

[ycombinator]

Yes, this is a breaking change and is OK to merge (only) in 9.0. @swiatekm go ahead and merge whenever you get the chance.