8.5.0-SNAPSHOT Endpoint can't connect to ES

[kevinlog]

When installing the latest 8.5.0-SNAPSHOT Agent and Endpoint, it cannot connect to Elasticsearch.

The version I used in this bug report: https://snapshots.elastic.co/8.5.0-c7913db3/downloads/beats/elastic-agent/elastic-agent-8.5.0-SNAPSHOT-linux-arm64.tar.gz

There are two somewhat different scenarios depending on the protocol you use when installing the Agent.

Using --insecure flag when installing Agent

The first scenario I encountered was when I was doing local development and testing. I used the --insecure flag when installing the Agent on my host to connect to my insecure local development server.

When installing, I can see in the logs that the Endpoint cannot connect to Elasticsearch. Find a snippet below, full logs attached.

...
{"@timestamp":"2022-09-14T20:06:34.801859694Z","agent":{"id":"6d7207c2-1e6e-4fa8-b8b9-70548a06e232","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":86,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:86 Elasticsearch connection is down","process":{"pid":106912,"thread":{"id":106919}}}
...

I can see in the policy yaml pulled from the host itself that there is ES connection information. Using this same Policy, I can successfully connect 8.4.0 Agent/Endpoint. Find a snipped below, full policy yml attached.

...
output:
  elasticsearch:
    api_key: ****
    hosts:
    - http://10.0.0.199:9200
revision: 4
...

endpoint.log.zip
elastic-endpoint.yml.zip

Connecting to a secure cloud instance (no --insecure flag)

I also was unable to get the Endpoint to connect to ES when connecting to a secure cloud instance.

I tried to reproduce again on a secure cloud server to eliminate my insecure, local http connection and I'm still seeing rhe same end result of data not making it to my stack, although the logs have a bit more context surrounding invalidated API keys.

{"@timestamp":"2022-09-14T20:44:48.926180374Z","agent":{"id":"e4763429-78f6-4532-9e53-09655cec0814","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"notice","origin":{"file":{"line":86,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:86 Elasticsearch connection is down","process":{"pid":114033,"thread":{"id":114039}}}
{"@timestamp":"2022-09-14T20:44:53.96152332Z","agent":{"id":"e4763429-78f6-4532-9e53-09655cec0814","type":"endpoint"},"ecs":{"version":"1.11.0"},"log":{"level":"error","origin":{"file":{"line":232,"name":"ElasticsearchClient.cpp"}}},"message":"ElasticsearchClient.cpp:232 HTTP Status Code (401): {\"error\":{\"additional_unsuccessful_credentials\":\"API key: api key [****] has been invalidated\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]},\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"root_cause\":[{\"additional_unsuccessful_credentials\":\"API key: api key [****] has been invalidated\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]},\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"type\":\"security_exception\"}],\"type\":\"security_exception\"},\"status\":401}","process":{"pid":114033,"thread":{"id":114039}}}

Endpoint Version
8.5.0-SNAPSHOT

version: 8.5.0-SNAPSHOT, compiled: Tue Sep 13 13:00:00 2022, branch: main, commit: c7a37b9cfcbccdf257e024fb31119b91f24ce5ba

endpoint-secure-connection.log.zip
elastic-endpoint-secure-connection.yml.zip

For confirmed bugs, please report:

• Version: 8.5.0-SNAPSHOT Agent
• Operating System: Ubuntu arm64

Steps to reproduce the behavior:

1. Install Agent/Endpoint from Fleet
2. Wait for data to come in, but it doesn't.
3. Check the Endpoint logs
4. See that it can't connect to ES

[cmacknz]

We are seeing the same problem with Beats started by agent, with a similar log message that an API key has been invalidated. This is not specific to Endpoint.

[joshdover]

@AndersonQ can we test reverting #1684? If that fixes the issue, we may just want to do that to unblock other teams while we figure out the real fix.

[cmacknz]

+1 to reverting #1684 while we debug this, we need to unblock others and we will not want this bug in the first 8.5.0 build candidate scheduled to be created on Wednesday.

[AndersonQ]

I have no problem reverting it, that's a good idea. I'm still trying to reproduce it to debug, but reverting it is also a way to check if it's indeed the root cause of the issue :)

[AndersonQ]

2 PRs need to be reverted, in the right I guess. I'll open the PRs to revert it

[kevinlog]

@AndersonQ

The QA team on our side encountered this bug, but found an odd workaround. They add os_query to the Agent and then Endpoint seems to install. I'm not sure if that helps with finding the root cause.

More about the bug and potential workaround here: elastic/kibana#141036