[Elastic Agent] Enable log shipping of endpoint-security by Elastic Agent

[blakerouse]

What does this PR do?

Starts shipping the logs of endpoint-security to elasticsearch.

This also refactors the code some to pass the program.Spec around instead of needing to keep checking for it from the SupportedMap. This is how the code path determines the path for the endpoint-security log paths.

Why is it important?

So all log information from an Elastic Agent running endpoint-security can be observed.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Build the Elastic Agent with endpoint-security manually placed in the download folder. Enable Endpoint Security in Fleet and notice that all the Endpoint Security logs show up with event.dataset: elastic_agent.endpoint-security

Related issues

• Closes [Elastic Agent] Allow the agent to collect and send the log of endpoint. #21796

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: [Pull request #22526 updated]

• Start Time: 2020-11-11T14:47:29.245+0000

• Duration: 23 min 27 sec

Test stats 🧪

Test	Results
Failed	0
Passed	1396
Skipped	4
Total	1400

[elasticmachine]

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	1396
Skipped	4
Total	1400

[ph]

@blakerouse do we have everything we need in the data send from endpoint to make sure we can display the log of the endpoint in the logstream UI made by @jen-huang. ?

[blakerouse]

@ph Yes it included the agent ID in the events so the LogStream component can show these logs.

[michalpristas]

I like the change, probably will make future changes simpler.
i left a comments but they are related to segmented imports. nothing critical

[ph]

@blakerouse Can add details on the following for testing that change:

AC:

• When endpoint integration is installed and runnings.
• Endpoint will write logs at :
  darwin: "/Library/Elastic/Endpoint/state/log/endpoint-.log"
  linux: "/opt/Elastic/Endpoint/state/log/endpoint-.log"
  windows: "C:\Program Files\Elastic\Endpoint\state\log\endpoint-*.log"
  - Endploint Log events will be sent to the datastream XXXX
• The events will have show up in the Logs UI. Easiest to filter by event.dataset: elastic_agent.endpoint-security

Fyi @EricDavisX

[blakerouse]

Should really need to worry about the log location, well unless it doesn't work ;-)

The events will have show up in the Logs UI. Easiest to filter by event.dataset: elastic_agent.endpoint-security

[ph]

@blakerouse but there are sent to a distinct datastream right?

[blakerouse]

@ph Yes should be similar to dataset name logs-elastic_agent-endpoint-security-default.

[EricDavisX]

The PR and the issue are closed (which is ok!) so I've logged a test tracking issue for any remaining discussion questions about test coverage. #22549

[ph]

@blakerouse We can't use logs-elastic_agent-endpoint-security-default this will cause issue when we are extracting dataset and namespace. I will create an issue. I think the other datastreams too are problematic.

[ph]

Created this issue as a followup #22551