Integrate Elastic Endpoint's log output and available stats into Elastic Agent 'diagnostic' collection

[EricDavisX]

Describe the enhancement:
would like the Elastic Agent to automatically be able to collect Endpoint logs and any configuration or state that is useful during debugging, when the 'diagnostic' call is made at cli.

Describe a specific use case for the enhancement or feature:
Refer to the Agent's implementation Documentation ticket that started the discussion:
elastic/observability-docs#1284 (comment)

We have confirmed that Beats and OSQuery logs are collected, but that Endpoint is not, when the Agent's 'diagnostic' cli usage is invoked. We can discourse various levels of effort and cost-value proposition here.

1. Lowest effort / value: on Agent side when invoked Agent can at least note that no Endpoint logs are collected and maybe point to the Endpoint doc's for ref.

2. higher value / more effort (how much is TBD): Have Agent actually include Endpoint's logs in its collection process. The tools to do this are presumably / believed written, but it would need adjustment on Agent side to take in the log location after getting it sent in automatically (at started up?) or calling out to Endpoint to get it.

---

This is for the Agent side work, Endpoint will have a separate ticket logged to track, and I'll like back.
This was discussed in email/slack first and I'm transferring it here for tracking.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[jlind23]

@AndersonQ @lykkin can one of you take this one as soon as the 8.2 start?

[AndersonQ]

I could take it :)

[amolnater-qasource]

Hi Team
We have revalidated this issue on 8.3.1 BC1 Kibana cloud-staging environment and found it fixed now.

• sudo elastic-agent diagnostics collect now collects elastic agent and endpoint security logs.

Screenshot:

Logs:
elastic-agent-diagnostics-2022-06-30T09-06-31Z-00.zip

Build details:
BUILD: 53549
COMMIT: a4f8dc60edb19553f16c166ea79c83c16572897a

Hence marking this as QA:Validated.
Thanks