Support for Elastic Agent V2 status

[aleksmaus]

What is the problem this PR solves?

Adds support for Elastic Agent V2 status

How does this PR solve the problem?

Extends the current checkin logic with new payload.

Open question: What fields of any need to be indexed? Need to update the .fleet-agents index mapping with newly added fields, and need to know if any details of the components/units need to be searchable or we should not index it at all.

How to test this PR locally

This needs to be tested along with the corresponding agent changes from elastic/elastic-agent#916

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation

Related issues

• Relates Display which integration is causing the agent to become unhealthy elastic-agent#100
• Requires Expand check-in payload for V2 elastic-agent#916 (should be backwards compatible though and work for the older agent)

Screenshots

The fragment of the .fleet-agents document with V2 data.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-08-16T12:46:05.857+0000

• Duration: 12 min 38 sec

Test stats 🧪

Test	Results
Failed	0
Passed	304
Skipped	1
Total	305

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[ph]

@kpollich Maybe you review concerning the fields that need to be indexed to correctly report the state of integration.

I would assume that the id, type and status need to be indexed probably keyword would be appropriate for them.
I think I would like to answer the following question.

Which Integration are unhealthy?
What is the state of of all my endpoint integration?
Is there a problem with the filestream input?

@nimarezainia You may want to take a look at the format above if there is are telemetry that could help us?

[ph]

@aleksmaus We should probably assert the data we are receiving from Elastic Agent before sending that to Elasticsearch? Maybe we should also put some kind of limit, maybe a size based limit on the payload field to reduce the risk or document size explosion?

[aleksmaus]

This approach is basically copied from parseMeta that is used currently for "local_metadata" handling which is another blob that agent can send whatever to the fleet server.
There is already a limit check on the checking body size, default is defaultCheckinMaxBody = 1024 * 1024.
There is already the parsing of the components happens in the parseComponents which validates that is a parsable JSON. I can add a check if the components is an array.

Do we need to add field by field check? What are guarantees which fields are required vs which fields are optional?

[aleksmaus]

Will add strongly typed parsing to the "components" property tomorrow, only things that are specified in the V2 at the moment will be stored for the components, and it will need to be extended for anything extra in the future.

[ph]

Yes, once everything is V2 we should be able to be more strict on any incoming data, not sure during the transition period we can do this.

[aleksmaus]

Sounds good. Will leave it as is for now until we stabilize the schema, can tighten validation later.

[ph]

I always forgot the bytes.Equal does check the len before doing the bytes check, I think.

[aleksmaus]

This approach is copied from parseMeta, works fine for nils. It seems like the compiler is optimized for this as well
https://cs.opensource.google/go/go/+/refs/tags/go1.19:src/bytes/bytes.go;l=19

[blakerouse]

Looks great.

I think we need to determine what we want to do when the status, message, and components change from the Elastic Agent side. Should it disconnect from the long-poll and reconnect and send updated status, message, and components everytime it changes? At the moment the Elastic Agent only sends it based on time-interval. I think changing it to send it on every change would provide really good observability but the scale and load of that might be high.

Something to think about, but not something that is blocking this to land as it is now. Same with my inline comment to Josh.

[blakerouse]

@joshdover Do we want to store the status, message, and components as they change across time in a datastream so we can track the changes of an Elastic Agent across time? I think from an alerting standpoint it would be beneficial but from a scalability it would add more data and load.