[Ingest Manager] Allow to filter agent list by the package they are running

[nchaulet]

Description

We want to allow users, and other plugins (endpoint) to filter agent by the packages they are running.

Implementation

We can populate a field packages: string[] when the agent ack for a config change. Users will be able to filter agents with a kuery like fleet-agents.packages:endpoint

cc @nnamdifrankie

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[nnamdifrankie]

This will work, we will have another ticket for status changes.

[nnamdifrankie]

https://github.com/elastic/endpoint-app-team/issues/451

[ruflin]

Should this information be per agent or per config? We know on the config level for which packages it contains configs. Do we really need to duplicate the data? I understand that the query is run agains the agent but this could forward the request to the config object. Issue with that is perhaps speed?

[jen-huang]

++ to the above comment. The PR (#69731) implements this in such a way that package information is written to the agent saved object when the agent acknowledges a config change. If a package was added to an agent config but the agent has not yet picked up the change, the results of a query for configs that use that package vs. a query for agents that use that package will differ.

Personally I'd like to avoid writing to the agent saved object as it's yet another piece of information we need to keep in sync, but maybe knowing what is actually running on the agent vs what should be running on the agent was an important distinction here?

If it isn't, we can expose another service/endpoint that queries packages against agent configs and returns agents that are using those matching configs.

[nnamdifrankie]

but maybe knowing what is actually running on the agent vs what should be running on the agent was an important distinction here

I think it is preferable to know what is actually running because that is more likely to have corresponding event history inside in metadata index. i.e. the agent may have also transmitted some data. The downside of the "should be running" is waste of processing events that may not produce any work on the endpoint side.

[nnamdifrankie]

Also if you don't mind me asking which approach is a larger effort, hopefully the running approach. I am asking so we can gauge how we are tracking for 7.9 on delivering the services.

[nchaulet]

@ruflin @jen-huang I think endpoint want to know what is currently running on one agent, (this will allow to detect when endpoint is added/removed). Also from an end-user perspective I think it's a nice feature too.