[Fleet] Agent tags not loading with default status filters, bulk update tags not working

[juliaElastic]

Kibana version: 8.7.0

Describe the bug: With the default filters (Healthy, Unhealthy, Updating, Offline) Agent Tags are not loading, and bulk update tags doesn't update any agents. Something might be wrong with the default filters.

Steps to reproduce:

1. Start kibana on prem or in cloud
2. Enroll agents with horde
3. See agents with tags in agent list
4. Tags filters are not loading with default filters, they load when removing all status filters
5. Select all agents, add/remove tags with default status filters, the action does not do anything, and Agent activity shows "0 agents updated"

Expected behavior:
Tags filters should be loaded with default status filters, bulk update tags should work.

Screenshots (if relevant):

Any additional context:

Caused by #147552 where the status field has changed.

Tags request:

http://localhost:5601/julia/api/fleet/agents/tags?kuery=status%3Aonline%20or%20(status%3Aerror%20or%20status%3Adegraded)%20or%20(status%3Aupdating%20or%20status%3Aunenrolling%20or%20status%3Aenrolling)%20or%20status%3Aoffline&showInactive=false

Bulk update tags request:

POST http://localhost:5601/julia/api/fleet/agents/bulk_update_agent_tags"
"{\"agents\":\"status:online or (status:error or status:degraded) or (status:updating or status:unenrolling or status:enrolling) or status:offline\",\"tagsToAdd\":[\"new\"],\"tagsToRemove\":[]}"

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[juliaElastic]

I think the root cause is that we have several places where the status filters can be used in the code, and those places have to be updated to use the runtime fields: https://github.com/elastic/kibana/pull/147552/files#diff-f7377ed9ad56eaa8ea188b64e957e771ccc7a7652fd1eaf44251c25b930f8448

It is easy to fix the /tags by passing the runtime fields, but the update agent tags uses an updateByQuery call, which doesn't support runtime fields. The fix here is trickier, we might have to bring back the previous logic of status filters without the runtime field.

As a quick fix we could remove the status filters from update tags (at least when the filters are set to the default values) to unblock the feature.

I reviewed the code where the agents index is queried with filters, and none of the other places have missing runtime fields.

cc @hop-dev @kpollich

[joshdover]

It is easy to fix the /tags by passing the runtime fields, but the update agent tags uses an updateByQuery call, which doesn't support runtime fields.

This is a pretty big caveat to runtime fields usage. Wouldn't this also apply to bulk upgrades where we update all the agent documents' update_started_at fields?

I wonder if we could find a way to re-use the runtime script from the mappings in the query itself. Not sure if it's supported in the ways we would need, but we could explore using a stored script that is called from the runtime field and the update_by_query calls.

[juliaElastic]

This is a pretty big caveat to runtime fields usage. Wouldn't this also apply to bulk upgrades where we update all the agent documents' update_started_at fields?

No, upgrade is not using updateByQuery.

I wonder if we could find a way to re-use the runtime script from the mappings in the query itself. Not sure if it's supported in the ways we would need, but we could explore using a stored script that is called from the runtime field and the update_by_query calls.

I thought about this, though update tags already uses a script with updateByQuery, so I don't think we could combine that with a stored script.

@hop-dev suggested another solution here to do a search first with runtime fields, and then do the updateByQuery, I've started exploring that, though it is a bigger refactor, as we have to bring back batch processing, as we can't pass all agent ids at once from the search.