[Fleet] Create task that periodically unenrolls inactive agents

[criamico]

Closes #179399

Summary

Create a new periodic task that unenrolls inactive agents based on unenroll_timeout set on agent policies

In the agent policy settings there is now a new section:

Testing

• Create a policy with unenroll_timeout set to any value
• Enroll many agents to a policy and make them inactive - you can use Horde or the script in `fleet/scripts/create_agents' that can directly create inactive agents
• Leave the local env running for at least 10 minutes
• You should see logs that indicate that the task ran successfully and remove the inactive agents
  
  Note that the executed unenroll action is also visible in the UI:
  
• If there are no agent policies with unenroll_timeout set or there are no inactive agents on those policies, you should see logs like these:
  

Checklist

• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios

[obltmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[criamico]

/ci

[criamico]

The PR is not 100% complete but I'm opening it for early review to get some feedback on the approach I used.

The agent policy query fetches max 500 policies and the agents query max 1000 to avoid scale issues; for this reason the task runs every 10 min. If there are no policies with unenroll_timeout set or there are no inactive agents on the policies with unenroll_timeout, the task exits early.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[kpollich]

I tested locally and left the env running for a while, sometimes I was reassigning inactive agents to a policy with unenrollment timeout set and I could see the agents unenrolling every 10 minutes:

Could we somehow flag these actions as coming from the task somehow so the user knows the "origin" of the unenrollment? It'd be great if we could say "17 inactive agents were automatically unenrolled" or something along those lines.

[criamico]

Could we somehow flag these actions as coming from the task somehow so the user knows the "origin" of the unenrollment?

@kpollich I'll see if it's possible to distinguish those actions from the ones performed from the user. I don't know if there's a way to mark the actions differently.

[criamico]

@kpollich in 6758e8c I added a prefix to the actionId so those unenrollment actions are marked differently.

This is how it appears in the UI:

[juliaElastic]

LGTM, we should probably create an ingest-docs request to document this feature, stating that the inactive agents are going to be unenrolled at a rate of 1k/10m.

[kpollich]

Thanks Cristina that UI change looks great!

[criamico]

@elasticmachine merge upstream

[ersin-erdal]

ResponseOps changes LGTM

[criamico]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 08ffcfc

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
fleet	1.8MB	1.8MB	+14.0B

History

• 💔 Build #228181 failed 2de2dc9
• 💔 Build #228131 failed dafca57
• 💔 Build #228112 failed 16f6562
• 💚 Build #226568 succeeded 5c1eab3
• 💚 Build #226525 succeeded 6758e8c
• 💚 Build #226467 succeeded a45a9ef

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @criamico