Should the operator create enrolment tokens instead of configuring Agents to talk to Kibana

[pebrc]

The current implementation of Elastic Agent with the ECK operator configures Agents to enrol themselves in Fleet by providing them with connection information to talk to Kibana themselves:

https://www.elastic.co/guide/en/fleet/master/running-on-kubernetes-managed-by-fleet.html#_settings

- name: KIBANA_HOST
  value: ""
- name: KIBANA_FLEET_USERNAME
  value: ""
- name: KIBANA_FLEET_PASSWORD
  value: ""

That is not ideal as individual Agents have access to the Kibana API with more privileges than strictly necessary or more privileges than they would have if they were configured directly with an enrolment API key instead.

The question is whether we should change the implementation so that:

• the operator creates enrolment API keys for the Agents by talking to the Kibana API (a complication here is that we need to pick a policy in order to create such a key)
• the operator then creates the Agent deployment/daemonset with this API key obviating the need for a direct Kibana connection from individual Agents IIUC

The Fleet API is not stable/internal but documented here:
https://raw.githubusercontent.com/elastic/kibana/master/x-pack/plugins/fleet/common/openapi/bundled.json

[pebrc]

We got some feedback from the Fleet team that enrolment tokens are indeed the preferred way. I think we should strive to refactor our Agent support accordingly before we remove the "experimental" flag.

[pebrc]

We could

• introduce a new field in the CRD called agentPolicy
• if the field is populated by the user with an Agent policy name or ID we use that policy to enrol the agent/fleet server
• if the field is not set we call the /api/fleet/agent_polices/ endpoint in Kibana which produces output similar to this:

{
  "items": [
    {
      "id": "eck-agent",
      "namespace": "default",
      "monitoring_enabled": [
        "logs",
        "metrics"
      ],
      "is_default": true,
      "name": "Elastic Agent on ECK policy",
      "unenroll_timeout": 900,
      "is_preconfigured": true,
      "status": "active",
      "is_managed": false,
      "revision": 3,
      "updated_at": "2022-06-22T14:39:09.683Z",
      "updated_by": "system",
      "package_policies": [
        "b58bd4d8-3fa4-54a1-8776-a733732c8a3d",
        "a3b718bd-efec-54f4-b513-7711c744a8ec"
      ],
      "agents": 0
    },
    {
      "id": "eck-fleet-server",
      "namespace": "default",
      "monitoring_enabled": [
        "logs",
        "metrics"
      ],
      "is_default_fleet_server": true,
      "name": "Fleet Server on ECK policy",
      "is_preconfigured": true,
      "status": "active",
      "is_managed": false,
      "revision": 2,
      "updated_at": "2022-06-22T14:39:03.604Z",
      "updated_by": "system",
      "package_policies": [
        "fleet_server-1"
      ],
      "agents": 0
    }
  ],
  "total": 2,
  "page": 1,
  "perPage": 20
}

• we could then create an enrolment token with the corresponding policy ID and configure Agent with the token.

Questions

• What is the earliest version we can support with this approach?
  The first version to explicitly support the is_default* attribute is 8.1
  The first version ECK supports with fleet is 7.14. Even though the is_default* attributes where not part of the documented model back then it seems that they are still available on the pre-configured policies as a test API call showed. So we should be good