Supporting extra config files for k8s components

[frezbo]

Feature Request

Kubelet, API server, Kube scheduler etc require extra config files to be passed in as arguments for advanced tweaking. Currently Talos lacks an easy way to configure them. See:

• AWS ECR Image Credential Provider extensions#241
• Support for KubeSchedulerConfiguration #7905

Proposal

Support something in the order of extraConfigFiles for the components. Eg:

machine:
  kubelet:
    extraConfigFiles:
      - name: credentialproviderconfig.yaml
        value:
          apiVersion: kubelet.config.k8s.io/v1
          kind: CredentialProviderConfig

The value can be any valid yaml and Talos will not validate the schema.

Talos can write down these files to /etc/kubernetes/<component-name>.d/ directory and fully managed by the configuration in the machine config.

These folders would be mounted by default by Talos to the corresponding components. The user would still have to pass in extraArgs/extraConfig to use these files.

[smira]

I think we should avoid /etc/kubernetes as it is backed by /var via overlay mount... much better place will be under /system with other config files we already generate. We can still mount it into the container as /etc/kubernetes/<something>

[frezbo]

I think we should avoid /etc/kubernetes as it is backed by /var via overlay mount... much better place will be under /system with other config files we already generate. We can still mount it into the container as /etc/kubernetes/<something>

kubelet.conf is already under /var right? and since we deterministically generate those, shouldn't it be okay?

[smira]

I don't like the fact that we leave some files in /var for no reason. kubelet.conf is there for historical reasons, plus kubelet keeps its own state there. But we shouldn't put new stuff there if there's no reason to do so.

We don't need these files to persist reboots, we don't even need /var for them at all, so why put them there, if we could better handle them in a tmpfs location we own completely?

[frezbo]

We don't need these files to persist reboots, we don't even need /var for them at all, so why put them there, if we could better handle them in a tmpfs location we own completely?

that makes sense 👍