[RFC] Flux Multi-Tenancy Mode

[stefanprodan]

For multi-tenant environments, we want to offer an easy way of configuring Flux to enforce tenant isolation.

This RFC describes how that can be achieved with a single flag added to flux bootstrap.

Supersedes #582

[scottrigby]

Note to document this in fluxcd/website#598

[jonathan-innis]

Can we clarify the exact permissions that we are giving to the Flux controllers in this mode? I'm assuming GET/LIST access to .fluxcd.io CRDS, secrets, etc. across all namespaces?

[stefanprodan]

The controller cluster role is defined here: https://github.com/fluxcd/flux2/blob/main/manifests/rbac/controller.yaml

Should I paste this in RFC?

[jonathan-innis]

Yes, this would be good to add into the RFC for everyone's visibility

[stefanprodan]

I'm assuming that people read Flux docs not just this RFC. The RBAC and impersonation is covered here https://fluxcd.io/docs/security/#controller-permissions.

[jonathan-innis]

With the implementation of this security proposal, we can drop the cluster-reconciler in this case for the kustomize-controller and helm-controller, though. I'm thinking this would be good to call out vs. the status-quo where they are given cluster-admin as default

[stefanprodan]

We can’t drop the cluster role binding to cluster-admin, admins can switch between single-tenant and multi-tenant at any time. The controllers use the cluster-admin to impersonate the tenant account.

From the docs I linked above:

However in a soft multi-tenancy setup, Flux does not reconcile a tenant’s repo under the cluster-admin role. Instead you specify a different service account in your manifest, and the Flux controllers will use the Kubernetes Impersonation API under cluster-admin to impersonate that service account.

[pjbgf]

In this mode, cluster-reconciler could bind to a ClusterRole that only provides the impersonation powers it requires. (i.e. allowing a Platform administrator to further limit the ClusterRoleBinding to specific accounts it can run as)

[jonathan-innis]

+1 for this being configurable

[kubasobon]

Likewise. This provides a nice way to adopt ServiceAccounts that are already set up.

[jonathan-innis]

With this proposed method, the kustomize-controller or helm-controller can reconcile any service account within the namespace of the CR. With this in mind, a user needs to be hyper-aware that any service account in that namespace could be used for reconciliation and must not role-bind to give any service account any kind of elevated privilege that it did not intend for the flux reconcilers (such as giving access to create HelmReleases across the cluster, which essentially opens up Pandora's box). Will this be noted somewhere specifically/is there a way to ensure that flux can only reconcile with service accounts with .metadata.labels intended specifically for flux?

[stefanprodan]

I think we shouldn't enforce any limitations for the service accounts that are provisioned by platform admins because they may chose to allow a tenant to own multiple namespaces, so they can bind a SA to all those namespaces. A tenant can't create cluster role bindings so it's not possible for them to evade their boundary.

[hiddeco]

Wouldn't this same concern apply to e.g. a Deployment which accepts any arbitrary service account as a configuration value? Which I assume, isn't a problem. Raising the question why it would be different for a Flux Custom Resource?

[squaremo]

Wouldn't this same concern apply to e.g. a Deployment which accepts any arbitrary service account as a configuration value?

These are the same technically -- a Deployment can run a container that uses kubectl to cause mischief. But the opportunity for mistakes is higher when Flux is introduced. With only Deployments and other "normal" objects you would not usually consider creating a service account with wide-ranging permissions. When you're using Flux, you have much more (RBAC) to think about.

[stefanprodan]

With only Deployments and other "normal" objects you would not usually consider creating a service account with wide-ranging permissions.

Depend on what you consider to be "normal" here, every single Kubernetes addon/controller needs wide-ranging permissions, are these abnormal?

[hiddeco]

I think my point is more that if you make extensive use of RBAC, you should be aware of what this entails (for your tenants).

The recommendation of not sharing a service account between multiple resources, or using any arbitrary account that is available to you, does not change for a Deployment or a Flux resource.

[hiddeco]

Not saying here that this means we should not highlight or be upfront about it, but rather that I am not really enthusiastic about adding e.g. label selectors for service accounts.

[squaremo]

I am not really enthusiastic about adding e.g. label selectors for service accounts.

Right -- requiring labels is not a protection, since any process that can run with the service account can do mischief, not just the Flux controllers.

[pjbgf]

The main challenge on the model in which the tenant's service account is created in the tenant's namespace is that anyone with pod create permissions in the tenant's namespace would be able to privesc to whatever permissions the flux tenant service account is running as - current example is cluster-admin within that namespace.

In this model platform admins may have to lean on admission controllers to block that path to privilege escalation.

[pjbgf]

I added some recommendations on how the tenant can be structured to protect their service accounts in the RFC004 (xref: #2086 (comment)) without requiring admission controllers.

[jonathan-innis]

+1 from Microsoft 🚀

[pjbgf]

This could also be beneficial beyond the multi-tenancy use-case, allowing Platform admins to run flux in a least-privileged mode, enforcing the permissions the controllers operate under - like in the user story 1. Instead of multi-tenants, a cluster could simply have different trust boundaries that require specific permission levels.

The change would impact most terminology, as the scope would be the same. Meaning, instead of:

flux bootstrap --security-profile=multi-tenant

We could have something more multi-tenancy-agnostic (or any other relevant naming):

flux bootstrap --security-profile=least-privileged

[pjbgf]

In this mode, cluster-reconciler could bind to a ClusterRole that only provides the impersonation powers it requires. (i.e. allowing a Platform administrator to further limit the ClusterRoleBinding to specific accounts it can run as)

[pjbgf]

The main challenge on the model in which the tenant's service account is created in the tenant's namespace is that anyone with pod create permissions in the tenant's namespace would be able to privesc to whatever permissions the flux tenant service account is running as - current example is cluster-admin within that namespace.

In this model platform admins may have to lean on admission controllers to block that path to privilege escalation.

[schrej]

One policy to get rid of 😄
+1 from DTAG

[stefanprodan]

Closing this with the same resolution resolution as #2092