diff --git a/keps/prod-readiness/sig-auth/2718.yaml b/keps/prod-readiness/sig-auth/2718.yaml new file mode 100644 index 000000000000..fd56a9f978b3 --- /dev/null +++ b/keps/prod-readiness/sig-auth/2718.yaml @@ -0,0 +1,3 @@ +kep-number: 2718 +alpha: + approver: "@johnbelamaric" diff --git a/keps/sig-auth/2718-20210511-client-exec-proxy/README.md b/keps/sig-auth/2718-20210511-client-exec-proxy/README.md new file mode 100644 index 000000000000..12927b9866b1 --- /dev/null +++ b/keps/sig-auth/2718-20210511-client-exec-proxy/README.md @@ -0,0 +1,853 @@ + +# KEP-2718: Client Executable Proxy + + +- [Release Signoff Checklist](#release-signoff-checklist) +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Proposal](#proposal) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [Proxy Proposal](#proxy-proposal) + - [Authentication to Proxy](#authentication-to-proxy) + - [Caching](#caching) + - [Proxy Shutdown](#proxy-shutdown) + - [API](#api) + - [Test Plan](#test-plan) + - [Integration](#integration) + - [E2E](#e2e) + - [Graduation Criteria](#graduation-criteria) + - [Alpha](#alpha) + - [Alpha -> Beta Graduation](#alpha---beta-graduation) + - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy) + - [Version Skew Strategy](#version-skew-strategy) +- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire) + - [Feature Enablement and Rollback](#feature-enablement-and-rollback) + - [Rollout, Upgrade and Rollback Planning](#rollout-upgrade-and-rollback-planning) + - [Monitoring Requirements](#monitoring-requirements) + - [Dependencies](#dependencies) + - [Scalability](#scalability) + - [Troubleshooting](#troubleshooting) +- [Implementation History](#implementation-history) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) + - [Alternative Proposal: Request Replacement](#alternative-proposal-request-replacement) + + +## Release Signoff Checklist + + + +Items marked with (R) are required *prior to targeting to a milestone / release*. + +- [ ] (R) Enhancement issue in release milestone, which links to KEP dir in [kubernetes/enhancements] (not the initial KEP PR) +- [ ] (R) KEP approvers have approved the KEP status as `implementable` +- [ ] (R) Design details are appropriately documented +- [ ] (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input +- [ ] (R) Graduation criteria is in place +- [ ] (R) Production readiness review completed +- [ ] Production readiness review approved +- [ ] "Implementation History" section is up-to-date for milestone +- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io] +- [ ] Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes + + + +[kubernetes.io]: https://kubernetes.io/ +[kubernetes/enhancements]: https://git.k8s.io/enhancements +[kubernetes/kubernetes]: https://git.k8s.io/kubernetes +[kubernetes/website]: https://git.k8s.io/website + +## Summary + + + +Client authentication in Kubernetes can be achieved via one of a handful of +mechanisms: by passing a bearer token directly to kubectl using the `--token` +flag, by inserting a bearer token into the kubeconfig, by inserting a client +certificate and key into the kubeconfig, or by defining an exec-based token +plugin in the kubeconfig which supports calling an arbitrary binary which is +expected to return a bearer token and/or client certificate/key. The client +exec proxy adds an extension point for additional authentication mechanisms +that are difficult to set up with the current architecture. Client request +signing, and an external TLS certificate authenticator are two use cases that +would benefit from this proposal. + +## Motivation + + + +The motivation for this KEP stems from the inability of the existing +authentication mechanisms to satisfy the author's requirement to implement a +Kubernetes client that implements AWS's request signing process, known as +Signature Version 4 (sig-v4) +[[1]](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html) +[[2]](https://github.com/kubernetes/kubernetes/issues/92535). In order to +implement sig-v4, a canonical version of the request must be created, combined +with any required additional metadata, signed by the secret key, and the +signature appended to the request in a header or query string parameter. In +order to achieve this, the request signer must have access to the entire +request, so the existing exec-based authentication is unsatisfactory. A +generic solution is desirable to satisfy other future request signing or +request modifying feature requests. + +The solution that exists today is to use the existing proxy configuration in +client-go. The includes configuring a proxy via the environment variables, +HTTP_PROXY or HTTPS_PROXY, or using the explicit proxy URL supported by +kubeconfig [[3]](https://github.com/kubernetes/kubernetes/pull/81443). The +problem with this solution is that the operator experience of setting up and +managing an external proxy is poor. For example, to implement a request +signing proxy for kubelet, one has to manage the proxy process with an init +system of some kind, and secure the proxy endpoint, both of which require +significant, non-trivial configuration outside of Kubernetes. + +Another use case is using an external TLS certificate signer +[[4]](https://github.com/kubernetes/enhancements/pull/1749). This would allow +usage of TLS client certificates within client-go, by delegating digital key +operations to external processes, for example, an HSM. + +### Goals + + + +The goals of this KEP include: +1. Defining a specification which enables Kubernetes clients to do arbitrary +request modification before sending the request to the API server, via an +extensible mechanism to support the following use cases: + - Client request signing. + - Using hardware protected keys for request signing or mTLS. + - Adding or manipulating arbitrary headers to the request. +2. Defining a specification which would provide the ability for the client-side +proxy server to be executed and communicated with according to configuration +solely contained in a kubeconfig file. +3. Defining a specification which is interoperable with an existing client-side +proxy setup, e.g. a proxy configured with the HTTP_PROXY environment variable. +4. Defining a specification for a client-side proxy server that is easy to +secure without taking additional setup steps, by: + - Using a unix domain socket for the communication between the client and + the proxy, secured with file permissions and TLS. + +### Non-Goals + + + +The following are non-goals of this KEP: +1. Solving use cases outside of the above-mentioned client authentication + workflows. +2. Implementing a specific flavor of request signing (like SigV4), or adding + support for any hardware security module protocol directly in the Kubernetes + client libraries. +3. Defining any new mechanisms for validating request signatures server-side. + This is because they can be validated by a front-proxy, which is already a + feature of Kubernetes. + +## Proposal + + + +This proposal is to implement an extension in Kubernetes client libraries to +forward requests to a proxy process for request modification. The proxy will +receive an HTTP request and will have the opportunity to sign the request, +attach headers, use hardware protected keys for mTLS, and/or modify the +request. The final request will be constructed and forwarded to the apiserver. + +It is already possible, by setting the appropriate HTTP_PROXY, HTTPS_PROXY +environment variables, to configure a Kubernetes client to send requests to a +proxy. While this may work in some situations, it is not ideal for a generic +request signing solution across all client configurations. Firstly, http +proxies are already commonly integrated in client networking configurations, so +for these clients setting up an additional proxy configuration might not be +possible, or at least not straightforward. Secondly, this approach requires a +long running external proxy process which introduces operational complexity. +Thirdly, communicating with a proxy for request signing requires a secure +communication channel between the client and the proxy, which means in a case +where multiple users share a client computer (for example, a shared bastion), +configuring and securing a local proxy to prevent signing requests as a +different user becomes more complicated. + +### Risks and Mitigations + + + +The risks of this proposal are similar to the existing exec credential +provider. There are risks associated with execing a binary, for example in a +scenario where users are downloading binaries from untrustworthy sources to use +in their cluster. However, this is not a departure from existing security +posture of the exec credential feature present in k8s.io/client-go today. + +## Design Details + + + +### Proxy Proposal + +The proxy exec plugin is a long-running proxy process which is executed by the +client on first use. When executed, the exec proxy plugin will bind and listen +on a socket, and will a output URL on stdout, over which the client would proxy +all API server bound requests. The proxy binary location is defined in the +`ExecConfig` section of a kubeconfig `user`. + +The proxy plugin process will be executed from client-go the first time it is +needed, on transport setup. If the client fails to connect to the socket +because no one is listening, client-go will reattempt to exec the plugin. +Client-go will support both plugins binaries which are themselves the proxy +process, and additionally plugin binaries which complete, but are responsible +for starting a proxy process and returning a URL to the proxy. + +The client will send its request to the proxy plugin process, and the proxy +will forward the request to the API server, rather than returning the modified +request to the client to be sent (see alternatives considered). The downside +of this approach is that the proxy must duplicate the some of the transport +functionality of client packages, like respecting existing proxy configuration, +but allows for more flexibility for proxy plugin developers. + +### Authentication to Proxy + +Options: + +1. The client will authenticate to its proxy using mTLS. On exec, the proxy + will pass back client key and certificate authority data for the client to + use for authentication when establishing a connection. +2. An alternative method for authentication to the proxy would be file + permission based. The GID owner of the client process could passed to the + proxy over stdin, which the proxy should give read/write permissions on the + socket to. However, this could be less portable if we need to use a non UDS + communication mechanism for windows, or if we support TCP/IP in the future. +3. A third mechanism for authentication for windows based systems would be + using named pipes, which support the Win32 access control model. + +An option will be chosen during implemenation of alpha. + +### Caching + +The connection from the client to the proxy can be cached in the client based +on the cluster details, the existing HTTP_PROXY configuration, the exec-proxy +URL, and the certificates used for mTLS. + +### Proxy Shutdown + +In the case where the client dies unexpectedly, due to a crash or SIGKILL, the +proxy should also die. To handle this, the proxy will attempt to take a file +lock that the parent holds and quit if it ever successfully takes the lock. +Before exec, the client should take the lock. The lock file path will be +provided to the proxy in the ExecCredentialSpec. + +Alterntatively, the proxy could quit after a certain amount of idle time. For +example, proxies that are idle for more than 5 minutes will be required to +quit. Idle quit is simpler, but the lock approach handles the kubectl use case +better, so it is preferred. + +### API + +The communication over stdin and stdout with the exec'd binary will be done +with using the existing ExecCredentialSpec and ExecCredentialStatus objects. + +```go +// ExecCredentialSpec holds request and runtime specific information provided by +// the transport. +type ExecCredentialSpec struct { + // Cluster contains information to allow an exec plugin to communicate with the + // kubernetes cluster being authenticated to. Note that Cluster is non-nil only + // when provideClusterInfo is set to true in the exec provider config (i.e., + // ExecConfig.ProvideClusterInfo). + // +optional + Cluster *Cluster `json:"cluster,omitempty"` + + // Interactive declares whether stdin has been passed to this exec plugin. + Interactive bool `json:"interactive"` + + // New: + + // LockFilePath specifies a lock file to be taken by the client + // +optional + LockFilePath string `json:"lockFilePath,omitempty"` +} + +// ExecCredentialStatus holds credentials for the transport to use. +// +// Token and ClientKeyData are sensitive fields. This data should only be +// transmitted in-memory between client and exec plugin process. The Exec plugin +// itself should at least be protected via file permissions. +type ExecCredentialStatus struct { + // ExpirationTimestamp indicates a time when the provided credentials expire. + // +optional + ExpirationTimestamp *metav1.Time `json:"expirationTimestamp,omitempty"` + // Token is a bearer token used by the client for request authentication. + Token string `json:"token,omitempty" datapolicy:"token"` + // PEM-encoded client TLS certificates (including intermediates, if any). + ClientCertificateData string `json:"clientCertificateData,omitempty"` + // PEM-encoded private key for the above certificate. + ClientKeyData string `json:"clientKeyData,omitempty" datapolicy:"security-key"` + + // New: + + // ProxyConfig is the config returned by the exec-proxy + // +optional + ProxyConfig *ProxyConfig `json:"proxyConfig,omitempty"` +} + + +// ProxyConfig configures the client to proxy requests via the returned address. +type ProxyConfig struct { + // URL is the location of the exec proxy listener. + URL string `json: "url"` + + CertificateAuthorityData []byte `json:"certificate-authority-data"` +} +``` + +### Test Plan + + + +Both integration and e2e tests will be required in addition to unit tests. + +#### Integration + +Integration tests will cover: + +* Authentication between client and proxy. +* Connection from client to apiserver through proxy. +* Proxy use with kubectl (possible test location: [command-line integration + test + suite](https://github.com/kubernetes/kubernetes/tree/ec560b9737537be8c688776461bc700e8ddedb9d/test/cmd)). +* UDS connections from the client to the proxy will be tested. + +#### E2E + +The e2e tests will test communication from client to apiserver through proxy in +a real cluster. The e2e tests will require a proxy implementation and +apiserver, etcd, and a client. + +### Graduation Criteria + + + +#### Alpha + +- Implementation of UDS connections to proxy. + +#### Alpha -> Beta Graduation + +- Implementation of UDS (or equivalent) on windows. +- Full reference proxy implementation with TLS offload. This will be built using code from client-go that provides a harness to offload to a Go `crypto.Signer`. +- As part of client-go, it needs to work on macOS, windows and linux. +- Tests that show SPDY based APIs like pods/exec still work. +- Tests that exercise webhooks from API server because they share the same auth code. +- Gather feedback from developers, SIG Auth, SIG CLI. +- Tests are in Testgrid and linked in KEP. + +### Upgrade / Downgrade Strategy + + + +For clients which are not currently using an exec plugin, a modification to the +kubeconfig must be made and a proxy binary must be located as specified in the +kubeconfig. For clients already using an exec plugin without proxy support, +the exec plugin would need to upgraded. For clients moving from one version of +the proxy to another, the proxy binary must be upgraded. + +### Version Skew Strategy + + + +The proxy binary may be required to be updated before the client is updated in +situations where the API over stdio between them changes. + +## Production Readiness Review Questionnaire + + + +### Feature Enablement and Rollback + +_This section must be completed when targeting alpha to a release._ + +* **How can this feature be enabled / disabled in a live cluster?** + - [x] Feature gate (also fill in values in `kep.yaml`) + - Feature gate name: ClientExecProxy + - Components depending on the feature gate: kubelet, + kube-controller-manager, kube-scheduler (for any that need to use the + proxy--this will be environment dependent). + - For kubectl, an env var will be used to enable when its an alpha feature. + - [ ] Other + - Describe the mechanism: + - Will enabling / disabling the feature require downtime of the control + plane? + - Will enabling / disabling the feature require downtime or reprovisioning + of a node? (Do not assume `Dynamic Kubelet Config` feature is enabled). + +* **Does enabling the feature change any default behavior?** + Any change of default behavior may be surprising to users or break existing + automations, so be extremely careful here. + + No. + +* **Can the feature be disabled once it has been enabled (i.e. can we roll back + the enablement)?** + Also set `disable-supported` to `true` or `false` in `kep.yaml`. Describe + the consequences on existing workloads (e.g., if this is a runtime feature, + can it break the existing applications?). + + Yes, the feature can be disabled. In the case where it is being used by long + running clients to talk to the API server, the client would have to be + reconfigured to stop using the proxy for connections. Depending on how the + client is configured, this may require a restart. + +* **What happens if we reenable the feature if it was previously rolled back?** + + The feature can be reenabled after being disabled without consequence. + +* **Are there any tests for feature enablement/disablement?** + The e2e framework does not currently support enabling or disabling feature + gates. However, unit tests in each component dealing with managing data, created + with and without the feature, are necessary. At the very least, think about + conversion tests if API types are being modified. + + No. + +### Rollout, Upgrade and Rollback Planning + +_This section must be completed when targeting beta graduation to a release._ + +* **How can a rollout fail? Can it impact already running workloads?** + Try to be as paranoid as possible - e.g., what if some components will restart + mid-rollout? + +* **What specific metrics should inform a rollback?** + +* **Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?** + Describe manual testing that was done and the outcomes. + Longer term, we may want to require automated upgrade/rollback tests, but we + are missing a bunch of machinery and tooling and can't do that now. + +* **Is the rollout accompanied by any deprecations and/or removals of features, APIs, +fields of API types, flags, etc.?** + Even if applying deprecation policies, they may still surprise some users. + +### Monitoring Requirements + +_This section must be completed when targeting beta graduation to a release._ + +* **How can an operator determine if the feature is in use by workloads?** + Ideally, this should be a metric. Operations against the Kubernetes API (e.g., + checking if there are objects with field X set) may be a last resort. Avoid + logs or events for this purpose. + +* **What are the SLIs (Service Level Indicators) an operator can use to determine +the health of the service?** + - [ ] Metrics + - Metric name: `rest_client_exec_plugin_with_proxy_call_total` + - Components exposing the metric: client-go (kube-controller-manager, kube-scheduler, kubelet, kube-apiserver) + - [ ] Other (treat as last resort) + - Details: + +* **What are the reasonable SLOs (Service Level Objectives) for the above SLIs?** + At a high level, this usually will be in the form of "high percentile of SLI + per day <= X". It's impossible to provide comprehensive guidance, but at the very + high level (needs more precise definitions) those may be things like: + - per-day percentage of API calls finishing with 5XX errors <= 1% + - 99% percentile over day of absolute value from (job creation time minus expected + job creation time) for cron job <= 10% + - 99,9% of /health requests per day finish with 200 code + +* **Are there any missing metrics that would be useful to have to improve observability +of this feature?** + Describe the metrics themselves and the reasons why they weren't added (e.g., cost, + implementation difficulties, etc.). + +### Dependencies + +_This section must be completed when targeting beta graduation to a release._ + +* **Does this feature depend on any specific services running in the cluster?** + Think about both cluster-level services (e.g. metrics-server) as well + as node-level agents (e.g. specific version of CRI). Focus on external or + optional services that are needed. For example, if this feature depends on + a cloud provider API, or upon an external software-defined storage or network + control plane. + + For each of these, fill in the following—thinking about running existing user workloads + and creating new ones, as well as about cluster-level services (e.g. DNS): + - [Dependency name] + - Usage description: + - Impact of its outage on the feature: + - Impact of its degraded performance or high-error rates on the feature: + + +### Scalability + +_For alpha, this section is encouraged: reviewers should consider these questions +and attempt to answer them._ + +_For beta, this section is required: reviewers must answer these questions._ + +_For GA, this section is required: approvers should be able to confirm the +previous answers based on experience in the field._ + +* **Will enabling / using this feature result in any new API calls?** + Describe them, providing: + - API call type (e.g. PATCH pods) + - estimated throughput + - originating component(s) (e.g. Kubelet, Feature-X-controller) + focusing mostly on: + - components listing and/or watching resources they didn't before + - API calls that may be triggered by changes of some Kubernetes resources + (e.g. update of object X triggers new updates of object Y) + - periodic API calls to reconcile state (e.g. periodic fetching state, + heartbeats, leader election, etc.) + +* **Will enabling / using this feature result in introducing new API types?** + Describe them, providing: + - API type + - Supported number of objects per cluster + - Supported number of objects per namespace (for namespace-scoped objects) + +* **Will enabling / using this feature result in any new calls to the cloud +provider?** + +* **Will enabling / using this feature result in increasing size or count of +the existing API objects?** + Describe them, providing: + - API type(s): + - Estimated increase in size: (e.g., new annotation of size 32B) + - Estimated amount of new objects: (e.g., new Object X for every existing Pod) + +* **Will enabling / using this feature result in increasing time taken by any +operations covered by [existing SLIs/SLOs]?** + Think about adding additional work or introducing new steps in between + (e.g. need to do X to start a container), etc. Please describe the details. + +* **Will enabling / using this feature result in non-negligible increase of +resource usage (CPU, RAM, disk, IO, ...) in any components?** + Things to keep in mind include: additional in-memory state, additional + non-trivial computations, excessive access to disks (including increased log + volume), significant amount of data sent and/or received over network, etc. + This through this both in small and large cases, again with respect to the + [supported limits]. + +### Troubleshooting + +The Troubleshooting section currently serves the `Playbook` role. We may consider +splitting it into a dedicated `Playbook` document (potentially with some monitoring +details). For now, we leave it here. + +_This section must be completed when targeting beta graduation to a release._ + +* **How does this feature react if the API server and/or etcd is unavailable?** + +The proxy's behavior when the API server is unavailable will be up to the proxy implementor. + +* **What are other known failure modes?** + For each of them, fill in the following information by copying the below template: + - [Failure mode brief description] + - Detection: How can it be detected via metrics? Stated another way: + how can an operator troubleshoot without logging into a master or worker node? + - Mitigations: What can be done to stop the bleeding, especially for already + running user workloads? + - Diagnostics: What are the useful log messages and their required logging + levels that could help debug the issue? + Not required until feature graduated to beta. + - Testing: Are there any tests for failure mode? If not, describe why. + +* **What steps should be taken if SLOs are not being met to determine the problem?** + +[supported limits]: https://git.k8s.io/community//sig-scalability/configs-and-limits/thresholds.md +[existing SLIs/SLOs]: https://git.k8s.io/community/sig-scalability/slos/slos.md#kubernetes-slisslos + +## Implementation History + + + +- 2021-05-07 KEP PR created, first reviews received. +- 2022-01-27 PR updated, more feedback received. +- 2022-06-22 PR updated to use ExecCredentialSpec/ExecCredentialStatus. + +## Drawbacks + + + +It introduces a new feature into client libraries which comes with a +maintenance burden. It also introduces a new extension point which might make +client installs which choose to use this feature slightly more complicated, in +that the user must install an additional binary. + +## Alternatives + + + +### Alternative Proposal: Request Replacement + +In this proposal, the the proxy would be replaced by a localhost/UDS server. +The client would package the HTTP request into the body of a GRPC or HTTP +request and send to the server, expecting a response back immediately with the +modified request as a payload. The client would then extract the request, +replace the original request with the modified request, and send it as it +normally would. This would be done in the client transport. Otherwise, the +proposal is largely the same--the client uses configuration in the kubeconfig +to start the signing process, and the client receives a unix domain socket or +localhost address from the server process on exec. The server process listens +at the unix domain socket address, and the client connects to it. The client +attempts to exec the server binary if it is unable to connect using the +provided address. + +This proposal was not preferred to the chosen proposal because, after +discussions at SIG-Auth, the overall feeling was that defining the request +response protocol between the client and the signing process would be more +cumbersome, and be less suited to a generic solution that accommodated any +proxy related use case. It is also not possible to implement TLS offload with +this approach. + diff --git a/keps/sig-auth/2718-20210511-client-exec-proxy/kep.yaml b/keps/sig-auth/2718-20210511-client-exec-proxy/kep.yaml new file mode 100644 index 000000000000..d48a564c84a2 --- /dev/null +++ b/keps/sig-auth/2718-20210511-client-exec-proxy/kep.yaml @@ -0,0 +1,52 @@ +title: Client Executable Proxy +kep-number: 2718 +authors: + - "@nckturner" +owning-sig: sig-auth +participating-sigs: + - sig-cli + - sig-api-machinery +status: provisional +creation-date: 2021-05-11 +reviewers: + - "@mikedanese" + - "@enj" + - "@liggitt" +approvers: + - "@mikedanese" + - "@enj" + - "@liggitt" + +see-also: + - https://github.com/kubernetes/enhancements/issues/541 +replaces: + - https://github.com/kubernetes/enhancements/pull/1749 + +# The target maturity stage in the current dev cycle for this KEP. +stage: alpha + +# The most recent milestone for which work toward delivery of this KEP has been +# done. This can be the current (upcoming) milestone, if it is being actively +# worked on. +latest-milestone: "v1.26" + +# The milestone at which this feature was, or is targeted to be, at each stage. +milestone: + alpha: "v1.26" + beta: "v1.27" + stable: "v1.28" + +# The following PRR answers are required at alpha release +# List the feature gate name and the components for which it must be enabled +feature-gates: + - name: ClientExecProxy + components: + - kubelet + - kube-controller-manager + - kube-scheduler +disable-supported: true + +# The following PRR answers are required at beta release +metrics: + # Represents failures returned from executing the binary + - client_exec_proxy_errors