Merge pull request #1276 from bowei/pod-resolv-conf

Automatic merge from submit-queue.

Add proposal for configurable pod resolv.conf

cc: @thockin @MrHohn

contributors/design-proposals/network/pod-resolv-conf.md

@@ -0,0 +1,210 @@
+# Custom /etc/resolv.conf
+
+* Status: pending
+* Version: alpha
+* Implementation owner: Bowei Du <[bowei@google.com](mailto:bowei@google.com)>,
+  Zihong Zheng <[zihongz@google.com](mailto:zihongz@google.com)>
+
+# Overview
+
+The `/etc/resolv.conf` in a pod is managed by Kubelet and its contents are
+generated based on `pod.dnsPolicy`. For `dnsPolicy: Default`, the `search` and
+`nameserver` fields are taken from the `resolve.conf` on the node where the pod
+is running. If the `dnsPolicy` is `ClusterFirst`, the search contents of the
+resolv.conf is the hosts `resolv.conf` augmented with the following options:
+
+*   Search paths to add aliases for domain names in the same namespace and
+    cluster suffix.
+*   `options ndots` to 5 to ensure the search paths are searched for all
+    potential matches.
+
+The configuration of both search paths and `ndots` results in query
+amplification of five to ten times for non-cluster internal names. This is due
+to the fact that each of the search path expansions must be tried before the
+actual result is found. This order of magnitude increase of query rate imposes a
+large load on the kube-dns service. At the same time, there are user
+applications do not need the convenience of the name aliases and do not wish to
+pay this performance cost.
+
+
+## Existing workarounds
+
+The current work around for this problem is to specify an FQDN for name
+resolution. Any domain name that ends with a period (e.g. `foo.bar.com.`) will
+not be search path expanded. However, use of FQDNs is not well-known practice
+and imposes application-level changes. Cluster operators may not have the luxury
+of enforcing such a change to applications that run on their infrastructure.
+
+It is also possible for the user to insert a short shell script snippet that
+rewrites `resolv.conf` on container start-up. This has the same problems as the
+previous approach and is also awkward for the user. This also forces the
+container to have additional executable code such as a shell or scripting engine
+which increases the applications security surface area.
+
+
+# Proposal sketch
+
+This proposal gives users a way to overlay tweaks into the existing
+`DnsPolicy`. A new PodSpec field `dnsParams` will contains fields that are
+merged with the settings currently selected with `DnsPolicy`.
+
+The fields of `DnsParams` are:
+
+* `nameservers` is a list of additional nameservers to use for resolution. On
+  `resolv.conf` platforms, these are entries to `nameserver`.
+* `search` is a list of additional search path subdomains. On `resolv.conf`
+  platforms, these are entries to the `search` setting. These domains will be
+  appended to the existing search path.
+* `options` that are an OS-dependent list of (name, value) options. These values
+  are NOT expected to be generally portable across platforms. For containers that
+  use `/etc/resolv.conf` style configuration, these correspond to the parameters
+  passed to the `option` lines. Options will override if their names coincide,
+  i.e, if the `DnsPolicy` sets `ndots:5` and `ndots:1` appears in the `Spec`,
+  then the final value will be `ndots:1`.
+
+For users that want to completely customize their resolution configuration, we
+add a new `DnsPolicy: Custom` that does not define any settings. This is
+essentially an empty `resolv.conf` with no fields defined.
+
+## Pod API examples
+
+### Host `/etc/resolv.conf`
+
+Assume in the examples below that the host has the following `/etc/resolv.conf`:
+
+```bash
+nameserver 10.1.1.10
+search foo.com
+options ndots:1
+```
+
+### Override DNS server and search paths
+
+In the example below, the user wishes to use their own DNS resolver and add the
+pod namespace and a custom expansion to the search path, as they do not use the
+other name aliases:
+
+```yaml
+# Pod spec
+apiVersion: v1
+kind: Pod
+metadata: {"namespace": "ns1", "name": "example"}
+spec:
+  ...
+  dnsPolicy: Custom
+  dnsParams:
+    nameservers: ["1.2.3.4"]
+    search:
+    - ns1.svc.cluster.local
+    - my.dns.search.suffix
+    options:
+    - name: ndots
+      value: 2
+    - name: edns0
+```
+
+The pod will get the following `/etc/resolv.conf`:
+
+```bash
+nameserver 1.2.3.4
+search ns1.svc.cluster.local my.dns.search.suffix
+options ndots:2 edns0
+```
+
+## Overriding `ndots`
+
+Override `ndots:5` in `ClusterFirst` with `ndots:1`. This keeps all of the
+settings intact:
+
+```yaml
+dnsPolicy: ClusterFirst
+dnsParams:
+- options:
+  - name: ndots
+  - value: 1
+```
+
+Resulting `resolv.conf`:
+
+```bash
+nameserver 10.0.0.10
+search default.svc.cluster.local svc.cluster.local cluster.local foo.com
+options ndots:1
+```
+
+# API changes
+
+```go
+type PodSpec struct {
+    ...
+    DNSPolicy string        `json:"dnsPolicy,omitempty"`
+    DNSParams *PodDNSParams `json:"dnsParams,omitempty"`
+    ...
+}
+
+type PodDNSParams struct {
+    Nameservers []string             `json:"nameservers,omitempty"`
+    Search      []string             `json:"search,omitempty"`
+    Options     []PodDNSParamsOption `json:"options,omitempty" patchStrategy:"merge" patchMergeKey:"name"`
+}
+
+type PodDNSParamsOption struct {
+    Name   string  `json:"name"`
+    Value *string `json:"value,omitempty"`
+}
+```
+
+## Semantics
+
+Let the following be the Go representation of the `resolv.conf`:
+
+```go
+type ResolvConf struct {
+  Nameserver []string // "nameserver" entries
+  Search     []string // "search" entries
+  Options    []PodDNSParamsOption  // "options" entries
+}
+```
+
+Let `var HostResolvConf ResolvConf` be the host `resolv.conf`.
+
+Then the final Pod `resolv.conf` will be:
+
+```go
+func podResolvConf() ResolvConf {
+    var podResolv ResolvConf
+
+    switch (pod.DNSPolicy) {
+    case "Default":
+        podResolv = HostResolvConf
+    case "ClusterFirst:
+        podResolv.Nameservers = []string{ KubeDNSClusterIP }
+        podResolv.Search = ... // populate with ns.svc.suffix, svc.suffix, suffix, host entries...
+        podResolv.Options = []PodDNSParamsOption{{"ndots","5" }}
+    case "Custom": // start with empty `resolv.conf`
+        break
+    }
+
+    // Append the additional nameservers.
+    podResolv.Nameservers = append(Nameservers, pod.DNSParams.Nameservers...)
+    // Append the additional search paths.
+    podResolv.Search = append(Search, pod.DNSParams.Search...)
+    // Merge the DnsParams.Options with the options derived from the given DNSPolicy.
+    podResolv.Options = mergeOptions(pod.Options, pod.DNSParams.Options)
+
+    return podResolv
+}
+```
+
+### Invalid configurations
+
+The follow configurations will result in an invalid Pod spec:
+
+* Nameservers or search paths exceed system limits. (Three nameservers, six
+  search paths, 256 characters for `glibc`).
+* Invalid option appears for the given platform.
+
+# References
+
+*   [Kubernetes DNS name specification](https://github.com/kubernetes/dns/blob/master/docs/specification.md)
+*   [`/etc/resolv.conf manpage`](http://manpages.ubuntu.com/manpages/zesty/man5/resolv.conf.5.html)