Merge pull request #40 from iovisor/32-pod-security-policy

feat(kubectl-trace): allow passing a service account to enable usage of pod security policies
iovisor · Jan 17, 2019 · 3e8de86 · 3e8de86
2 parents 9e635f2 + b159809
commit 3e8de86
Show file tree

Hide file tree

Showing 3 changed files with 145 additions and 32 deletions.
diff --git a/README.md b/README.md
@@ -4,7 +4,12 @@
 
 - [Installation](#installation)
 - [Usage](#usage)
+  * [Run a program from string literal](#run-a-program-from-string-literal)
+  * [Run a program from file](#run-a-program-from-file)
+  * [Run a program against a Pod](#run-a-program-against-a-pod)
   * [Running against a Pod vs against a Node](#running-against-a-pod-vs-against-a-node)
+  * [Using a custom service account](#using-a-custom-service-account)
+  * [Executing in a cluster using Pod Security Policies](#executing-in-a-cluster-using-pod-security-policies)
   * [More bpftrace programs](#more-bpftrace-programs)
 - [Status of the project](#status-of-the-project)
 - [Contributing](#contributing)
@@ -29,7 +34,7 @@ This will download and compile `kubectl-trace` so that you can use it as a kubec
 You don't need to setup anything on your cluster before using it, please don't use it already
 on a production system, just because this isn't yet 100% ready.
 
-**Run a program from string literal:**
+### Run a program from string literal
 
 In this  case we are running a program that probes a tracepoint
 on the node `ip-180-12-0-152.ec2.internal`.
@@ -39,15 +44,15 @@ kubectl trace run ip-180-12-0-152.ec2.internal -e "tracepoint:syscalls:sys_enter
 ```
 
 
-**Run a program from file:**
+### Run a program from file
 
 Here we run a program named `read.bt` against the node `ip-180-12-0-152.ec2.internal`
 
 ```
 kubectl trace run ip-180-12-0-152.ec2.internal -f read.bt
 ```
 
-**Run a program against a Pod**
+### Run a program against a Pod
 
 ![Screenshot showing the read.bt program for kubectl-trace](docs/img/pod.png)
 
@@ -83,6 +88,114 @@ So, running against a pod **doesn't mean** that your bpftrace program will be co
 knowledge of the context of a container, in this case only the root process id is supported via the `$container_pid` variable.
 
 
+### Using a custom service account
+
+By default `kubectl trace` will use the `default` service account in the target namespace (that is also `default`), to schedule the pods needed for your bpftrace program.
+
+If you need to pass a service account you can use the `--serviceaccount` flag.
+
+```bash
+kubectl trace run --serviceaccount=kubectltrace ip-180-12-0-152.ec2.internal -f read.bt
+```
+
+### Executing in a cluster using Pod Security Policies
+
+If your cluster has pod security policies you will need to make so that `kubectl trace` can
+use a service account that can run privileged containers.
+
+That service account, then will need to be in a group that uses the proper privileged `PodSecurityPolicy`.
+
+First, create the service account that you will use with `kubectl trace`,
+you can use a different namespace other than `default`, just remember to pass that namespace to the `run` command when you will use `kubectl trace`:
+
+```yaml
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: kubectltrace
+  namespace: default
+```
+
+Now that we have a `kubectltrace` service account let's create a Pod Security Policy:
+
+```yaml
+apiVersion: policy/v1beta1
+kind: PodSecurityPolicy
+metadata:
+  name: kubectltrace
+spec:
+  fsGroup:
+    rule: RunAsAny
+  privileged: true
+  runAsUser:
+    rule: RunAsAny
+  seLinux:
+    rule: RunAsAny
+  supplementalGroups:
+    rule: RunAsAny
+  volumes:
+  - '*'
+  allowedCapabilities:
+  - '*'
+  hostPID: true
+  hostIPC: true
+  hostNetwork: true
+  hostPorts:
+  - min: 1
+    max: 65536
+```
+
+Ok, this `PodSecurityPolicy` will allow users assigned to it to run privileged containers,
+`kubectl trace` needs that because of the extended privileges eBPF programs need to run with
+to trace your kernel and programs running in it.
+
+Now with a `ClusterRoleBinding` you bind the `ClusterRole` with the `ServiceAccount`, so that
+they can work together with the `PodSecurityPolicy` we just created.
+
+You can change the `namespace: default` here if you created the service account in a namespace other than `default`.
+
+```yaml
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: kubectltrace-psp
+rules:
+- apiGroups:
+  - policy
+  resources:
+  - podsecuritypolicies
+  resourceNames:
+  - kubectltrace
+  verbs:
+  - use
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+    name: kubectltrace-psp
+subjects:
+- kind: ServiceAccount
+  name: kubectltrace
+  namespace: default
+roleRef:
+   apiGroup: rbac.authorization.k8s.io
+   kind: ClusterRole
+   name: kubectltrace-psp
+```
+
+OK! Now that we are all set we can just run the program by specifying the service account
+we just created and it will use our pod security policy!
+
+```bash
+kubectl trace run --serviceaccount=kubectltrace ip-180-12-0-152.ec2.internal -f read.bt
+```
+
+If you used a different namespace other than default for your service account, you will want to specify the namespace too, like this:
+
+```bash
+kubectl trace run --namespace=mynamespace --serviceaccount=kubectltrace ip-180-12-0-152.ec2.internal -f read.bt
+```
+
 ### More bpftrace programs
 
 Need more programs? Look [here](https://github.com/iovisor/bpftrace/tree/master/tools).
@@ -119,10 +232,6 @@ kubectl trace run pod/<pod-name> -c <container> f read.bt
 
 So I would say, the next thing is to run bpftrace programs at a pod scope other than at node scope.</strike>
 
-**bpftrace work**
-
-I also plan to contribute some IO functions to bpftrace to send data to a backend database like InfluxDB instead of only stdout
-because that would enable having things like graphs showing 
 
 ## Contributing
 

diff --git a/pkg/cmd/run.go b/pkg/cmd/run.go
@@ -54,13 +54,14 @@ type RunOptions struct {
 	explicitNamespace bool
 
 	// Local to this command
-	container   string
-	eval        string
-	program     string
-	resourceArg string
-	attach      bool
-	isPod       bool
-	podUID      string
+	container      string
+	eval           string
+	program        string
+	resourceArg    string
+	attach         bool
+	isPod          bool
+	podUID         string
+	serviceAccount string
 
 	nodeName string
 
@@ -103,6 +104,7 @@ func NewRunCommand(factory factory.Factory, streams genericclioptions.IOStreams)
 	cmd.Flags().BoolVarP(&o.attach, "attach", "a", o.attach, "Wheter or not to attach to the trace program once it is created")
 	cmd.Flags().StringVarP(&o.eval, "eval", "e", "", "Literal string to be evaluated as a bpftrace program")
 	cmd.Flags().StringVarP(&o.program, "filename", "f", "", "File containing a bpftrace program")
+	cmd.Flags().StringVar(&o.serviceAccount, "serviceaccount", "default", "Service account to use to set in the pod spec of the kubectl-trace job")
 
 	return cmd
 }
@@ -265,14 +267,15 @@ func (o *RunOptions) Run() error {
 	}
 
 	tj := tracejob.TraceJob{
-		Name:          fmt.Sprintf("%s%s", meta.ObjectNamePrefix, string(juid)),
-		Namespace:     o.namespace,
-		ID:            juid,
-		Hostname:      o.nodeName,
-		Program:       o.program,
-		PodUID:        o.podUID,
-		ContainerName: o.container,
-		IsPod:         o.isPod,
+		Name:           fmt.Sprintf("%s%s", meta.ObjectNamePrefix, string(juid)),
+		Namespace:      o.namespace,
+		ServiceAccount: o.serviceAccount,
+		ID:             juid,
+		Hostname:       o.nodeName,
+		Program:        o.program,
+		PodUID:         o.podUID,
+		ContainerName:  o.container,
+		IsPod:          o.isPod,
 	}
 
 	job, err := tc.CreateJob(tj)

diff --git a/pkg/tracejob/job.go b/pkg/tracejob/job.go
@@ -24,14 +24,15 @@ type TraceJobClient struct {
 }
 
 type TraceJob struct {
-	Name          string
-	ID            types.UID
-	Namespace     string
-	Hostname      string
-	Program       string
-	PodUID        string
-	ContainerName string
-	IsPod         bool
+	Name           string
+	ID             types.UID
+	Namespace      string
+	ServiceAccount string
+	Hostname       string
+	Program        string
+	PodUID         string
+	ContainerName  string
+	IsPod          bool
 }
 
 // WithOutStream setup a file stream to output trace job operation information
@@ -101,7 +102,6 @@ func (t *TraceJobClient) findConfigMapsWithFilter(nf TraceJobFilter) ([]apiv1.Co
 }
 
 func (t *TraceJobClient) GetJob(nf TraceJobFilter) ([]TraceJob, error) {
-
 	jl, err := t.findJobsWithFilter(nf)
 	if err != nil {
 		return nil, err
@@ -218,7 +218,8 @@ func (t *TraceJobClient) CreateJob(nj TraceJob) (*batchv1.Job, error) {
 			Template: apiv1.PodTemplateSpec{
 				ObjectMeta: commonMeta,
 				Spec: apiv1.PodSpec{
-					HostPID: true,
+					HostPID:            true,
+					ServiceAccountName: nj.ServiceAccount,
 					Volumes: []apiv1.Volume{
 						apiv1.Volume{
 							Name: "program",