Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for cri-resource-manager project #232

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add support for cri-resource-manager project #232

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions clr-k8s-examples/10-cri-resource-manager/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# CRI Resource Manager
CRI Resource Manager serves as a relay/proxy between kubelet and the container runtime, relaying requests and responses back and forth between these two, potentially altering requests as they fly by.

This document explains a very simple use case for the `cri-resource-manager`, for more details and tweaks
on CRI Resource Manager service, you can go to https://github.com/intel/cri-resource-manager.

## Install

[`install.sh`](install.sh) script will download the binary and install it as an `systemd` service unit. This script will be executed in all nodes where `cri-resmgr` is required.

Below you can see the available variables you can use to customize the usage of your CRI Resource Manager service.

| Variable | Description | Default Value |
|-----------------------------|-------------------------------------------|--------------------------------------------------|
| `RUNNER` | Default Container Runtime | `containerd` |
| `CRI_RESMGR_POLICY` | CRI Resource Manager Policy type | `null` |
| `CRI_RESMGR_POLICY_OPTIONS` | CRI Resource Manager extra policy options | `-dump='reset,full:.*' -dump-file=/tmp/cri.dump` |
| `CRI_RESMGR_DEBUG_OPTIONS` | CRI Resource Manager debugging options | `<none>` |

**Example:**
```bash
$ RUNNER=containerd ./install.sh
```

Verify that the cri-resource-manager service is actually running.

```bash
$ systemctl status cri-resource-manager
```

Verify that the `cri-resmgr` socket is created, it will indicate that `cri-resource-manager` is ready to receive requests.
```bash
$ sudo ls -la /var/run/cri-resmgr/cri-resmgr.sock
```

grahamwhaley marked this conversation as resolved.
Show resolved Hide resolved
## Setup as a container runtime in `kubelet`

The [`setup.sh`](setup.sh) script will configure the `kubelet` service to use the `cri-resource-manager` relay as its remote container runtime. This script will be executed in all nodes where `cri-resmgr` is being configured.

**Example:**
```bash
$ ./setup.sh
```

Kubelet service should be restarted and now using `cri-resource-manager` as its container runtime

```bash
$ ps aux | grep kubelet | grep container-runtime
root 28703 1.7 2.0 1246348 83088 ? Ssl 20:03 0:06 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime remote --container-runtime-endpoint unix:///var/run/cri-resmgr/cri-resmgr.sock
```

`cri-resource-manager` service's logs will be located at `/tmp/cri.dump`

```bash
$ tail /tmp/cri.dump
```

## Cleanup

The [`clean.sh`](clean.sh) will first clean the `kubelet` service as it was before the `cri-resource-manager` and restarts `kubelet` service. This script will be executed in all nodes where `cri-resmgr` is being uninstalled.
Then, it will proceed to stop the `cri-resource-manager` service.

**Example:**
```bash
$ ./clean.sh
```

## More kubernetes native approach (experimental)

In case that you're interested in a more Kubernetes native way of deploying the CRI Resource manager, take a look on: https://github.com/intel/cri-resource-manager/pull/55
23 changes: 23 additions & 0 deletions clr-k8s-examples/10-cri-resource-manager/clean.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
#!/bin/bash
grahamwhaley marked this conversation as resolved.
Show resolved Hide resolved
# Copyright (c) 2019 Intel Corporation
#
# SPDX-License-Identifier: Apache-2.0
#
# Uninstall and stop the CRI Resource Manager service

set -o errexit
set -o nounset

# Kubelet
KUBEADM_FLAGS="/var/lib/kubelet/kubeadm-flags.env"
sudo rm -f /etc/systemd/system/kubelet.service.d/99-cri-resource-manager.conf
sudo systemctl daemon-reload
sudo systemctl restart kubelet

if sudo test -f "$KUBEADM_FLAGS.bkp" ; then
sudo mv $KUBEADM_FLAGS.bkp $KUBEADM_FLAGS
fi

# CRI Resource Manager
sudo systemctl stop cri-resource-manager
sudo systemctl disable cri-resource-manager
36 changes: 36 additions & 0 deletions clr-k8s-examples/10-cri-resource-manager/install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/bin/bash
# Copyright (c) 2019 Intel Corporation
#
# SPDX-License-Identifier: Apache-2.0
#
# Install and start the CRI Resource Manager service

set -o errexit
set -o nounset

RUNNER=${RUNNER:-"containerd"}
CRI_RESMGR_POLICY=${CRI_RESMGR_POLICY:-"null"}
CRI_RESMGR_POLICY_OPTIONS=${CRI_RESMGR_POLICY_OPTIONS:-"-dump='reset,full:.*' -dump-file=/tmp/cri.dump"}
CRI_RESMGR_DEBUG_OPTIONS=${CRI_RESMGR_DEBUG_OPTIONS:-""}

curl https://raw.githubusercontent.com/obedmr/cri-resource-manager/master/godownloader.sh | bash
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I'm going to struggle to ack a script that curls down a binary from a user account and installs it...
Can we try and make this something from either the official CRI-RM repo (or release, if they are maybe released on docker hub for instance), or a 'pull and build from official sources' script?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and, curl | bash is generally just seen as a big security hole.... ;-)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, CRI-Resource-Manager project's plans for releasing binaries is still on definition, we need something to continue, we may use this and later update when official container images or binaries are released.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to hold off until CRI-RM gets their binary release method sorted out. I see you have the in-flight intel/cri-resource-manager#55 which creates a deployment style method, which is great. That will require a formal process of pushing a pre-built deployment container image somwhere for CRI-RM.
It would be good to hear if there is some general timeline for CRI-RM binary releases planned - can we ask the CRI-RM team?

sudo cp ./bin/* /usr/bin/

runtime_socket=$(sudo find /run/ -iname $RUNNER.sock | head -1)
CRI_RESMGR_POLICY_OPTIONS+=" -runtime-socket=$runtime_socket -image-socket=$runtime_socket"

sudo mkdir -p /etc/sysconfig/
cat <<EOF | sudo tee /etc/sysconfig/cri-resource-manager
POLICY=$CRI_RESMGR_POLICY
POLICY_OPTIONS=$CRI_RESMGR_POLICY_OPTIONS
DEBUG_OPTIONS=$CRI_RESMGR_DEBUG_OPTIONS
EOF

sudo mkdir -p /etc/systemd/system/
curl https://raw.githubusercontent.com/obedmr/cri-resource-manager/master/cmd/cri-resmgr/cri-resource-manager.service | sudo tee /etc/systemd/system/cri-resource-manager.service

sudo sed -i '/Requires=/d' /etc/systemd/system/cri-resource-manager.service
sudo systemctl daemon-reload
sudo systemctl restart cri-resource-manager.service
sudo systemctl enable cri-resource-manager.service

29 changes: 29 additions & 0 deletions clr-k8s-examples/10-cri-resource-manager/setup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
#!/bin/bash
# Copyright (c) 2019 Intel Corporation
#
# SPDX-License-Identifier: Apache-2.0
#
# Configure CRI Resource Manager as container runtime endpoint for kubelet

set -o errexit
set -o nounset

CRI_RESMGR_SOCKET="/var/run/cri-resmgr/cri-resmgr.sock"
KUBEADM_FLAGS="/var/lib/kubelet/kubeadm-flags.env"

if sudo test -S "$CRI_RESMGR_SOCKET" ; then
sudo mkdir -p /etc/systemd/system/kubelet.service.d/
cat <<EOF | sudo tee /etc/systemd/system/kubelet.service.d/99-cri-resource-manager.conf
[Service]
Environment=KUBELET_EXTRA_ARGS=
Environment=KUBELET_EXTRA_ARGS="--container-runtime remote --container-runtime-endpoint unix://${CRI_RESMGR_SOCKET}"
EOF

if sudo test -f "$KUBEADM_FLAGS" ; then
sudo mv $KUBEADM_FLAGS $KUBEADM_FLAGS.bkp
fi

sudo systemctl daemon-reload
sudo systemctl restart cri-resource-manager
sudo systemctl restart kubelet
fi
3 changes: 3 additions & 0 deletions clr-k8s-examples/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,9 @@ kubectl -n monitoring port-forward svc/grafana 3000
Grafana is available at this URL http://localhost:3000 . Default credentials are
`admin/admin`. Upon entering you will be asked to chose a new password.

### CRI Resource Manager
Go to [`10-cri-resource-manager`](./10-cri-resource-manager).

## Cleaning up the cluster (Hard reset to a clean state)

Run `reset_stack.sh` on all the nodes