Delta Update Server for Open Balena

Open source delta update server for openbalena, a platform to deploy and manage connected devices.

Features

The goal of this project is to faciliate delta image updates, which result in significantly less traffic to devices when downloading images from your open-balena servers. Traditional updates result in devices re-downloading all layers after the point where a change is made in the dockerfile, even if those subsequent layers have not changed. Delta updates utilize balena-engine (v3) or rsync (v2) to do a binary compare of the image, and only result in downloading of the changed data relative to the image already installed on a device.

Compatibility

This project is compatible with open-balena and specifically relies on the open-balena-registry component.

Installation

open-balena-delta is meant to be installed as part of open-balena, and ideally at the same time. For thoes running open-balena on k8s, we have included services to build it in the open-balena helm project. If you are running open-balena via docker-compose, you will need to modify the scripts to mirror the setup in the helm charts or recreate it using the configuration steps below.

To configure open-balena-delta you must define four environment variables and two volumes for the container:

Environment Variables:

• BALENA_TLD: open-balena base hostname i.e. openbalena.<yourdomain.com>
• REGISTRY_HOST: open-balena-registry hostmame, i.e. registry.openbalena.<yourdomain.com>
• DOCKER_HOST: You likely want to set this to "unix:///var/run/docker.sock"
• DOCKER_BUILDKIT: You likely want to set this to "0"
• TOKEN_AUTH_BUILDER_TOKEN: The master builder token which was generated by the qucikstart script when you installed open-balena

Volumes:

• /var/lib/balena-engine: This volume is needed to support overlay2 file system in a container
• /delta-rsync: This volume is needed to store rsync deltas, which are necessary if using v2 deltas

You will also need to set up a new hostname for your open-balena-delta instance, which likely will just be mapped to your open-balena haproxy instance, assuming you used the helm script or otherwise integrated it with your docker-compose scripts.

Hostnames:

• delta.<yourdomain.com>: IP address / hostname of open-balena-haproxy

Certificates:

If you want to enable authentication (i.e. only accept incoming delta requests from authenticated devices) you will need to include the open-balena-api public key certificate (which is used for JWT signing) in the open-balena-delta container, stored at /certs/private/api.<BALENA_TLD>.pem. If you don't provide this file, authentication will be automatically disabled.

Integration

To integrate open-balena-delta in your open-balena environment, you will need to make some configuration changes as outlined below. If you utilize the open-balena-helm scripts, this will be handled for you. Below is a summary of what needs to be done.

Changes to open-balena-api

You will need to add an the following environment variable to your open-balena-api container:

• DELTA_HOST: The hostname of your open-balena-delta instance, i.e. delta.openbalena.<yourdomain.com>

Fleet / device config variables

There are two config variables that need to be added to your fleet (to apply to all devices in the fleet) or device (to apply only to one device):

• BALENA_SUPERVISOR_DELTA: Set this to 1
• BALENA_SUPERVISOR_DELTA_VERSION: You likely want to set this to 3 (to use balena-engine delta updates), but for older devices you will need to set it to 2 (to use rsync delta updates)

Device config.json changes

You will need to add a parameter to config.json on devices that will be paritcipating in delta updates:

• deltaEndpoint: The hostname of your open-balena-delta instance, i.e. delta.openbalena.<yourdomain.com>

Limitations and Known Issues

• When base images change, i.e. if the balenalib/raspberrypi4-64-debian:bookworm tag is pointed to a new build, delta updates will not work as expected per this thread which is a known limitation of balena-engine. In this scenario, devices will download new base images and new layers, even if the delta changes are minimal. Recommend pinning your base images to hashes instead of tags to avoid this behavior, otherwise ensure that devices have sufficient storage capacity to handle 2x your app stack. Alternatively, per the thread above, you can run a "delete then download" update if devices are storage constrained.
• Needs some form of authentication for rsync download endpoint.

Credits

• Major kudos to the balena team for developing balena-engine, an amazing tool that improves on docker specifically for embedded and IoT use cases.