images: some refactoring and version bumps for arm64 compatible Dockerfiles #423

consideRatio · 2021-08-21T21:49:56Z

This is a step along the way for #421 about compatibility on arm64 machines like Raspberry Pi.

consideRatio · 2021-08-21T21:53:48Z

dask-gateway-server/Dockerfile

@@ -1,18 +1,19 @@
-FROM python:3.8.6-slim-buster as dependencies
+FROM python:3.8.11-slim-buster as dependencies


General maintenance keeping the base Python version updated

consideRatio · 2021-08-21T21:58:16Z

dask-gateway-server/Dockerfile

 LABEL MAINTAINER="Jim Crist-Harif"

 RUN apt-get update \
    && apt-get install -y tini \
-    && apt-get clean \


apt-get clean is automatically run in this and most images via a config (/etc/apt/apt.conf.d/docker-clean).

consideRatio · 2021-08-21T22:00:01Z

dask-gateway-server/Dockerfile

-    aiohttp==3.7.2 \
+    aiohttp==3.7.4 \
    colorlog \
    cryptography \
-    traitlets==4.3.3 \
+    traitlets==5.0.5 \
    pyyaml \
-    kubernetes-asyncio==11.0.0
+    kubernetes-asyncio==12.1.1


Systematic version updates as I had to update aiohttp in order to have a arm64 (aka. aarch64) compatible conda package in the dask-gateway image.

consideRatio · 2021-08-21T22:00:42Z

dask-gateway/Dockerfile

@@ -2,27 +2,34 @@
 FROM debian:buster-slim as miniconda
 LABEL MAINTAINER="Jim Crist-Harif"

-ARG CONDA_VERSION=py38_4.8.3
-ARG CONDA_SHA256=879457af6a0bf5b34b48c12de31d4df0ee2f06a8e68768e5758c3293b2daf688


Removal of checksums as amd64 / arm64 will have different checksums and it is a bit tedious to keep track of both.

consideRatio · 2021-08-21T22:01:42Z

dask-gateway/Dockerfile

 # - Create user dask
+RUN useradd -m -U -u 1000 dask
+
 # - Install tini
 # - Install miniconda build dependencies
-# - Download miniconda and check the sha256 checksum
-# - Install miniconda
+RUN apt-get update \
+    && apt-get install -y tini wget bzip2 \
+    && rm -rf /var/lib/apt/lists/*


I split this apart from a section below making so many different things in the same RUN layer that it became a bit hard to read.

consideRatio · 2021-08-21T22:03:59Z

dask-gateway/Dockerfile

@@ -32,39 +39,40 @@ RUN useradd -m -U -u 1000 dask \
    && echo "aggressive_update_packages: []" >> /home/dask/.condarc \
    && find /opt/conda/ -follow -type f -name '*.a' -delete \
    && /opt/conda/bin/conda clean -afy \
-    && chown -R dask:dask /opt/conda \
-    && apt-get autoremove --purge -y wget bzip2 \


By not removing wget / bzip2 the image will not grow further because adding + removing won't help if its not done in the same layer, and since it is no longer in the same layer I don't remove these binaries later.

The reason for having one more RUN statements was for readability.

Do you have a rough idea of how much larger the image is now?

My rough idea is that it is minimal (<1MB) and not worth worrying about. I did not confirm this thoroughly in an e2e build comparison though.

$ apt-cache --no-all-versions show wget | grep '^Size: ' Size: 348824 $ apt-cache --no-all-versions show bzip2 | grep '^Size: ' Size: 34064

Ah, so it seems like 992 kB + 195 kB.

$ apt-cache --no-all-versions show wget | grep Installed-Size Installed-Size: 992 $ apt-cache --no-all-versions show bzip2 | grep Installed-Size Installed-Size: 195

consideRatio · 2021-08-21T22:04:43Z

dask-gateway/Dockerfile

-ARG DASK_VERSION=2021.7.1
-ARG DISTRIBUTED_VERSION=2021.7.1
+ARG DASK_VERSION=2021.8.1
+ARG DISTRIBUTED_VERSION=2021.8.1


I just bumped these while I was at it so all the conda/pip packages installed were updated across now that I had to update at least aiohttp.

consideRatio · 2021-08-21T22:05:33Z

dask-gateway/Dockerfile

+
+

For readability as separation between the build steps

jacobtomlinson

This generally looks good to me!

We recently added Arm support to the Dask docker images in dask/dask-docker#166.

I wonder if there are any learnings we can take from that PR, or anything we can reuse to ensure a bit more consistency between projects.

consideRatio · 2021-08-23T11:22:50Z

We recently added Arm support to the Dask docker images in dask/dask-docker#166.

I wonder if there are any learnings we can take from that PR, or anything we can reuse to ensure a bit more consistency between projects.

Ah one key thing to keep consistent is a transition to mamba and mambaforge to install it. I'd be happy to help with that as well but it felt like out of scope for this PR to do that also.

Oh @holdenk and I worked on these kinds of things in jupyter/docker-stacks also btw! :) 👋 Happy to see you contribute here as well @holdenk!!! Want to help by contributing a review of this PR also?

consideRatio · 2021-08-26T20:13:15Z

I can address the conda to mamba transition as well, but I think it should be a dedicated PR. I opened #432 to represent this.

TomAugspurger

Thanks for this, one question about image sizes inline.

I'd love it if we could come up with a policy about what library versions to use in the Dockerfiles, to avoid having repeated conversations about versions. Is there any reason not to use the latest version of all libraries (dask, distributed, aiohttp, etc) at the time of release?

One maybe special consideration is Python itself. Right now we're on 3.8.6 (bumping to 3.8.11). Is there any reason to keep that on 3.8? Or should we move to 3.9?

TomAugspurger · 2021-09-01T14:33:39Z

dask-gateway/Dockerfile

@@ -32,39 +39,40 @@ RUN useradd -m -U -u 1000 dask \
    && echo "aggressive_update_packages: []" >> /home/dask/.condarc \
    && find /opt/conda/ -follow -type f -name '*.a' -delete \
    && /opt/conda/bin/conda clean -afy \
-    && chown -R dask:dask /opt/conda \
-    && apt-get autoremove --purge -y wget bzip2 \


Do you have a rough idea of how much larger the image is now?

consideRatio · 2021-09-01T17:29:20Z

I'd love it if we could come up with a policy about what library versions to use in the Dockerfiles, to avoid having repeated conversations about versions. Is there any reason not to use the latest version of all libraries (dask, distributed, aiohttp, etc) at the time of release?

UPDATE: @TomAugspurger I created #437 regarding this question. Let's move this discussion to there.

I think a good policy is to use a requirements.in file that is automatically refrozen to requirements.txt by dependabot, and then we install such requirements.txt in the dockerfile. That helps users get info about what version has been used in each release via git history, we avoid a maintenance burden, and we get security warnings via automation etc.

We have a mix of conda/pip in these files though, making my idea a bit troublesome. I'd consider going 100% pip in these images or 100% conda and hope there is a good tool like there is for pip that dependabot can use for an environment.yml file

consideRatio · 2021-09-11T13:09:38Z

Go for merge on this?

jcrist

Thanks @consideRatio!

consideRatio · 2021-09-14T16:02:01Z

Thanks @jacobtomlinson @TomAugspurger @jcrist for your review efforts!

images: some refactoring and arm64 support

882c628

consideRatio mentioned this pull request Aug 21, 2021

Helm chart: make images arm64 compatible and publish images supporting both amd64 and arm64 #421

Closed

consideRatio changed the title ~~images: some refactoring and arm64 support~~ images: some refactoring and version bumps for arm64 compatible Dockerfiles Aug 21, 2021

consideRatio commented Aug 21, 2021

View reviewed changes

jacobtomlinson reviewed Aug 23, 2021

View reviewed changes

This was referenced Aug 26, 2021

dask-gateway image far behind mainline dask version. #405

Closed

Dask helm deployment not working in AKS #386

Open

Release #381

Closed

TomAugspurger approved these changes Sep 1, 2021

View reviewed changes

consideRatio mentioned this pull request Sep 1, 2021

Let our Dockerfiles reference a dependency list, and let dependabot keep it updated #437

Open

consideRatio added codebase:helm-chart maintenance labels Sep 4, 2021

jcrist approved these changes Sep 14, 2021

View reviewed changes

jcrist merged commit 8dd5b70 into dask:main Sep 14, 2021

consideRatio deleted the pr/support-image-build-on-arm64 branch March 25, 2022 01:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

images: some refactoring and version bumps for arm64 compatible Dockerfiles #423

images: some refactoring and version bumps for arm64 compatible Dockerfiles #423

consideRatio commented Aug 21, 2021 •

edited

Loading

consideRatio Aug 21, 2021

consideRatio Aug 21, 2021

consideRatio Aug 21, 2021

consideRatio Aug 21, 2021

consideRatio Aug 21, 2021

consideRatio Aug 21, 2021

TomAugspurger Sep 1, 2021

consideRatio Sep 1, 2021

consideRatio Sep 1, 2021

consideRatio Aug 21, 2021

consideRatio Aug 21, 2021

jacobtomlinson left a comment

consideRatio commented Aug 23, 2021 •

edited

Loading

consideRatio commented Aug 26, 2021

TomAugspurger left a comment

TomAugspurger Sep 1, 2021

consideRatio commented Sep 1, 2021 •

edited

Loading

consideRatio commented Sep 11, 2021

jcrist left a comment

consideRatio commented Sep 14, 2021

		@@ -1,18 +1,19 @@
		FROM python:3.8.6-slim-buster as dependencies
		FROM python:3.8.11-slim-buster as dependencies

images: some refactoring and version bumps for arm64 compatible Dockerfiles #423

images: some refactoring and version bumps for arm64 compatible Dockerfiles #423

Conversation

consideRatio commented Aug 21, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jacobtomlinson left a comment

Choose a reason for hiding this comment

consideRatio commented Aug 23, 2021 • edited Loading

consideRatio commented Aug 26, 2021

TomAugspurger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

consideRatio commented Sep 1, 2021 • edited Loading

consideRatio commented Sep 11, 2021

jcrist left a comment

Choose a reason for hiding this comment

consideRatio commented Sep 14, 2021

consideRatio commented Aug 21, 2021 •

edited

Loading

consideRatio commented Aug 23, 2021 •

edited

Loading

consideRatio commented Sep 1, 2021 •

edited

Loading