[ci][1/n] Rebuild Docker images if necessary #11329

driazati · 2022-05-16T20:47:36Z

This sets us up to rebuild Docker images and uses them in later stages in the same build. This is not yet enabled (see the commented out code in DockerBuild.groovy.j2. If the build is running on main, then the images are uploaded to Docker Hub automatically once the run is complete. To avoid landing a complicated infra PR all at once this mostly adds the scaffolding and a follow up PR will: 1. upload images to tlcpack instead of tlcpackstaging 2. dynamically determine the docker image to use based on the latest hash 3. always rebuild docker images if an updated one hasn't been uploaded

cc @Mousius @areusch

areusch

thanks @driazati, added some comments

jenkins/Jenkinsfile.j2

tests/scripts/git_utils.py

tests/scripts/should_rebuild_docker.py

areusch · 2022-05-18T23:47:46Z

cool! i think this looks good to me, will let @Mousius have a look.

also cc @leandron

driazati · 2022-05-19T00:55:43Z

just a note for reviewers: the code is 99% done but will take a couple more iterations through CI once it’s verified before it can be merged (ie right now it still has testing code specific to this PR)

Mousius

Phew, this Jenkinsfile is getting fun 😸 Thanks for working through this @driazati, is it possible to document some of the flow here in terms of using ECR as an intermediary cache before Docker Hub? Not necessarily this PR but it'd be great to have as a reference

tests/scripts/git_utils.py

tests/python/ci/test_ci.py

tests/scripts/cmd_utils.py

tests/python/ci/test_ci.py

jenkins/Jenkinsfile.j2

Mousius · 2022-05-19T10:45:12Z

jenkins/Jenkinsfile.j2

+          ).trim()
+          def tag = "${date_Ymd_HMS}-${upstream_revision.substring(0, 8)}"
+          {% for image in images %}
+          update_docker({{ image.name }}, "tlcpackstaging/test_{{ image.name }}:${tag}")


I think this is literally for testing, but I'll leave a comment just in case 😸

Suggested change

update_docker({{ image.name }}, "tlcpackstaging/test_{{ image.name }}:${tag}")

update_docker({{ image.name }}, "tlcpackstaging/{{ image.name }}:${tag}")

jenkins/Jenkinsfile.j2

driazati · 2022-05-31T20:05:16Z

@Mousius I've re-worked this PR a bit, I'm weary about landing it all at once since it can potentially have a big impact on CI times. In a follow up PR I'll enable the docker image rebuilds based on the Docker Hub hashes behind a flag in Jenkins so we can turn it off if we need to, but this one is ready to go now

driazati · 2022-06-03T19:07:27Z

@tvm-bot rerun

Mousius

@driazati I left a few questions but I'm ok with follow up in the next in the PR series. I also manually triggered CI to re-run as the bot didn't seem to be doing it?

Mousius · 2022-06-06T09:02:39Z

jenkins/Build.groovy.j2

+  timeout(time: max_time, unit: 'MINUTES') {
+    try {
+      cmake_build(docker_type, path, make_flag)
+      // always run cpp test when build


I think this comment has been sneaking around with this function but actually no longer relates to this function?

Mousius · 2022-06-06T09:02:51Z

jenkins/Deploy.groovy.j2

+            ).trim()
+            def tag = "${date_Ymd_HMS}-${upstream_revision.substring(0, 8)}"
+            {% for image in images %}
+            update_docker({{ image.name }}, "tlcpackstaging/test_{{ image.name }}:${tag}")


I presume this is just to avoid conflicts with the over night build?

right, in the follow up(s) we'll switch it over

driazati · 2022-06-07T20:34:38Z

I also manually triggered CI to re-run as the bot didn't seem to be doing it?

the bot is broken but fixes are in flight, we're looking into better ways to test these kinds of things before deploying (e.g. the merge functionality also isn't working due to some auth errors I haven't been able to sort out yet)

This rebuilds Docker images and uses them in later stages in the same build. If the build is running on `main`, then the images are uploaded to Docker Hub automatically once the run is complete. Images are always rebuilt, but Docker Hub functions as a cache. If there have been no changes to `docker/` since the last available hash on Docker Hub, then the build will just use the images from Hub.

Mousius · 2022-06-09T22:03:22Z

Nice work @driazati, looking forward to seeing this fully functional 😸

This rebuilds Docker images and uses them in later stages in the same build. If the build is running on `main`, then the images are uploaded to Docker Hub automatically once the run is complete. Images are always rebuilt, but Docker Hub functions as a cache. If there have been no changes to `docker/` since the last available hash on Docker Hub, then the build will just use the images from Hub.

This activates the code from apache#11329 behind a couple Jenkins flags: * `USE_AUTOUPDATED_DOCKER_IMAGES` - whether or not to use the dynamically determined docker images in the build * `PUSH_DOCKER_IMAGES` - whether or not to push built images to `tlcpack` on `main` These are there so we can quickly toggle this behavior off it breaks anything. Once this is on though, any docker changes in a PR should be reflected entirely in that build, so there would be no need for a separate PR to update the Docker images. We will have to see if this works in practice, as the docker image builds do download quite a bit of data (which can flake) and add some runtime overhead (about 30m), so when an update lands all PRs will end up needing to rebuild until the merged commit finishes.

This rebuilds Docker images and uses them in later stages in the same build. If the build is running on `main`, then the images are uploaded to Docker Hub automatically once the run is complete. Images are always rebuilt, but Docker Hub functions as a cache. If there have been no changes to `docker/` since the last available hash on Docker Hub, then the build will just use the images from Hub.

driazati force-pushed the rebuild_images2 branch 10 times, most recently from f45948c to 7fbdda7 Compare May 17, 2022 22:40

driazati mentioned this pull request May 17, 2022

[RFC] A proposed update to the Docker images ci_* tag pattern apache/tvm-rfcs#66

Open

driazati force-pushed the rebuild_images2 branch from 7fbdda7 to bb9e25b Compare May 18, 2022 15:51

driazati changed the title ~~[ci][wip] Rebuild Docker images if necessary~~ [ci] Rebuild Docker images if necessary May 18, 2022

driazati marked this pull request as ready for review May 18, 2022 18:01

areusch reviewed May 18, 2022

View reviewed changes

github-actions bot requested a review from Mousius May 18, 2022 19:30

driazati force-pushed the rebuild_images2 branch 3 times, most recently from 677df48 to e21ffa8 Compare May 18, 2022 22:38

areusch approved these changes May 18, 2022

View reviewed changes

Mousius requested changes May 19, 2022

View reviewed changes

driazati force-pushed the rebuild_images2 branch from e21ffa8 to 726dfcd Compare May 19, 2022 19:15

driazati mentioned this pull request May 19, 2022

[ci] Use Docker images built in prior steps #10664

Closed

driazati force-pushed the rebuild_images2 branch 5 times, most recently from e26df5b to 6feabe4 Compare May 23, 2022 20:21

driazati force-pushed the rebuild_images2 branch 4 times, most recently from 97688a8 to 5c4613d Compare May 31, 2022 20:03

driazati changed the title ~~[ci] Rebuild Docker images if necessary~~ [ci] Add code to rebuild Docker images if necessary May 31, 2022

driazati changed the title ~~[ci] Add code to rebuild Docker images if necessary~~ [ci][1/n] Add code to rebuild Docker images if necessary May 31, 2022

driazati changed the title ~~[ci][1/n] Add code to rebuild Docker images if necessary~~ [ci][1/n] Rebuild Docker images if necessary May 31, 2022

driazati requested a review from Mousius June 1, 2022 14:59

driazati force-pushed the rebuild_images2 branch 2 times, most recently from 11ff3d4 to a462745 Compare June 2, 2022 22:56

Mousius approved these changes Jun 6, 2022

View reviewed changes

driazati force-pushed the rebuild_images2 branch 2 times, most recently from 9110565 to aa75d80 Compare June 7, 2022 20:33

driazati force-pushed the rebuild_images2 branch from aa75d80 to 69363eb Compare June 9, 2022 16:42

Mousius merged commit 6d557ff into apache:main Jun 9, 2022

driazati mentioned this pull request Jun 10, 2022

[ci][docker] Automatically update docker images #11673

Closed

driazati mentioned this pull request Jul 14, 2022

TVM v0.9.0.rc0 Release Candidate Notes #12102

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ci][1/n] Rebuild Docker images if necessary #11329

[ci][1/n] Rebuild Docker images if necessary #11329

driazati commented May 16, 2022 •

edited

Loading

areusch left a comment

areusch commented May 18, 2022

driazati commented May 19, 2022

Mousius left a comment

Mousius May 19, 2022

driazati commented May 31, 2022

driazati commented Jun 3, 2022

Mousius left a comment

Mousius Jun 6, 2022

Mousius Jun 6, 2022

driazati Jun 7, 2022

driazati commented Jun 7, 2022

Mousius commented Jun 9, 2022

	update_docker({{ image.name }}, "tlcpackstaging/test_{{ image.name }}:${tag}")
	update_docker({{ image.name }}, "tlcpackstaging/{{ image.name }}:${tag}")

[ci][1/n] Rebuild Docker images if necessary #11329

[ci][1/n] Rebuild Docker images if necessary #11329

Conversation

driazati commented May 16, 2022 • edited Loading

areusch left a comment

Choose a reason for hiding this comment

areusch commented May 18, 2022

driazati commented May 19, 2022

Mousius left a comment

Choose a reason for hiding this comment

Mousius May 19, 2022

Choose a reason for hiding this comment

driazati commented May 31, 2022

driazati commented Jun 3, 2022

Mousius left a comment

Choose a reason for hiding this comment

Mousius Jun 6, 2022

Choose a reason for hiding this comment

Mousius Jun 6, 2022

Choose a reason for hiding this comment

driazati Jun 7, 2022

Choose a reason for hiding this comment

driazati commented Jun 7, 2022

Mousius commented Jun 9, 2022

driazati commented May 16, 2022 •

edited

Loading