Add cache action to speed up mvn workflow [skip ci] #9674

YanxuanLiu · 2023-11-10T08:06:42Z

part of fix #9559

.github/workflows/mvn-verify-check.yml

YanxuanLiu · 2023-11-20T01:24:00Z

Compared the log line by line, the cache action does not help for most of dependencies in pom.

gerashegalov · 2023-11-20T22:06:23Z

This is closed prematurely IMO:

First this PR did not explore the recommendation from #9609 (comment), to add a job to the workflow that would populate most of the cache once, and all the jobs in the matrix should depend on it.

Without it , we currently fan out right away into a parallel execution that don't benefit from each other in terms of caching.

Depending which ones you compared, some workflow runs on this PR are days or weeks apart. So nightly SNAPSHOTs, including the ~500M spark-rapids-jni is repulled.

YanxuanLiu · 2023-11-21T01:05:25Z

This is closed prematurely IMO:

First this PR did not explore the recommendation from #9609 (comment), to add a job to the workflow that would populate most of the cache once, and all the jobs in the matrix should depend on it.

Without it , we currently fan out right away into a parallel execution that don't benefit from each other in terms of caching.

Depending which ones you compared, some workflow runs on this PR are days or weeks apart. So nightly SNAPSHOTs, including the ~500M spark-rapids-jni is repulled.

Got it, let me add a cache job and run some tests

.github/workflows/mvn-verify-check.yml

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

.github/workflows/mvn-verify-check.yml

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

YanxuanLiu · 2023-11-30T07:24:39Z

build

pxLi · 2023-11-30T07:30:00Z

@YanxuanLiu please keep in mind that we should add [skip ci] save resources if no need blossom resources

gerashegalov

looks pretty good already

gerashegalov · 2023-11-30T08:00:13Z

.github/workflows/mvn-verify-check.yml

+          while true; do
+            mvn help:evaluate -pl dist -Dexpression=included_buildvers \
+            -DforceStdout -PnoSnapshots -q | tr -d ',' | \
+            xargs -n 1 bash -c 'mvn initialize -pl sql-plugin-api -am -Dbuildver=$1' _ && break || {


We still do not populate the cache with Scala 2.13 artifacts: the second line in #9674 (comment)

If one statement works better for your logic we could use a loop

for pom_xml in pom.xml scala2.13/pom.xml; do mvn -f $pom_xml help:evaluate -pl dist \ -Dexpression=included_buildvers -DforceStdout -PnoSnapshots -q | tr -d ',' | \ xargs -n 1 bash -c 'mvn -f $1 initialize -pl sql-plugin-api -am -Dbuildver=$2' _ $pom_xml; done

.github/workflows/mvn-verify-check.yml

Co-authored-by: Gera Shegalov <gshegalov@nvidia.com>

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

YanxuanLiu · 2023-11-30T11:56:42Z

I've merged latest branch-24.02 to feature branch, but scala 2.13 still failed. Do you have any idea? @gerashegalov

gerashegalov

I've merged latest branch-24.02 to feature branch, but scala 2.13 still failed. Do you have any idea? @gerashegalov

need to fix a typo for buildver=$2 @YanxuanLiu

.github/workflows/mvn-verify-check.yml

Co-authored-by: Gera Shegalov <gshegalov@nvidia.com>

YanxuanLiu · 2023-11-30T14:21:10Z

I've merged latest branch-24.02 to feature branch, but scala 2.13 still failed. Do you have any idea? @gerashegalov

need to fix a typo for buildver=$2 @YanxuanLiu

looks good now. Shall we merge the PR?

gerashegalov

LGTM

YanxuanLiu · 2023-11-30T15:06:49Z

build

pxLi

LGTM. Please help keep monitoring if the cache storage usage would affect others who may share the same storage like https://github.com/orgs/NVIDIA/projects/4 or https://github.com/NVIDIA/spark-rapids/discussions thanks

@revans2

- Save the overhead of setting up the runner by folding the short-lived runner get-shim-versions-from-dist steps into cache-dependencies job - Add restore-keys to make sure cache-dependencies does not start on an empty m2, stable versions like for spark artifacts do not change - Use de.qaware.maven:go-offline-maven-plugin suggested by @revans2 - Save scala compiler bridge into a cacheable location Previous mvn workflow duration: - before #9674: ~8min or much longer with retries - on cache hit: ~7min - on cache miss: ~11min After this PR: - on cache hit: 5min - on cache miss: 7min Signed-off-by: Gera Shegalov <gera@apache.org>

gerashegalov reviewed Nov 13, 2023

View reviewed changes

.github/workflows/mvn-verify-check.yml Outdated Show resolved Hide resolved

YanxuanLiu closed this Nov 20, 2023

gerashegalov reopened this Nov 20, 2023

pxLi changed the base branch from branch-23.12 to branch-24.02 November 22, 2023 02:59

gerashegalov requested changes Nov 22, 2023

View reviewed changes

.github/workflows/mvn-verify-check.yml Outdated Show resolved Hide resolved

gerashegalov reviewed Nov 24, 2023

View reviewed changes

.github/workflows/mvn-verify-check.yml Outdated Show resolved Hide resolved

.github/workflows/mvn-verify-check.yml Show resolved Hide resolved

YanxuanLiu added 21 commits November 27, 2023 18:12

add cache for testing

e902052

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

change cache key file

75229e8

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

add cache for all stages

d0f62a8

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

change key path

086dbf0

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

add mvn help to test

af5ed08

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

add isolated cache job

bd2df16

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

add checkout

dc57563

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

add java setup

aa887b1

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

remove distribution

3d1c41e

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

add distribution

7612e46

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

add retry for cache

ce8f3e9

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

add project param

96353f9

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

make sure cache finished before shim version matrix

7e2c75e

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

add cache

0af4d88

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

add cache

c44ee1b

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

add cache

b357d6e

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

add check

47c6eaa

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

remove test step

c4ee2d2

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

add ls before cache

7a12ba9

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

add check

0a7172a

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

removed test steps

2ade9fd

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

YanxuanLiu added 4 commits November 30, 2023 10:13

change ref to target branch

2dcbc70

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

generate key in cache job

671fbc0

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

upadte key

271af46

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

echo key for testing

845a40c

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

YanxuanLiu force-pushed the cache-deps branch from a809c96 to 845a40c Compare November 30, 2023 06:18

YanxuanLiu added 2 commits November 30, 2023 14:26

fix bug of time

98b0ad5

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

test date

67c9236

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

gerashegalov reviewed Nov 30, 2023

View reviewed changes

.github/workflows/mvn-verify-check.yml Outdated Show resolved Hide resolved

.github/workflows/mvn-verify-check.yml Outdated Show resolved Hide resolved

YanxuanLiu added 2 commits November 30, 2023 14:49

remove java version matrix

a1bc646

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

add runner os

bc16dfc

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

pxLi changed the title ~~Add cache action to speed up mvn workflow~~ Add cache action to speed up mvn workflow [skip ci] Nov 30, 2023

gerashegalov reviewed Nov 30, 2023

View reviewed changes

YanxuanLiu and others added 5 commits November 30, 2023 16:29

Update .github/workflows/mvn-verify-check.yml

7726775

Co-authored-by: Gera Shegalov <gshegalov@nvidia.com>

Update .github/workflows/mvn-verify-check.yml

035ce8b

Co-authored-by: Gera Shegalov <gshegalov@nvidia.com>

add scala 213, and adjusted sequence of steps

973e947

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

change iteration var name

e098445

Signed-off-by: YanxuanLiu <yanxuanl@nvidia.com>

Merge branch 'NVIDIA:branch-24.02' into cache-deps

6fbac2b

gerashegalov reviewed Nov 30, 2023

View reviewed changes

.github/workflows/mvn-verify-check.yml Outdated Show resolved Hide resolved

Update .github/workflows/mvn-verify-check.yml

1ac4110

Co-authored-by: Gera Shegalov <gshegalov@nvidia.com>

gerashegalov approved these changes Nov 30, 2023

View reviewed changes

pxLi approved these changes Dec 1, 2023

View reviewed changes

YanxuanLiu merged commit 8adc3cb into NVIDIA:branch-24.02 Dec 1, 2023
38 checks passed

gerashegalov mentioned this pull request Dec 2, 2023

Use threads, cache Scala compiler in GH mvn workflow #9925

Merged

sameerz added the build Related to CI / CD or cleanly building label Dec 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cache action to speed up mvn workflow [skip ci] #9674

Add cache action to speed up mvn workflow [skip ci] #9674

YanxuanLiu commented Nov 10, 2023

YanxuanLiu commented Nov 20, 2023

gerashegalov commented Nov 20, 2023

YanxuanLiu commented Nov 21, 2023

YanxuanLiu commented Nov 30, 2023

pxLi commented Nov 30, 2023

gerashegalov left a comment

gerashegalov Nov 30, 2023

YanxuanLiu commented Nov 30, 2023

gerashegalov left a comment •

edited

Loading

YanxuanLiu commented Nov 30, 2023

gerashegalov left a comment

YanxuanLiu commented Nov 30, 2023

pxLi left a comment

Add cache action to speed up mvn workflow [skip ci] #9674

Add cache action to speed up mvn workflow [skip ci] #9674

Conversation

YanxuanLiu commented Nov 10, 2023

YanxuanLiu commented Nov 20, 2023

gerashegalov commented Nov 20, 2023

YanxuanLiu commented Nov 21, 2023

YanxuanLiu commented Nov 30, 2023

pxLi commented Nov 30, 2023

gerashegalov left a comment

Choose a reason for hiding this comment

gerashegalov Nov 30, 2023

Choose a reason for hiding this comment

YanxuanLiu commented Nov 30, 2023

gerashegalov left a comment • edited Loading

Choose a reason for hiding this comment

YanxuanLiu commented Nov 30, 2023

gerashegalov left a comment

Choose a reason for hiding this comment

YanxuanLiu commented Nov 30, 2023

pxLi left a comment

Choose a reason for hiding this comment

gerashegalov left a comment •

edited

Loading