Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use threads, cache Scala compiler in GH mvn workflow #9925

Merged
merged 20 commits into from
Dec 28, 2023
Merged

Use threads, cache Scala compiler in GH mvn workflow #9925

merged 20 commits into from
Dec 28, 2023

Conversation

gerashegalov
Copy link
Collaborator

@gerashegalov gerashegalov commented Dec 1, 2023 •
edited
Loading

  • Save the overhead of setting up the runner by folding the short-lived runner get-shim-versions-from-dist steps into cache-dependencies job
  • Add restore-keys to make sure cache-dependencies does not start on an empty m2, stable versions like for spark artifacts do not change
  • Use de.qaware.maven:go-offline-maven-plugin suggested by @revans2
  • Save scala compiler bridge into a cacheable location

Previous mvn workflow duration:

After this PR:

  • on cache hit: 5min
  • on cache miss: 7min

@gerashegalov
[WIP] Use multi-threaded maven and go-offline-maven-plugin in GH work…
67447e2 
…flow

Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov gerashegalov changed the title [WIP] Use multi-threaded maven and go-offline-maven-plugin in GH work… [WIP] Use threads and go-offline-maven-plugin in GH mvn workflow Dec 1, 2023
@gerashegalov gerashegalov changed the title [WIP] Use threads and go-offline-maven-plugin in GH mvn workflow [WIP DON'T REVIEW] Use threads and go-offline-maven-plugin in GH mvn workflow Dec 1, 2023
@gerashegalov
Common flags; fix local repo
64b241a
@gerashegalov
Empty commit to test cache
7991405 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov
4 x cores
617fee0 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov
2 x cores
65dc511 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov
fold steps from get-shim-version into cache-dependencies
65a4d6d 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov
eliminate get-shim-versions
6343385 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov
cache scala compiler bridge
c3c5a84 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov
restore key, fix command
a0d73d0 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov
replicate maven.repo.local default
b41b304 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov gerashegalov changed the title [WIP DON'T REVIEW] Use threads and go-offline-maven-plugin in GH mvn workflow [WIP DON'T REVIEW] Use threads and and cache Scala compiler in mvn workflow Dec 1, 2023
@gerashegalov
just fetch threads
c889ebc 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov
Merge remote-tracking branch 'origin/branch-24.02' into t1cMvnWorkflow
12978ec
@gerashegalov
go-offline again
5d66399 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov
refactor
eef95fb 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov
common secondaryCacheDir setting
62488e8 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov
PtestFaster
43ac9ff 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov gerashegalov marked this pull request as ready for review December 2, 2023 21:58
@gerashegalov gerashegalov changed the title [WIP DON'T REVIEW] Use threads and and cache Scala compiler in mvn workflow Use threads, cache Scala compiler in GH mvn workflow Dec 2, 2023
@gerashegalov
Copy link
Collaborator Author

build

YanxuanLiu
YanxuanLiu previously approved these changes Dec 4, 2023
View reviewed changes
Copy link
Collaborator

@YanxuanLiu YanxuanLiu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

jlowe
jlowe reviewed Dec 4, 2023
View reviewed changes
integration_tests/run_pyspark_from_build.sh Outdated Show resolved Hide resolved
jdk-profiles/pom.xml Outdated Show resolved Hide resolved
@sameerz sameerz added the build Related to CI / CD or cleanly building label Dec 5, 2023
@gerashegalov gerashegalov self-assigned this Dec 11, 2023
@gerashegalov
review: single for loop
711a723 
Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov
Copy link
Collaborator Author

build

NvTimLiu
NvTimLiu approved these changes Dec 28, 2023
View reviewed changes
Copy link
Collaborator

@NvTimLiu NvTimLiu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gerashegalov gerashegalov merged commit 30e20b9 into NVIDIA:branch-24.02 Dec 28, 2023
38 checks passed
@gerashegalov gerashegalov deleted the t1cMvnWorkflow branch December 28, 2023 16:16
@gerashegalov gerashegalov mentioned this pull request Dec 28, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jlowe jlowe jlowe approved these changes

@NvTimLiu NvTimLiu NvTimLiu approved these changes

@YanxuanLiu YanxuanLiu YanxuanLiu left review comments

@revans2 revans2 Awaiting requested review from revans2 revans2 is a code owner

@tgravescs tgravescs Awaiting requested review from tgravescs tgravescs is a code owner

@GaryShen2008 GaryShen2008 Awaiting requested review from GaryShen2008 GaryShen2008 is a code owner

@pxLi pxLi Awaiting requested review from pxLi

Assignees

@gerashegalov gerashegalov

Labels
build Related to CI / CD or cleanly building
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@gerashegalov @jlowe @NvTimLiu @YanxuanLiu @sameerz