Parallelize all jenkins stages #18901
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
botelastic
bot
added
needs_team
jsoriano
force-pushed
the
parallelize-jenkins-stages
branch
from
b904796
to
37ef9a2
Compare
💔 Build Failed
Expand to view the summary
Build stats
Test stats 🧪
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
|
most of the grouped tasks take less than 10 min (most less than 5min), so put then in parallel does not change the time of the build because they will end before others (e.g. Metricbeat takes 30min each). Put all in parallel will run all stages even do on the grouped doesn't, when one fails in the group does not continue with that group. Also, launch everything in parallel consumes more resources (more workers) and can have the side effect of increase the time of the build because there are no workers free to launch. |
|
I guess that we have to find a balance between the fact that we are going to add more stages per beat, and the overhead that parallelizing more can have on the occupation of workers.
My main motivation to do this change is that I have already seen several builds waiting up to 10 or 20 minutes for the last Auditbeat or Heartbeat builds to finish, and we need to add more stages to cover more cases (#17411). So builds for beats that currently only have two or three ~10 minutes stages are going to end up having at least a couple of stages more. This is why I decided to parallelize Auditbeat while I was adding more stages in #18835, and I would expect something similar for other beats.
I am actually not sure if we want this in all cases. It is ok to block the build in a lint stage if the code doesn't even compile, but for example I would like to have the feedback of all stages for a beat to see if a test fails or not in on all platforms.
Yep, I thought about this, but we are also doing many other efforts to have more selective testing. So maybe one thing compensates the other. Also, more workers are going to be launched, but they are going to be occupied during less time, this might help to improve the general throughput.
Do you mean to have one pipeline per beat? Or completely separate beats to different repos as apm-server? I think this would be a bigger discussion to have 🙂 Maybe something else we can try to reduce overhead on workers is to serialize per platform instead of per beat, but this would require a lot of manual assignation of stages to workers. I would go with full parallelization by now, and see how it works. |
Different pipelines, one per beat |
|
Closing this by now, there are other ongoing efforts to refactor the pipeline. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Some related stages are being grouped inside other stages, even if this makes sense logically, it prevents nested stages to be run in parallel, making some of these stages to take a lot of time.
Remove the stages that contain nested stages, and move them directly to the root parallel, so all stages are parallelized.