Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI]: use the downstream packaging pipeline for branches/tags #28589

Merged
merged 2 commits into from
Oct 26, 2021
Merged

[CI]: use the downstream packaging pipeline for branches/tags #28589

merged 2 commits into from
Oct 26, 2021

Conversation

v1v
Copy link
Member

@v1v v1v commented Oct 21, 2021 •
edited
Loading

Context

To clarify the existing CI pipelines please see the below details:

  1. Main pipeline is the one that validates every single commit in this repository. It's implemented in the Jenkinsfile and specifically in each <beat>/Jenkinsfile.yml
  2. Packaging pipeline is the one that validates the packages are generated as expected for each beat:
    a) On merge to any branches it gets triggered if the Main pipeline build finished correctly.
    b) On a PR basis if someone comment with /package or use the Jenkins UI.
  3. E2e pipeline is the one that runs the tests defined in https://github.com/elastic/e2e-testing. It gets triggered immediately after the Packaging pipeline finished correctly.
  4. Beats-tester is the one that runs the tests defined in https://github.com/elastic/beats-tester. It gets triggered immediately after the Main pipeline build finished correctly. Or if someone adds the GitHub comment /beats-tester

What does this PR do?

Remove packaging/e2e in the Main pipeline for merge-commits in branches or tags.

Why is it important?

Avoid duplicated tasks since the Packaging pipeline is already doing it.

Eventually, we will tweak the Packaging pipeline to let running individual beats, and then we will be able to remove all the packaging implementation details and e2e defined in the Main pipeline. As we will eventually use the downstream jobs. TBD

But this proposal should reduce the build load in Beats-CI quite a bit!

@v1v v1v added backport-v7.15.0 Automated backport with mergify backport-v7.16.0 Automated backport with mergify labels Oct 21, 2021
@v1v v1v requested review from a team as code owners October 21, 2021 14:17
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 21, 2021
@v1v v1v added automation Team:Automation Label for the Observability productivity team labels Oct 21, 2021
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 21, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Oct 21, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2021-10-25T13:34:31.708+0000

  • Duration: 81 min 9 sec

  • Commit: 3ccda2e

Test stats 🧪

Test Results
Failed 0
Passed 345
Skipped 9
Total 354

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

  • /package : Generate the packages and run the E2E tests.

  • /beats-tester : Run the installation tests with beats-tester.

@v1v v1v requested a review from a team October 25, 2021 07:21
@cachedout
Copy link
Contributor

This makes a lot of sense to me. I think this is a good change and sets us off in the right direction.

I think we should tag @jlind23 and @ruflin here as well, though, just for visibility so there's no confusion about why this changed.

@jlind23
Copy link
Collaborator

jlind23 commented Oct 25, 2021

Thanks for the ping. So packaging/e2e will be triggered only on PR instead of on every commit.
What if i add some more commits after my PR creation?

v1v
v1v commented Oct 25, 2021
View reviewed changes
auditbeat/Jenkinsfile.yml
Comment on lines +105 to +107
changeset: ## when PR contains any of those entries in the changeset
- "^auditbeat/.*"
- "@oss" ## special token regarding the changeset for the oss
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is required to be able to run on a PR basis when there are certain matches in the changeset. The stage-when conditions are explicit, therefore they cannot use the ones in the top-level when condition with the current implementation.

@v1v
Copy link
Member Author

v1v commented Oct 25, 2021

Thanks for the ping. So packaging/e2e will be triggered only on PR instead of on every commit.

No, packaging/e2e meta-stages in the Main pipeline will only run on PR basis instead of on every commit.
Packaging/e2e pipelines will run on every commit as usual.
This change avoids running the packaging/e2e meta-stages in the Main pipeline for every commit in branches, since it was duplicated in a certain way.

What if i add some more commits after my PR creation?

Same behaviour, every commit triggers a new build. Old running builds are aborted if a new commit is pushed.

@v1v
Copy link
Member Author

v1v commented Oct 25, 2021

/test

@v1v v1v self-assigned this Oct 26, 2021
@v1v
Copy link
Member Author

v1v commented Oct 26, 2021

Unrelated e2e test failures -> build with this particular changeset

@v1v v1v merged commit ccc880e into elastic:master Oct 26, 2021
mergify bot pushed a commit that referenced this pull request Oct 26, 2021
@v1v
[CI]: use the downstream packaging pipeline for branches/tags (#28589)
d3660f8 
(cherry picked from commit ccc880e)
@mergify mergify bot mentioned this pull request Oct 26, 2021
Merged
mergify bot pushed a commit that referenced this pull request Oct 26, 2021
@v1v
[CI]: use the downstream packaging pipeline for branches/tags (#28589)
0b44196 
(cherry picked from commit ccc880e)
@mergify mergify bot mentioned this pull request Oct 26, 2021
Merged
@v1v v1v mentioned this pull request Oct 26, 2021
Merged
v1v added a commit to v1v/beats that referenced this pull request Oct 26, 2021
@v1v
Merge remote-tracking branch 'upstream/master' into feature/remove-jo…
fcc08d9 
…urnalbeat-ci

* upstream/master: (49 commits)
  [CI]: use the downstream packaging pipeline for branches/tags (elastic#28589)
  fix: use declarative style for complete variant of the elastic-agent (elastic#28526)
  x-pack/auditbeat/tracing: fix regexp for kprobe description line (elastic#28609)
  docs: Update `api_key` example on elasticsearch output (elastic#28606)
  chore: add build scripts to CODEOWNERS (elastic#28615)
  Osquerybeat: Fix host_processes missing cmdline arguments (elastic#28622)
  Add note about changes to regexp package in Golang (elastic#28616)
  CI: nightly/weekly builds for 7.x targeting 7.16 instead (elastic#28612)
  Osquerybeat: Fix extenstion unable to start on windows (elastic#28598)
  Osquerybeat: Return the query result count with the action response (elastic#28576)
  Agent: Allow custom response properties in the action response (elastic#28575)
  [Heartbeat] Only setuid in elastic-agent image (elastic#28577)
  Fix formatting of `mapStateJSON` and `layerListJSON` in dashboard assets (elastic#28530)
  CI: refactor the run e2e build (elastic#28502)
  Use fsnotify with long windows name-safe changes (elastic#28517)
  Remove unneeded mergify config
  backport: Add 7.16 branch (elastic#28560)
  Add proxy_url support to threatintel module's malwarebazaar fileset (elastic#28533)
  Osquerybeat: Implement host_users, host_groups, host_processes tables as a part of our osquery_extension. (elastic#28434)
  [Heartbeat] Make run_once syntax a boolean (elastic#28548)
  ...
v1v added a commit that referenced this pull request Oct 26, 2021
@mergify @v1v
[CI]: use the downstream packaging pipeline for branches/tags (#28589) (
ea914ea 
#28639)

(cherry picked from commit ccc880e)

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
v1v added a commit that referenced this pull request Oct 26, 2021
@mergify @v1v
[CI]: use the downstream packaging pipeline for branches/tags (#28589) (
8a5bcbb 
#28640)

(cherry picked from commit ccc880e)

Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
Icedroid pushed a commit to Icedroid/beats that referenced this pull request Nov 1, 2021
@v1v
[CI]: use the downstream packaging pipeline for branches/tags (elasti…
661c5f7 
…c#28589)
@v1v v1v mentioned this pull request Jan 27, 2022
Merged
6 tasks
@v1v v1v mentioned this pull request Jan 29, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kuisathaverat kuisathaverat kuisathaverat approved these changes

Assignees

@v1v v1v

Labels
automation backport-v7.15.0 Automated backport with mergify backport-v7.16.0 Automated backport with mergify Team:Automation Label for the Observability productivity team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@v1v @elasticmachine @cachedout @jlind23 @kuisathaverat