Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[20843] Detect flaky tests Github workflow #4752

Open
wants to merge 15 commits into
base: master
Choose a base branch
from
Open

[20843] Detect flaky tests Github workflow #4752

wants to merge 15 commits into from

Conversation

elianalf
Copy link
Contributor

@elianalf elianalf commented May 3, 2024 •
edited
Loading

Description

This PRs adds a workflow to detect flaky tests.
It adds a nightly workflow:

  • run the tests and save the results in a junit format
  • download the previous results saved in artifacts
  • detect the flaky tests
  • upload the new test result in the artifact

The detection of flaky tests is executed through a python script:

Contributor Checklist

  • Commit messages follow the project guidelines.
  • The code follows the style guidelines of this project.
  • N/A Tests that thoroughly check the new feature have been added/Regression tests checking the bug and its fix have been added; the added tests pass locally
  • N/A Any new/modified methods have been properly documented using Doxygen.
  • N/A Any new configuration API has an equivalent XML API (with the corresponding XSD extension)
  • Changes are ABI compatible.
  • Changes are API compatible.
  • N/A New feature has been added to the versions.md file (if applicable).
  • N/A New feature has been documented/Current behavior is correctly described in the documentation.
  • N/A Applicable backports have been included in the description.

Reviewer Checklist

  • The PR has a milestone assigned.
  • The title and description correctly express the PR's purpose.
  • Check contributor checklist is correct.
  • Check CI results: changes do not issue any warning.
  • Check CI results: failing tests are unrelated with the changes.

@elianalf elianalf added this to the v3.0.0 milestone May 3, 2024
@EduPonz EduPonz force-pushed the 3.0.x-devel branch 2 times, most recently from 08ee62b to 623db82 Compare May 13, 2024 09:26
@EduPonz EduPonz deleted the branch master May 14, 2024 07:22
@EduPonz EduPonz closed this May 14, 2024
@EduPonz EduPonz reopened this May 14, 2024
@EduPonz EduPonz changed the base branch from 3.0.x-devel to master May 14, 2024 07:26
@elianalf elianalf added needs-review PR that is ready to be reviewed needs rebase labels May 14, 2024
@elianalf elianalf force-pushed the feature/detect_flaky_tests branch from 3f5e774 to adb0412 Compare May 31, 2024 07:24
@elianalf elianalf force-pushed the feature/detect_flaky_tests branch 2 times, most recently from 1f7e650 to c9b82c1 Compare June 4, 2024 06:32
EduPonz
EduPonz suggested changes Jun 11, 2024
View reviewed changes
Copy link

@EduPonz EduPonz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great PoC for this! I've left some comments and suggestions on how to integrate this, that is moving from PoC to a tool to detect flaky tests across platforms and versions

.github/resources/ctest2junit.py Outdated Show resolved Hide resolved
.github/resources/check_flakes.py Outdated Show resolved Hide resolved
.github/resources/check_flakes.py Outdated Show resolved Hide resolved
.github/workflows/nightly-ubuntu-detect-flaky.yml Outdated Show resolved Hide resolved
fastdds.repos Show resolved Hide resolved
.github/resources/check_flakes.py Outdated Show resolved Hide resolved
.github/resources/check_flakes.py Outdated Show resolved Hide resolved
.github/resources/check_flakes.py Outdated Show resolved Hide resolved
.github/workflows/nightly-ubuntu-detect-flaky.yml Outdated Show resolved Hide resolved
.github/workflows/nightly-ubuntu-detect-flaky.yml Outdated Show resolved Hide resolved
@EduPonz EduPonz force-pushed the feature/detect_flaky_tests branch from c9b82c1 to 0ac36c4 Compare June 12, 2024 11:08
@EduPonz EduPonz mentioned this pull request Jun 12, 2024
Draft
7 tasks
@EduPonz EduPonz force-pushed the feature/detect_flaky_tests branch 3 times, most recently from 0187b22 to 9acd339 Compare June 12, 2024 17:06
@EduPonz EduPonz added to-do and removed needs-review PR that is ready to be reviewed labels Jun 12, 2024
@EduPonz EduPonz force-pushed the feature/detect_flaky_tests branch 8 times, most recently from f4e4743 to f4d76f9 Compare June 17, 2024 11:10
@elianalf @EduPonz
Refs #20843: Create workflow for ubuntu to run tests and detect flaki…
47dda09 
…ness

Signed-off-by: elianalf <62831776+elianalf@users.noreply.github.com>
elianalf and others added 13 commits June 17, 2024 14:12
@elianalf @EduPonz
Refs #20843: Add scripts in resources to generate reports and flaky a…
ddda0d8 
…nalysis

Signed-off-by: elianalf <62831776+elianalf@users.noreply.github.com>
@elianalf @EduPonz
Refs #20843: Add pull request as workflow trigger
26f0b69 
Signed-off-by: elianalf <62831776+elianalf@users.noreply.github.com>
@elianalf @EduPonz
Refs #20843: fix ctest args
edd03f4 
Signed-off-by: elianalf <62831776+elianalf@users.noreply.github.com>
@elianalf @EduPonz
Refs #20843: Update for 3.0.x
024afcc 
Signed-off-by: elianalf <62831776+elianalf@users.noreply.github.com>
@elianalf @EduPonz
Refs #20843: Move resource files and change path
7041b71 
Signed-off-by: elianalf <62831776+elianalf@users.noreply.github.com>
@elianalf @EduPonz
Refs #20843: (TO BE REMOVED) change fastdds branch
cadc41a 
Signed-off-by: elianalf <62831776+elianalf@users.noreply.github.com>
@elianalf @EduPonz
Refs #20843: change comment to make it run
9550351 
Signed-off-by: elianalf <62831776+elianalf@users.noreply.github.com>
@elianalf @EduPonz
Refs #20843: Remove print and not used function from main
bf04bf4 
Signed-off-by: elianalf <62831776+elianalf@users.noreply.github.com>
@elianalf @EduPonz
Refs #20843: Update comments and remove unnecessary steps
2e71681 
Signed-off-by: elianalf <62831776+elianalf@users.noreply.github.com>
@elianalf @EduPonz
Refs #20843: Add check the last workflow run if it has an artifact th…
b4bc30a 
…en will get the last available artifact from the previous workflow

Signed-off-by: elianalf <62831776+elianalf@users.noreply.github.com>
@EduPonz
Refs #20843: Refactor check_flakes.py
301f8ac 
Signed-off-by: eduponz <eduardoponz@eprosima.com>
@EduPonz
Refs #20843: Migrate to flakiness_report action
19d99af 
Signed-off-by: eduponz <eduardoponz@eprosima.com>
@EduPonz
Refs #20843: DROP: Enable testing on ubuntu nightly
357bf19 
Signed-off-by: eduponz <eduardoponz@eprosima.com>
@EduPonz EduPonz force-pushed the feature/detect_flaky_tests branch from f4d76f9 to 357bf19 Compare June 17, 2024 12:12
@EduPonz
Refs #20843: Pass flakiness JSON report to junit_summary action
39aa2e4 
Signed-off-by: eduponz <eduardoponz@eprosima.com>
@EduPonz EduPonz force-pushed the feature/detect_flaky_tests branch from 6ebec8e to 39aa2e4 Compare June 17, 2024 13:27
@EduPonz
Copy link

EduPonz commented Jun 17, 2024

Test run with only some tests running:

@EduPonz EduPonz modified the milestones: v3.0.0, v3.0.1 Jul 19, 2024
@rsanchez15 rsanchez15 modified the milestones: v3.0.1, v3.0.2 Sep 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@EduPonz EduPonz EduPonz requested changes

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
to-do
Projects
None yet
Milestone
v3.0.2
Development

Successfully merging this pull request may close these issues.
3 participants
@elianalf @EduPonz @rsanchez15