Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve failure reporting output for functional tests #70954

Merged

Conversation

dej611
Copy link
Contributor

@dej611 dej611 commented Jul 7, 2020

Summary

This PR contains some improvement in case of failures during the functional tests run: in particular when multiple tests are failing the reporter is only showing the number of failures, and to find which test has failed is a process of scrolling the log and look for the "fail" red word.
The alternative method was to use the --bail flag to stop at the first failure.

The improvement in this PR improves the investigation process by integrating the full error message into the custom reporter from mocha and show a short summary of the failures details at the end of the run.
This should simplify the process of debugging failing tests.

  • 🐛 Fix log hijack trick to capture mocha error message and format it properly
  • 🐛 Fix partial shadowing of test titles for deep nesting suites
  • ✨ Add new failures summary at the end of the execution of functional tests
  • ✅ Not sure yet how to test the output here

Before

Error inline: mocha was printing the Error breaking the format in the logger as shown below.

Screenshot 2020-07-07 at 15 10 23

Final summary: it shows how many test failed, but not which ones.

Screenshot 2020-07-07 at 15 11 06

After

Error inline: it shows the Error with the right format.

Screenshot 2020-07-07 at 16 36 41

Final summary: after the numbers also details on the name of the failed tests are shown.

Screenshot 2020-07-07 at 16 37 08

Checklist

Delete any items that are not applicable to this PR.

@dej611
Copy link
Contributor Author

dej611 commented Jul 7, 2020

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

kibanamachine commented Jul 7, 2020

💚 Build Succeeded

Build metrics

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@spalger
Copy link
Contributor

spalger commented Jul 8, 2020

@dej611 is this ready for review? It's still in draft mode but I'm excited to get it. If you're ready click that "ready for review" button :)

@dej611 dej611 marked this pull request as ready for review July 8, 2020 18:36
@dej611 dej611 requested a review from a team as a code owner July 8, 2020 18:36
@dej611 dej611 added release_note:skip Skip the PR/issue when compiling release notes v7.9.0 v8.0.0 labels Jul 8, 2020
Copy link
Contributor

@spalger spalger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dej611 dej611 merged commit dcb3820 into elastic:master Jul 9, 2020
@dej611 dej611 deleted the functional-tests/better-failing-reports branch July 9, 2020 09:00
dej611 added a commit to dej611/kibana that referenced this pull request Jul 9, 2020
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
dej611 added a commit that referenced this pull request Jul 9, 2020
…71210)

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:Functional Testing release_note:skip Skip the PR/issue when compiling release notes v7.9.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants