Getting flaky tests back in shape for reporting #46076

joelgriffith · 2019-09-18T19:29:17Z

This PR re-enables prior skipped flaky tests for reporting. A couple of highlights

Switching tabs to view the downloaded report broke in headless selenium. We fixed that by taking the URL of the report and downloaded it manually.
I’ve added an extra env variable to force a remote debugging port so we can remote debug headless
Added a new method to pause the browsers execution so we can see what’s going on at that exact moment.

elasticmachine · 2019-09-18T21:09:00Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

joelgriffith · 2019-09-18T21:11:18Z

retest

elasticmachine · 2019-09-18T22:50:59Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

joelgriffith · 2019-09-18T23:06:31Z

                 │      Error: expected '{{#each _all}} {{ data.formatted.[0] }} {{ data.raw.[0] }} {{/each}}}}' to equal 'Sep 22, 2015 @ 06:00:00.000,6 1442901600000,6'
                 │       at Assertion.assert (packages/kbn-expect/expect.js:100:11)
                 │       at Assertion.be.Assertion.equal (packages/kbn-expect/expect.js:221:8)
                 │       at Assertion.(anonymous function) [as be] (packages/kbn-expect/expect.js:69:22)
                 │       at Context.be (test/functional/apps/visualize/_tsvb_markdown.ts:82:33)
                 │       at process._tickCallback (internal/process/next_tick.js:68:7)

Looks like failures elsewhere

joelgriffith · 2019-09-19T16:35:37Z

test/functional/services/dashboard/add_panel.js


  return new class DashboardAddPanel {
+    // Dev-helper to pause the browser so we can inspect what's going on
+    // Not meant for usage in CI since it'll stop all execution


Added this so you can pause the browser in places where you need to inspect stuff. Might need a better home than this file (common?)

joelgriffith · 2019-09-19T17:32:44Z

Getting unrelated failures, merged master.

elasticmachine · 2019-09-19T18:09:31Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-09-19T19:15:27Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-09-19T23:06:16Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-09-20T00:12:29Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-09-20T16:49:07Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

joelgriffith · 2019-09-20T20:33:14Z

Some notes:

It appears headless webdriver has troubles changing tabs when one tab is not a site but a binary asset.
I've added a few sugar params for remote debugging headless.
Because of links to styles and js #1, I've instead opted to inject the payload URL into the toast notification via a data attribute. I'm not sure happy about it, however it seems to be the best alternative we have for the time being.
Some of percentage checks have bumped up in size, and I'm not sure as to why. Might be a good idea to use snapshots of some kind instead of this?

joelgriffith · 2019-09-20T20:34:46Z

test/functional/services/remote/webdriver.ts

@@ -79,6 +80,10 @@ async function attemptToCreateCommand(log: ToolingLog, browserType: Browsers) {
          // See: https://chromium.googlesource.com/chromium/src/+/lkgr/headless/README.md
          chromeOptions.push('headless', 'disable-gpu');
        }


I've added this so you can remotely debug (view with devtools) the session as it's running headlessly

joelgriffith · 2019-09-20T20:35:47Z

x-pack/legacy/plugins/reporting/public/hacks/job_completion_notifier.js

@@ -89,6 +89,7 @@ uiModules.get('kibana')
      const downloadReportButton = (
        <EuiButton
          size="s"


This is my hack to get the URL of the report, vs clicking on it and trying to navigate to that Tab in webdriver, which appears to not be working in headless (just freezes or crashes there). A pro is that this speeds up the tests!

This is genius!

joelgriffith · 2019-09-20T20:36:04Z

x-pack/test/functional/page_objects/reporting_page.js

@@ -30,34 +30,6 @@ export function ReportingPageProvider({ getService, getPageObjects }) {
      await browser.setWindowSize(1600, 850);
    }



No longer need theses now that we aren't tab-switching

elasticmachine · 2019-09-20T22:09:51Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-09-20T23:33:35Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

joelgriffith · 2019-09-20T23:38:31Z

retest

joelgriffith · 2019-09-21T01:24:14Z

Unrelated failure, our CI group passed. Merging master and pushing

elasticmachine · 2019-09-21T01:25:29Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-09-21T03:12:47Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

joelgriffith · 2019-09-21T04:35:58Z

restest

tsullivan · 2020-01-14T19:34:20Z

test/functional/services/browser.ts

+     * @return {Promise<void>}
+     */
+    public async pause() {
+      await driver.executeAsyncScript(`(async () => { debugger; return Promise.resolve(); })()`);


When/If this turns out to be useful, can you add a note in some developer docs about how to use this?

Ah! Great reminder, thanks!

joelgriffith · 2020-01-14T20:47:41Z

@elasticmachine merge upstream

…joelgriffith/kibana into chore/reporting-fix-flaky-tests

joelgriffith · 2020-01-14T23:27:52Z

@elasticmachine merge upstream

tsullivan

LGTM!

Reviewed the code

joelgriffith · 2020-01-15T00:49:40Z

@elasticmachine merge upstream

joelgriffith · 2020-01-15T00:49:54Z

One more run, then we'll see how the stress test does

joelgriffith · 2020-01-15T05:24:04Z

@elasticmachine merge upstream

joelgriffith · 2020-01-15T15:56:54Z

@elasticmachine merge upstream

joelgriffith · 2020-01-15T18:01:38Z

@elasticmachine merge upstream

tylersmalley

👍 on the license checker changes.

kibanamachine · 2020-01-15T19:15:07Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 7180be9

History

💚 Build #20586 succeeded 8d826d2
💔 Build #20466 failed e27c04f
💚 Build #20417 succeeded 45984d9
💚 Build #20393 succeeded 382be06
💚 Build #20363 succeeded cd1ce2b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

tsullivan · 2020-01-15T19:27:53Z

@joelgriffith this PR would close #23842 correct?

* Rebasing from master, updating test utils and getting report pdf/png generation * Removing legacy functions, packages and updating README/Licenses * Dropping duplicitive test * Better URL check for lens reporting Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

joelgriffith added the release_note:skip Skip the PR/issue when compiling release notes label Sep 18, 2019

joelgriffith commented Sep 19, 2019

View reviewed changes

joelgriffith commented Sep 20, 2019

View reviewed changes

joelgriffith mentioned this pull request Sep 20, 2019

[Reporting] Surfacing flaky test 2.0 #37895

Closed

joelgriffith marked this pull request as ready for review September 21, 2019 05:15

joelgriffith requested review from spalger and tsullivan September 21, 2019 05:16

joelgriffith added (Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead Team:Stack Services test_xpack_functional labels Sep 21, 2019

Merge branch 'master' into chore/reporting-fix-flaky-tests

c2d34a3

tsullivan reviewed Jan 14, 2020

View reviewed changes

elasticmachine and others added 3 commits January 14, 2020 14:47

Merge branch 'master' into chore/reporting-fix-flaky-tests

b24ab73

Better URL check for lens reporting

ac9dd96

Merge branch 'chore/reporting-fix-flaky-tests' of https://github.com/…

cd1ce2b

…joelgriffith/kibana into chore/reporting-fix-flaky-tests

Merge branch 'master' into chore/reporting-fix-flaky-tests

382be06

tsullivan approved these changes Jan 15, 2020

View reviewed changes

Merge branch 'master' into chore/reporting-fix-flaky-tests

45984d9

Merge branch 'master' into chore/reporting-fix-flaky-tests

e27c04f

Merge branch 'master' into chore/reporting-fix-flaky-tests

8d826d2

Merge branch 'master' into chore/reporting-fix-flaky-tests

7180be9

tylersmalley approved these changes Jan 15, 2020

View reviewed changes

tsullivan mentioned this pull request Jan 15, 2020

Dashboard PDFs not paginated correctly, title on one page and visualizations on the next #23289

Closed

joelgriffith merged commit 5b2e315 into elastic:master Jan 15, 2020

joelgriffith mentioned this pull request Jan 15, 2020

[Reporting] Re-enable Dashboard Preserve Layout Tests #23842

Closed

		@@ -30,34 +30,6 @@ export function ReportingPageProvider({ getService, getPageObjects }) {
		await browser.setWindowSize(1600, 850);
		}

Getting flaky tests back in shape for reporting #46076

Getting flaky tests back in shape for reporting #46076

Conversation

joelgriffith commented Sep 18, 2019 • edited Loading

elasticmachine commented Sep 18, 2019

💚 Build Succeeded

joelgriffith commented Sep 18, 2019

elasticmachine commented Sep 18, 2019

💔 Build Failed

joelgriffith commented Sep 18, 2019 • edited Loading

joelgriffith Sep 19, 2019

Choose a reason for hiding this comment

joelgriffith commented Sep 19, 2019

elasticmachine commented Sep 19, 2019

💔 Build Failed

elasticmachine commented Sep 19, 2019

💔 Build Failed

elasticmachine commented Sep 19, 2019

💔 Build Failed

elasticmachine commented Sep 20, 2019

💔 Build Failed

elasticmachine commented Sep 20, 2019

💔 Build Failed

joelgriffith commented Sep 20, 2019

joelgriffith Sep 20, 2019

Choose a reason for hiding this comment

joelgriffith Sep 20, 2019

Choose a reason for hiding this comment

tsullivan Sep 23, 2019

Choose a reason for hiding this comment

joelgriffith Sep 20, 2019

Choose a reason for hiding this comment

elasticmachine commented Sep 20, 2019

💔 Build Failed

elasticmachine commented Sep 20, 2019

💚 Build Succeeded

joelgriffith commented Sep 20, 2019

joelgriffith commented Sep 21, 2019

elasticmachine commented Sep 21, 2019

💔 Build Failed

elasticmachine commented Sep 21, 2019

💚 Build Succeeded

joelgriffith commented Sep 21, 2019

tsullivan Jan 14, 2020

Choose a reason for hiding this comment

joelgriffith Jan 14, 2020

Choose a reason for hiding this comment

joelgriffith commented Jan 14, 2020

joelgriffith commented Jan 14, 2020

tsullivan left a comment • edited Loading

Choose a reason for hiding this comment

joelgriffith commented Jan 15, 2020

joelgriffith commented Jan 15, 2020

joelgriffith commented Jan 15, 2020

joelgriffith commented Jan 15, 2020

joelgriffith commented Jan 15, 2020

tylersmalley left a comment

Choose a reason for hiding this comment

kibanamachine commented Jan 15, 2020

💚 Build Succeeded

History

tsullivan commented Jan 15, 2020

joelgriffith commented Sep 18, 2019 •

edited

Loading

joelgriffith commented Sep 18, 2019 •

edited

Loading

tsullivan left a comment •

edited

Loading