Add pipeline for flaky test runner job #46740

brianseeders · 2019-09-26T20:46:38Z

How to start the job: https://kibana-ci.elastic.co/job/kibana-pipeline-flaky-test-runner/build

The job will execute the chosen test suite NUMBER_EXECUTIONS times, using 14 parallel processes on one machine. The build only happens once, then the suite is executed many times.

Duplicate test results will show up, so if the same suite errors out 5 times, all 5 instances will show up.

Example: https://kibana-ci.elastic.co/job/kibana-pipeline-flaky-test-runner/4
Example of flaky test failing multiple times: https://kibana-ci.elastic.co/job/kibana-pipeline-flaky-test-runner/4/testReport/

End of console:

11:47:39  There were 8 test suite failures.
11:47:39  [Pipeline] echo
11:47:39  The executions that failed were:
11:47:39  [Pipeline] echo
11:47:39  worker-14-0
11:47:39  worker-4-0
11:47:39  worker-4-1
11:47:39  worker-9-1
11:47:39  worker-2-1
11:47:39  worker-3-1
11:47:39  worker-13-1
11:47:39  worker-5-1
11:47:39  [Pipeline] echo
11:47:39  Please check 'Test Result' and 'Pipeline Steps' pages for more info

I was able to move a lot of Jenkinsfile code to a shared location, but it unfortunately required creating a vars directory in the root of the kibana repo. This is required by Jenkins shared libraries.

elasticmachine · 2019-09-26T22:05:01Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-09-27T16:28:03Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-09-27T18:21:41Z

Pinging @elastic/kibana-operations

elasticmachine · 2019-09-27T19:36:08Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-09-30T20:26:26Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-09-30T21:04:28Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

Jenkinsfile

brianseeders · 2019-10-01T14:43:49Z

retest

elasticmachine · 2019-10-01T16:04:47Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

brianseeders · 2019-10-01T16:19:05Z

vars/kibanaPipeline.groovy

+
+  def buildStatus = buildUtils.getBuildStatus()
+  if (buildStatus != 'SUCCESS' && buildStatus != 'ABORTED') {
+    node('flyweight') {


I did change this function a little bit so that a node isn't allocated unless an e-mail actually needs to go out... Otherwise, the code in this file is just moved over from the original Jenkinsfile

Is this something we could just run on master?

Is there actually a label set up for master? I didn't realize

elasticmachine · 2019-10-01T17:26:53Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-10-01T17:36:47Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

* WIP Jenkinsfile for flaky test runner * Fix syntax * A few more jenkinsfile fixes * A few more fixes * Can't round numbers in Jenkins groovy sandbox apparently * More fixes * Only do build_kbn_tp_sample_panel_action once during flaky testing * Fix path and try setting a different JOB value * Keep track of flaky test failures * Introduce flaky test for testing * Some flaky test pipeline cleanup * Fix a couple of issues with flaky test failure tracking * Update flaky test runner build name/desc with metadata * Revert "Introduce flaky test for testing" This reverts commit 202e9d8. * Try adding a local shared library * Move local library loading logic to kibana pipeline library * Move shared groovy code to shared library * Add missed file * Add ability to specify multiple agents for flaky test runner * Update kibana-pipeline-library version * Fix bug causing early exit for oss ciGroup1

This reverts commit e894255.

… into console-token-iterator * 'console-token-iterator' of github.com:jloleysens/kibana: (184 commits) [functional/services] update webdriver lib and types (elastic#47381) Standardizing IconField implementation across the app (elastic#47196) Move ui/value_suggestions ⇒ NP data plugin (elastic#45762) Remove ui/persisted_log - Part 2 (elastic#47236) Update gulp related packages (elastic#47421) Update dependency idx to ^2.5.6 (elastic#47399) try running fewer jobs in parallel on the same worker (elastic#47403) Update webpack related packages (elastic#47402) Update jsonwebtoken related packages (elastic#47400) Update gulp related packages (major) (elastic#46665) Update dependency prettier to ^1.18.2 (elastic#47340) Update dependency @types/puppeteer to ^1.20.1 (elastic#47339) Update dependency @elastic/elasticsearch to ^7.4.0 (elastic#47338) Update dependency tar-fs to ^1.16.3 (elastic#47341) [Code] Code Integrator Component (elastic#47180) [Canvas][i18n] Sidebar (elastic#46090) Generate uuid in task Manager as Kibana uuid may not yet have been initialised [Code] Embedded Code Snippet Component (elastic#47183) Revert "Add pipeline for flaky test runner job (elastic#46740)" SearchSource: fix docvalue_fields and fields intersection logic (elastic#46724) ...

This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened

* flaky-test-runner-remix: Add missing variable Fix param parsing and add missed change Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)""

* Revert "Revert "Add pipeline for flaky test runner job (#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable

* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable

* Revert "Revert "Add pipeline for flaky test runner job (#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable

* Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable

* Re-add pipeline for flaky test runner job (#48781) * Revert "Revert "Add pipeline for flaky test runner job (#46740)"" This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened * Fix param parsing and add missed change * Add missing variable * Retry git clone up to 8 times before failing a build (#50734) * Flaky test runner improvements/fixes (#49367) * Allocate multiple flaky test runner agents as executions grow * WIP support for deleting kibana install dir during CI * Add setup script for testing scripts * Add REMOVE_KIBANA_INSTALL_DIR=1 to flaky test runner * Change flaky test runner worker processes from 8 to 12 * Add labels to shell scripts in Jenkins (#49657) * Change pipeline timeout from 3 hours to 2 hours (#51098) * Remove flaky pipeline step not relevant for 6.8 * Re-add failed_tests cli * Fix PARALLEL_PIPELINE_WORKER_INDEX var

brianseeders added v8.0.0 Feature:CI Continuous integration labels Sep 26, 2019

This comment has been minimized.

Sign in to view

brianseeders force-pushed the flaky-test-runner branch from 5b4e38e to b86426a Compare September 27, 2019 18:14

brianseeders added v6.8.4 v7.3.2 v7.4.0 v7.5.0 Team:Operations Team label for Operations Team labels Sep 27, 2019

brianseeders requested review from spalger, a team and wayneseymour September 27, 2019 18:21

brianseeders force-pushed the flaky-test-runner branch from b86426a to c0e9c28 Compare September 30, 2019 19:02

brianseeders force-pushed the flaky-test-runner branch from c0e9c28 to 903c898 Compare September 30, 2019 21:03

brianseeders commented Sep 30, 2019

View reviewed changes

Jenkinsfile Outdated Show resolved Hide resolved

snide removed the v7.4.0 label Oct 1, 2019

brianseeders force-pushed the flaky-test-runner branch from 903c898 to 65d26a8 Compare October 1, 2019 16:17

brianseeders commented Oct 1, 2019

View reviewed changes

brianseeders marked this pull request as ready for review October 1, 2019 16:19

brianseeders merged commit e894255 into elastic:master Oct 4, 2019

brianseeders deleted the flaky-test-runner branch October 4, 2019 17:53

brianseeders mentioned this pull request Oct 4, 2019

[7.x] Add pipeline for flaky test runner job (#46740) #47365

Closed

brianseeders mentioned this pull request Oct 4, 2019

[7.4] Add pipeline for flaky test runner job (#46740) #47366

Closed

brianseeders mentioned this pull request Oct 4, 2019

[7.3] Add pipeline for flaky test runner job (#46740) #47367

Closed

brianseeders mentioned this pull request Oct 4, 2019

[6.8] Add pipeline for flaky test runner job (#46740) #47368

Closed

brianseeders added a commit that referenced this pull request Oct 4, 2019

Revert "Add pipeline for flaky test runner job (#46740)"

7d96a13

This reverts commit e894255.

brianseeders added a commit to brianseeders/kibana that referenced this pull request Oct 21, 2019

Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)""

9724af4

This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened

brianseeders mentioned this pull request Oct 21, 2019

Re-add pipeline for flaky test runner job #48781

Merged

brianseeders added a commit to brianseeders/kibana that referenced this pull request Oct 22, 2019

Revert "Revert "Add pipeline for flaky test runner job (elastic#46740)""

33623ad

This reverts commit 7d96a13. Also reconcile changes to Jenkinsfile since original revert happened

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add pipeline for flaky test runner job #46740

Add pipeline for flaky test runner job #46740

brianseeders commented Sep 26, 2019 •

edited

Loading

elasticmachine commented Sep 26, 2019

elasticmachine commented Sep 27, 2019

This comment has been minimized.

elasticmachine commented Sep 27, 2019

elasticmachine commented Sep 27, 2019

elasticmachine commented Sep 30, 2019

elasticmachine commented Sep 30, 2019

brianseeders commented Oct 1, 2019

elasticmachine commented Oct 1, 2019

brianseeders Oct 1, 2019

spalger Oct 3, 2019

brianseeders Oct 3, 2019

elasticmachine commented Oct 1, 2019

elasticmachine commented Oct 1, 2019

Add pipeline for flaky test runner job #46740

Add pipeline for flaky test runner job #46740

Conversation

brianseeders commented Sep 26, 2019 • edited Loading

elasticmachine commented Sep 26, 2019

💚 Build Succeeded

elasticmachine commented Sep 27, 2019

💚 Build Succeeded

This comment has been minimized.

elasticmachine commented Sep 27, 2019

elasticmachine commented Sep 27, 2019

💚 Build Succeeded

elasticmachine commented Sep 30, 2019

💔 Build Failed

elasticmachine commented Sep 30, 2019

💔 Build Failed

brianseeders commented Oct 1, 2019

elasticmachine commented Oct 1, 2019

💚 Build Succeeded

brianseeders Oct 1, 2019

Choose a reason for hiding this comment

spalger Oct 3, 2019

Choose a reason for hiding this comment

brianseeders Oct 3, 2019

Choose a reason for hiding this comment

elasticmachine commented Oct 1, 2019

💚 Build Succeeded

elasticmachine commented Oct 1, 2019

💚 Build Succeeded

brianseeders commented Sep 26, 2019 •

edited

Loading