Skip to content

Commit

Permalink
[Security Solution] fixes flaky-test-runner Cypress Security Solution…
Browse files Browse the repository at this point in the history
… tests (#134205)

## Summary

Fixes issue with security solution cypress flaky test runner

After introducing [dynamic split for cypress tests in Security Solution](#125986), there was discovered an issue with flaky test runner, which is using parallelism in slightly different manner: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/697#_: tests were evenly split between N jobs, instead of repeating tests N times(as N jobs)

So, for flaky runner, I introducing a new ENV variable that would disable split for tests between parallel jobs and instead would run all test per each job.

Here is link to flaky runner build from this PR, which shows that all test runs per job
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/736
  • Loading branch information
vitaliidm authored Jun 16, 2022
1 parent bf934a9 commit 746e259
Show file tree
Hide file tree
Showing 3 changed files with 17 additions and 2 deletions.
6 changes: 6 additions & 0 deletions .buildkite/pipelines/flaky_tests/pipeline.js
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,12 @@ for (const testSuite of testSuites) {
concurrency: concurrency,
concurrency_group: process.env.UUID,
concurrency_method: 'eager',
env: {
// disable split of test cases between parallel jobs when running them in flaky test runner
// by setting chunks vars to value 1, which means all test will run in one job
CLI_NUMBER: 1,
CLI_COUNT: 1,
},
});
break;
default:
Expand Down
4 changes: 2 additions & 2 deletions .buildkite/scripts/steps/functional/security_solution.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ set -euo pipefail
source .buildkite/scripts/steps/functional/common.sh

export JOB=kibana-security-solution-chrome
export CLI_NUMBER=$((BUILDKITE_PARALLEL_JOB+1))
export CLI_COUNT=$BUILDKITE_PARALLEL_JOB_COUNT
export CLI_NUMBER=${CLI_NUMBER:-$((BUILDKITE_PARALLEL_JOB+1))}
export CLI_COUNT=${CLI_COUNT:-$BUILDKITE_PARALLEL_JOB_COUNT}

echo "--- Security Solution tests (Chrome)"

Expand Down
9 changes: 9 additions & 0 deletions x-pack/test/security_solution_cypress/runner.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,15 @@ export async function SecuritySolutionConfigurableCypressTestRunner(
});
}

/**
* Takes total CI jobs number(totalCiJobs) between which tests will be split and sequential number of the job(ciJobNumber).
* This helper will split file list cypress integrations into chunks, and run integrations in chunk which match ciJobNumber
* If both totalCiJobs === 1 && ciJobNumber === 1, this function will run all existing tests, without splitting them
* @param context FtrProviderContext
* @param {number} totalCiJobs - total number of jobs between which tests will be split
* @param {number} ciJobNumber - number of job
* @returns
*/
export async function SecuritySolutionCypressCliTestRunnerCI(
context: FtrProviderContext,
totalCiJobs: number,
Expand Down

0 comments on commit 746e259

Please sign in to comment.