[Security Solution] fixes flaky-test-runner Cypress Security Solution tests

[vitaliidm]

Summary

Fixes issue with security solution cypress flaky test runner

After introducing dynamic split for cypress tests in Security Solution, there was discovered an issue with flaky test runner, which is using parallelism in slightly different manner: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/697#_: tests were evenly split between N jobs, instead of repeating tests N times(as N jobs)

So, for flaky runner, I introducing a new ENV variable that would disable split for tests between parallel jobs and instead would run all test per each job.

Here is link to flaky runner build from this PR, which shows that all test runs per job
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/736

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[cavokz]

LGTM

[vitaliidm]

@cavokz , I had to revert changes you proposed

variables CLI_NUMBER, CLI_NUMBER are getting rewritten inside .buildkite/scripts/steps/functional/${CYPRESS_SUITE}.sh script
as these variables getting populated by from buildkite variables

https://github.com/elastic/kibana/blob/main/.buildkite/scripts/steps/functional/security_solution.sh#L8-L9

Let me know if you are fine with my initial solution or would like to approach it in other way

[cavokz]

Let me know if you are fine with my initial solution or would like to approach it in other way

You can surely go with the initial approach but this is the start of a flags war, where some override others. I'd consider why this is happening before pushing the button.

[vitaliidm]

You can surely go with the initial approach but this is the start of a flags war, where some override others. I'd consider why this is happening before pushing the button.

My understanding, It happens because buildkite context ENV variables in cypress bash script become available only in pipeline step runtime. So, only using these runtime variables in configuration, won't have any effect. That's why we need new ENV value that would be able to disable split tests from pipeline steps config

[cavokz]

variables CLI_NUMBER, CLI_NUMBER are getting rewritten inside .buildkite/scripts/steps/functional/${CYPRESS_SUITE}.sh script as these variables getting populated by from buildkite variables

https://github.com/elastic/kibana/blob/main/.buildkite/scripts/steps/functional/security_solution.sh#L8-L9

My apologies, I'm reading that file only now. What I prefer then is, in that file, populate CLI_NUMBER and CLI_COUNT only if they are not already defined. Then set them to 1, as in my previous request.

[cavokz]

Yes, exactly. Thanks.

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: b3bcf8c
• Storybooks Preview
• Webpack Bundle Analyzer

Metrics [docs]

✅ unchanged

History

• 💚 Build #50791 succeeded c84ff8b
• 💚 Build #50705 succeeded a483006
• 💚 Build #50487 succeeded 78ff4f1

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @vitaliidm