Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: X-Pack Alerting API Integration Tests.x-pack/test/alerting_api_integration/spaces_only/tests/alerting/bulk_edit·ts - alerting api integration spaces only Alerting bulkEdit should return mapped params after bulk edit #132195

Closed
kibanamachine opened this issue May 13, 2022 · 4 comments · Fixed by #133635
Closed

Failing test: X-Pack Alerting API Integration Tests.x-pack/test/alerting_api_integration/spaces_only/tests/alerting/bulk_edit·ts - alerting api integration spaces only Alerting bulkEdit should return mapped params after bulk edit #132195

kibanamachine opened this issue May 13, 2022 · 4 comments · Fixed by #133635
Assignees
@vitaliidm
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented May 13, 2022 •
edited
Loading

A test failed on a tracked branch

Error: expected [ { message: 'Saved object [alert/3920aa90-d2d2-11ec-b6f5-efaf8297ed5e] conflict',
    rule: 
     { id: '3920aa90-d2d2-11ec-b6f5-efaf8297ed5e',
       name: 'abc' } } ] to have a length of 0 but got 1
    at Assertion.assert (node_modules/@kbn/expect/expect.js:100:11)
    at Assertion.length (node_modules/@kbn/expect/expect.js:374:8)
    at Context.<anonymous> (x-pack/test/alerting_api_integration/spaces_only/tests/alerting/bulk_edit.ts:173:52)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Object.apply (node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)

First failure: CI Build - main
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label May 13, 2022
@botelastic botelastic bot added the needs-team Issues missing a team label label May 13, 2022
@kibanamachine kibanamachine added the Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) label May 13, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops (Team:ResponseOps)

@botelastic botelastic bot removed the needs-team Issues missing a team label label May 13, 2022
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - main

mistic added a commit that referenced this issue May 17, 2022
@mistic
skip flaky suite (#132195)
93ffd78
@mistic
Copy link
Member

mistic commented May 17, 2022

Skipped.

main: 93ffd78

This was referenced Jun 6, 2022
Merged
Merged
@vitaliidm vitaliidm self-assigned this Jun 13, 2022
This was linked to pull requests Jun 15, 2022
[Security Solution] fixes flaky-test-runner Cypress Security Solution tests #134205
Merged
[RAM] fixes flaky tests for alerting bulk_edit HTTP API, adds additional tests for alerting bulk_edit #133635
Merged
@vitaliidm vitaliidm removed a link to a pull request Jun 15, 2022
[Security Solution] fixes flaky-test-runner Cypress Security Solution tests #134205
Merged
vitaliidm added a commit that referenced this issue Jun 15, 2022
@vitaliidm
[RAM] fixes flaky tests for alerting bulk_edit HTTP API, adds additio…
0d0c3d6 
…nal tests for alerting bulk_edit (#133635)

## Summary

Addresses #132195
Follow-up to #132637

- Fixes flaky tests `x-pack/test/alerting_api_integration/spaces_only/tests/alerting/bulk_edit.ts`
- Adds new tests in spaces_only/tests/alerting/bulk_edit.ts for the following operations: `schedule`, `notifyWhen`, `throttle`
- Adds units test in x-pack/plugins/alerting/server/rules_client/lib/apply_bulk_edit_operation.test.ts

### Details to fixing flaky tests

Despite having bulkEdit retry when 409 conflict happens during bulk update of SO, sometimes there was still failing test.
So, all tests in bulk_edit.ts were skipped 93ffd78

The reason, it happens I believe because mocked rule that used for test is enabled, its update after run clashes with `SavedObject.bulkUpdate` method that used inside `RulesClient.bulkEdit`. So, to fix it, I made this mocked rule disabled, which seems like fixed the flakiness

Here is flaky test runner builds with ENABLED rule: 
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/717
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/713

and with DISABLED:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/718
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/714

In both enabled runs there was test failure, but not for disabled.

Another possible way to fix:
use retry in test for `rules.bulkEdit` call and assertion. Let me know if it can more preferable way to fix it

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vitaliidm vitaliidm

Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)
Projects
None yet
Milestone
No milestone
4 participants
@mistic @elasticmachine @kibanamachine @vitaliidm