Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution][Detections] Fixes nondeterminism in integration tests #93350

Merged
merged 1 commit into from
Mar 3, 2021
Merged

[Security Solution][Detections] Fixes nondeterminism in integration tests #93350

merged 1 commit into from
Mar 3, 2021

Conversation

rylnd
Copy link
Contributor

@rylnd rylnd commented Mar 2, 2021

Summary

Addresses #93152.

The threat intel data that we load for these tests was copy/pasted, resulting in all docs having the same @timestamp value.
This resulted in nondeterministic ordering which manifested as test failures.

This fixes the issue by giving a unique timestamp to each indicator document, and then unskips/updates tests according to this new ordering.

Checklist

Delete any items that are not applicable to this PR.

For maintainers

@rylnd
Make tests deterministic by providing unique timestamps
fe6af8e 
The threat intel data that we are loading for these tests was previously
copy/pasted, resulting in all docs having the same @timestamp value.
This resulted in nondeterministic ordering which manifested as test
failures.

This fixes the issue by giving a unique timestamp to each indicator
document, and then unskips/updates tests according to this new ordering.
@rylnd rylnd added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes v7.12.0 Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. labels Mar 2, 2021
@rylnd rylnd self-assigned this Mar 2, 2021
@rylnd rylnd requested a review from a team as a code owner March 2, 2021 23:34
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
triggersActionsUi 1.6MB 1.5MB -23.9KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
triggersActionsUi 104.1KB 104.2KB +82.0B
Unknown metric groups

async chunk count

id before after diff
triggersActionsUi 41 42 +1

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @rylnd

@rylnd rylnd merged commit 3ce614f into elastic:master Mar 3, 2021
@rylnd rylnd deleted the fix_indicator_tests branch March 3, 2021 17:53
@rylnd rylnd mentioned this pull request Mar 3, 2021
Closed
rylnd added a commit to rylnd/kibana that referenced this pull request Mar 3, 2021
@rylnd
Make tests deterministic by providing unique timestamps (elastic#93350)
c71e857 
The threat intel data that we are loading for these tests was previously
copy/pasted, resulting in all docs having the same @timestamp value.
This resulted in nondeterministic ordering which manifested as test
failures.

This fixes the issue by giving a unique timestamp to each indicator
document, and then unskips/updates tests according to this new ordering.
@rylnd rylnd mentioned this pull request Mar 3, 2021
Closed
rylnd added a commit to rylnd/kibana that referenced this pull request Mar 3, 2021
@rylnd
Make tests deterministic by providing unique timestamps (elastic#93350)
8c87366 
The threat intel data that we are loading for these tests was previously
copy/pasted, resulting in all docs having the same @timestamp value.
This resulted in nondeterministic ordering which manifested as test
failures.

This fixes the issue by giving a unique timestamp to each indicator
document, and then unskips/updates tests according to this new ordering.
mistic added a commit that referenced this pull request Mar 3, 2021
@mistic
Revert "Make tests deterministic by providing unique timestamps (#93350…
2d17821 
…)"

This reverts commit 3ce614f.
@mistic
Copy link
Member

mistic commented Mar 3, 2021

I've reverted the previous commit 3ce614f as asked by @rylnd 😃

@spalger spalger mentioned this pull request Mar 3, 2021
Closed
gmmorris added a commit to gmmorris/kibana that referenced this pull request Mar 4, 2021
@gmmorris
Merge branch 'master' into task-manager/docs-monitoring
6a3b8b3 
* master: (48 commits)
  Fix wrong import in data plugin causing 100kB bundle increase (elastic#93448)
  [Fleet] Correctly track install status of an integration (elastic#93464)
  Reviews data frame analytics UI text (elastic#93033)
  Display multiple copyable fields for process.args in resolver node detail panel (elastic#93280)
  [Security Solution][Detections] ML Popover overflow fix (elastic#93525)
  chore(NA): do not use execa on bazel workspace status update script (elastic#93532)
  Bump dependencies (elastic#93511)
  [dev/build_ts_refs] support disabling the ts-refs build completely (elastic#93529)
  [Security Solution] fix data provider cypress test (elastic#93465)
  Fix service map for All environment single service (elastic#93517)
  [Fleet] Fix package version comparaison in the UI (elastic#93498)
  [alerting] adds doc on JSON-expanded action variables and task manager max_workers (elastic#92720)
  [dev/build_ts_refs] ignore type checking failures when building ts refs (elastic#93473)
  [core-new-docs] Adds a dev-doc for core documentation (elastic#92976)
  remove opacity from maps legacy style (elastic#93456)
  [Security Solution][Lists] Escape quotes in list ids and quote the id in KQL query (elastic#93176)
  Revert "Make tests deterministic by providing unique timestamps (elastic#93350)"
  [Discover] Fix link from dashboard saved search to Discover (elastic#92937)
  update public api docs
  App Search - Polishing Analytics Views (elastic#92939)
  ...
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 93350 or prevent reminders by adding the backport:skip label.

@kibanamachine kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Mar 5, 2021
@kibanamachine
Copy link
Contributor

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 93350 or prevent reminders by adding the backport:skip label.

@rylnd rylnd added the backport:skip This commit does not require backporting label Mar 9, 2021
@kibanamachine kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Mar 9, 2021
@rylnd
Copy link
Contributor Author

rylnd commented Mar 9, 2021

skipping backport notifications as this was reverted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FrankHassanabad FrankHassanabad FrankHassanabad approved these changes

Assignees

@rylnd rylnd

Labels
backport:skip This commit does not require backporting release_note:skip Skip the PR/issue when compiling release notes Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v7.12.0 v8.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@rylnd @elasticmachine @kibanamachine @mistic @FrankHassanabad