[SIEM] [Detection Engine] Incorporate large lists to rule execution. #65372

dhurley14 · 2020-05-05T21:04:30Z

Summary

First run at incorporating lists plugin within detection engine rule execution.
I spent some time refactoring the searchAfterBulkCreate function so that I could easily incorporate the list client into the rule execution flow.

To do:

add support for keywords and future list types available in lists plugin (marking this as completed as we are still hammering out the format of the shared json contract for exceptions, but I have a working sample rule with the current contract)
utilize list id from the rule params
add unit tests for new functionality and ~~update current ones~~

Getting started:

Add the following to your kibana.yml

# Enable lists feature
xpack.lists.enabled: true
xpack.lists.listIndex: '.lists-{your-name}'
xpack.lists.listItemIndex: '.items-{your-name}'

then...

First, set the appropriate env var in order to enable exceptions features export ELASTIC_XPACK_SIEM_LISTS_FEATURE=true and export ELASTIC_XPACK_SIEM_EXCEPTIONS_LISTS=true and start kibana
Second, create a list of IP addresses, something that looks like this (DO NOT visit these IP addresses) and import a list of ips from a file named ci-badguys.txt (if you want to name the file something else, you will have to modify the rule json in the next step to point to that). The command should look like this: cd $HOME/kibana/x-pack/plugins/lists/server/scripts && ./import_list_items_by_filename.sh ip ~/ci-badguys.txt
Then, from the detection engine scripts folder (cd kibana/x-pack/plugins/siem/server/lib/detection_engine/scripts) run ./post_rule.sh rules/queries/lists/query_with_list_plugin.json

I would also suggest removing the config flags for lists and posting a standard, non exceptions rule using ./post_rule.sh to confirm backwards compatibility without lists plugin enabled.

Checklist

Delete any items that are not applicable to this PR.

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2020-05-12T02:45:19Z

Pinging @elastic/siem (Team:SIEM)

rylnd · 2020-05-12T17:04:56Z

x-pack/plugins/siem/server/lib/detection_engine/signals/signal_rule_alert_type.test.ts

@@ -68,6 +69,11 @@ describe('rules_notification_alert_type', () => {
    modulesProvider: jest.fn(),
    resultsServiceProvider: jest.fn(),
  };
+  const listMock = {


This is something that would be nice to have as part of the lists plugin itself: https://github.com/elastic/kibana/blob/master/src/core/MIGRATION.md#writing-mocks-for-your-plugin

x-pack/plugins/siem/server/lib/detection_engine/signals/filter_events_with_list.ts

x-pack/plugins/lists/server/index.ts

FrankHassanabad · 2020-05-13T15:04:10Z

x-pack/plugins/siem/server/lib/detection_engine/README.md

+* First, set the appropriate env var in order to enable exceptions features`export ELASTIC_XPACK_SIEM_LISTS_FEATURE=true` and `export ELASTIC_XPACK_SIEM_EXCEPTIONS_LISTS=true` and start kibana
+* Second, import a list of ips from a file called `ci-badguys.txt`. The command should look like this: 
+`cd $HOME/kibana/x-pack/plugins/lists/server/scripts && ./import_list_items_by_filename.sh ip ~/ci-badguys.txt`
+* Then, from the detection engine scripts folder (`cd kibana/x-pack/plugins/siem/server/lib/detection_engine/scripts`) run `./post_rule.sh rules/queries/lists/query_with_list_plugin.json`


x-pack/plugins/siem/server/lib/detection_engine/signals/types.ts

FrankHassanabad · 2020-05-13T15:53:48Z

x-pack/plugins/siem/server/lib/detection_engine/signals/signal_rule_alert_type.ts

+            buildRuleMessage(
+              `[+] Finished indexing ${result.createdSignalsCount} signals into ${outputIndex}`
+            )
+          );


👍 more data for logging

x-pack/plugins/siem/server/lib/detection_engine/signals/signal_rule_alert_type.ts

x-pack/plugins/siem/server/lib/detection_engine/signals/search_after_bulk_create.ts

x-pack/plugins/siem/server/lib/detection_engine/signals/filter_events_with_list.ts

x-pack/plugins/siem/server/plugin.ts

x-pack/plugins/siem/server/lib/detection_engine/signals/filter_events_with_list.test.ts

FrankHassanabad · 2020-05-26T20:16:34Z

x-pack/plugins/siem/server/lib/detection_engine/signals/filter_events_with_list.test.ts

+    const res = await filterEventsAgainstList({
+      logger: mockLogger,
+      listClient: ({
+        getListItemByValues: async () => ([] as unknown) as ListItemArraySchema,


You don't need the extra casting you can do:

getListItemByValues: async () => [],

And it will work

I'm wondering if it's worth putting a listClientMock in the lists plugin? similar to what @rylnd suggested #65372 (comment)

Yes, that is a great idea. Doesn't have to be on this PR unless you want to do it. Otherwise I will do it after this PR within lists.

Okay I'll write up an issue and throw it in the backlog.

x-pack/plugins/siem/server/lib/detection_engine/signals/filter_events_with_list.test.ts

…d in the siem plugin and was preventing the registration of the rule alert type. I removed this but once lists is ready for prime time we should consider adding the null check back

…XPACK_SIEM_LISTS_FEATURE env var to true without enabling the lists plugin

…ogic for empty exceptions list, updates types

…n error when exceptionItem is malformed. This will change in the very near future once the new json format for exception lists is incorporated

…e in siem plugin

… for exception lists

dhurley14 · 2020-05-28T17:48:25Z

jenkins test this

kibanamachine · 2020-05-28T19:28:22Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: d1bf091

History

💚 Build #50173 succeeded 19dccea
💔 Build #50160 failed b19e057
💚 Build #50099 succeeded 444d594
💚 Build #49818 succeeded 8fe45d1
💚 Build #49801 succeeded 2edae11

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…lastic#65372) * introduce lists plugin for use by executor * adds getListClient function on setup * refactors searchAfterBulkCreate to integrate with the lists plugin so we only generate signals from events not in the list * fixes type check issues * fixes unit tests, adds field and other parameters for using lists in executor. * cleaning up types and exports, updates to match new contracts with lists client from master * prior to this commit the refactored while loop was doing more search after loops than it needed to and this fixes two bugs in the list filter function where we were returning the wrong count, and we were not accessing the right field on the event * exception lists are optional * use exceptions list format, this works with given sample query in scripts * updates tests and fixes type issues * updates README doc in detection engine with example for rule with list exception * adds one test and removes commented out code * fix sample rule json from 30s to 5m * fix sample rule json from 30s to 5m * remove unused import * more cleanup * e2e test for prepackaged rules was failing because lists was undefined in the siem plugin and was preventing the registration of the rule alert type. I removed this but once lists is ready for prime time we should consider adding the null check back * can't reuse the same env var since the tests are setting the ELASTIC_XPACK_SIEM_LISTS_FEATURE env var to true without enabling the lists plugin * fixes from pr review, still needs more TLC * exports listspluginsetup type from top-level in lists plugin, fixes logic for empty exceptions list, updates types * utilize type.is to remove as casting, also do null checks and throw an error when exceptionItem is malformed. This will change in the very near future once the new json format for exception lists is incorporated * fix type issues after merging master into branch * update mock * remove bad null check for ml plugin before registering rule alert type in siem plugin * prettier linting * adds test for filter events with list * pr comments * adds logic for included vs excluded and updates tests * update test cases for search after bulk create to default to included for exception lists * filter out non-list exception items from the loop

…tion. (#65372) (#67753) * introduce lists plugin for use by executor * adds getListClient function on setup * refactors searchAfterBulkCreate to integrate with the lists plugin so we only generate signals from events not in the list * fixes type check issues * fixes unit tests, adds field and other parameters for using lists in executor. * cleaning up types and exports, updates to match new contracts with lists client from master * prior to this commit the refactored while loop was doing more search after loops than it needed to and this fixes two bugs in the list filter function where we were returning the wrong count, and we were not accessing the right field on the event * exception lists are optional * use exceptions list format, this works with given sample query in scripts * updates tests and fixes type issues * updates README doc in detection engine with example for rule with list exception * adds one test and removes commented out code * fix sample rule json from 30s to 5m * fix sample rule json from 30s to 5m * remove unused import * more cleanup * e2e test for prepackaged rules was failing because lists was undefined in the siem plugin and was preventing the registration of the rule alert type. I removed this but once lists is ready for prime time we should consider adding the null check back * can't reuse the same env var since the tests are setting the ELASTIC_XPACK_SIEM_LISTS_FEATURE env var to true without enabling the lists plugin * fixes from pr review, still needs more TLC * exports listspluginsetup type from top-level in lists plugin, fixes logic for empty exceptions list, updates types * utilize type.is to remove as casting, also do null checks and throw an error when exceptionItem is malformed. This will change in the very near future once the new json format for exception lists is incorporated * fix type issues after merging master into branch * update mock * remove bad null check for ml plugin before registering rule alert type in siem plugin * prettier linting * adds test for filter events with list * pr comments * adds logic for included vs excluded and updates tests * update test cases for search after bulk create to default to included for exception lists * filter out non-list exception items from the loop

elasticmachine · 2021-09-23T14:34:54Z

Pinging @elastic/security-solution (Team: SecuritySolution)

dhurley14 force-pushed the cp-big-loop branch from a4ba0b3 to a4009c0 Compare May 12, 2020 01:53

dhurley14 marked this pull request as ready for review May 12, 2020 02:43

dhurley14 requested review from a team as code owners May 12, 2020 02:43

dhurley14 self-assigned this May 12, 2020

dhurley14 changed the title ~~[DRAFT] [SIEM] [Detection Engine] Incorporate large lists to rule execution.~~ [SIEM] [Detection Engine] Incorporate large lists to rule execution. May 12, 2020

dhurley14 added release_note:enhancement review Team:SIEM v7.9.0 v8.0.0 labels May 12, 2020

dhurley14 force-pushed the cp-big-loop branch from a3d63e2 to 7319145 Compare May 12, 2020 15:24

rylnd reviewed May 12, 2020

View reviewed changes