[SIEM] [Detection Engine] Search signals index #52661

dhurley14 · 2019-12-10T17:55:44Z

Summary

Adds route for searching signals index and sample usage script. Also updates naming for signals status schemas.

There is a sample usage query in scripts/signals/query_signals.sh in addition to an aggs script in scripts/signals/aggs_signals.sh. Ensure you have a signal doc in the signals index. There is a sample doc that can be indexed via the signals/put_signal_doc.sh script.

Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

~~- [ ] This was checked for cross-browser compatibility, including a check against IE11~~
~~- [ ] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support~~

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
~~- [ ] This was checked for keyboard-only and screenreader accessibility~~

For maintainers

This was checked for breaking API changes and was labeled appropriately
This includes a feature addition or change that requires a release note and was labeled appropriately

elasticmachine · 2019-12-10T17:56:04Z

Pinging @elastic/siem (Team:SIEM)

FrankHassanabad · 2019-12-10T19:38:23Z

...k/legacy/plugins/siem/server/lib/detection_engine/routes/signals/query_signals_route.test.ts

+      expect(statusCode).toBe(400);
+    });
+  });
+});


Awesome tests! I should back fill my to have at least the basics like this. Really appreciate it!

FrankHassanabad · 2019-12-10T19:41:13Z

x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/__mocks__/request_responses.ts

  query: { range: { '@timestamp': { gte: 'now-2M', lte: 'now/M' } } },
  status: 'closed',
 });

-export const setStatusSignalMissingIdsAndQueryPayload = (): Partial<SignalsRestParams> => ({
+export const typicalSignalsQuery = (): Partial<SignalsQueryRestParams> => ({
+  search_query: { query: { match_all: {} } },


I'd just drop the search_query and use { query since you already have the URL for search. Would feel more natural to the ES querying of things.

FrankHassanabad · 2019-12-10T19:45:05Z

...legacy/plugins/siem/server/lib/detection_engine/routes/schemas/query_signals_index_schema.ts

+
+export const querySignalsSchema = Joi.object({
+  search_query: Joi.object().required(),
+});


If you dropped the search_query, you could change this to just be that query is required? Unless you were planning more than just query to be accepted.

If you wanted aggregations too you could then do a "at least one of these is required" with the top level keys being:

query aggregations

Just kind of asking since I have to do these type of endpoints myself for rules

Ah yeah I forgot about aggregations. Good call.

Yeah, ++ on aggregations -- we'll want to use this for the Signals histogram.

FrankHassanabad · 2019-12-10T19:50:00Z

x-pack/legacy/plugins/siem/server/lib/detection_engine/scripts/signals/query_signals.sh

+./check_env_variables.sh
+
+# Example: ./signals/query_signals.sh
+  curl -v -k \


Woa, I don't think you want the -v here for the example. That adds a lot of extra stuff! :-)

very true - was debugging and forgot to take it out. Thanks 👍

FrankHassanabad · 2019-12-10T23:25:28Z

x-pack/legacy/plugins/siem/server/lib/detection_engine/scripts/signals/aggs_signals.sh

+set -e
+./check_env_variables.sh
+
+# Example: ./signals/query_signals.sh


You mean signals/aggs_signals.sh on this line

FrankHassanabad · 2019-12-10T23:25:48Z

x-pack/legacy/plugins/siem/server/lib/detection_engine/routes/signals/query_signals_route.ts

+        payload: querySignalsSchema,
+      },
+    },
+    async handler(request: SignalsQueryRequest, _headers) {


Optional, if you're not using _headers you can just do: async handler(request: SignalsQueryRequest)

FrankHassanabad · 2019-12-10T23:26:12Z

...k/legacy/plugins/siem/server/lib/detection_engine/routes/signals/open_close_signals_route.ts

@@ -24,7 +24,7 @@ export const setSignalsStatusRouteDef = (server: ServerFacade): Hapi.ServerRoute
        payload: setSignalsStatusSchema,
      },
    },
-    async handler(request: SignalsRequest, headers) {
+    async handler(request: SignalsStatusRequest, _headers) {


Optional, if you're not using headers you can just do: async handler(request: SignalsStatusRequest)

FrankHassanabad

Found one comment in a sh file that was a copy-pasta and 2 optionals.

Outside of that, I checked this out, tested it, and everything looks great!

LGTM!

…e names

…query, adds aggs to signals search api, updates route and validation tests

… aggs script

elasticmachine · 2019-12-11T15:57:36Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 7ec68f9

History

💔 Build #14696 failed 907115d
💚 Build #14594 succeeded 6153f6e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* adds route for querying signals index, also updates signal status type names * first pass at happy path tests * fixes stuff after rebase with master * utilizes removes search_query from payload and replaces it with just query, adds aggs to signals search api, updates route and validation tests * removes _headers parameter from route handler and updates comment for aggs script

dhurley14 added release_note:enhancement v8.0.0 v7.6.0 labels Dec 10, 2019

dhurley14 self-assigned this Dec 10, 2019

dhurley14 added the Team:SIEM label Dec 10, 2019

FrankHassanabad reviewed Dec 10, 2019

View reviewed changes

FrankHassanabad approved these changes Dec 10, 2019

View reviewed changes

dhurley14 added 5 commits December 11, 2019 09:45

adds route for querying signals index, also updates signal status typ…

b0fdbe0

…e names

first pass at happy path tests

e875fc6

fixes stuff after rebase with master

bd53f0d

utilizes removes search_query from payload and replaces it with just …

aa8a08b

…query, adds aggs to signals search api, updates route and validation tests

removes _headers parameter from route handler and updates comment for…

7ec68f9

… aggs script

dhurley14 force-pushed the search-signals-api branch from 907115d to 7ec68f9 Compare December 11, 2019 14:45

dhurley14 merged commit a12d855 into elastic:master Dec 11, 2019

dhurley14 deleted the search-signals-api branch December 11, 2019 16:09

dhurley14 mentioned this pull request Dec 11, 2019

[7.x] [SIEM] [Detection Engine] Search signals index (#52661) #52772

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SIEM] [Detection Engine] Search signals index #52661

[SIEM] [Detection Engine] Search signals index #52661

dhurley14 commented Dec 10, 2019 •

edited

Loading

elasticmachine commented Dec 10, 2019

FrankHassanabad Dec 10, 2019

FrankHassanabad Dec 10, 2019

FrankHassanabad Dec 10, 2019

FrankHassanabad Dec 10, 2019

dhurley14 Dec 10, 2019

spong Dec 10, 2019

FrankHassanabad Dec 10, 2019

dhurley14 Dec 10, 2019

FrankHassanabad Dec 10, 2019

FrankHassanabad Dec 10, 2019 •

edited

Loading

FrankHassanabad Dec 10, 2019

FrankHassanabad left a comment

elasticmachine commented Dec 11, 2019

[SIEM] [Detection Engine] Search signals index #52661

[SIEM] [Detection Engine] Search signals index #52661

Conversation

dhurley14 commented Dec 10, 2019 • edited Loading

Summary

Checklist

For maintainers

elasticmachine commented Dec 10, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FrankHassanabad Dec 10, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FrankHassanabad left a comment

Choose a reason for hiding this comment

elasticmachine commented Dec 11, 2019

💚 Build Succeeded

History

dhurley14 commented Dec 10, 2019 •

edited

Loading

FrankHassanabad Dec 10, 2019 •

edited

Loading