Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] New rule fields for prerequisites: Related Integrations, Required Fields, and Setup #2069

Merged
merged 34 commits into from
Aug 25, 2022
Merged
Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
9c51cbc
Add new section to Advanced Settings
joepeeples Jun 13, 2022
c6f922e
Update page's mini-TOC
joepeeples Jun 13, 2022
40d690c
Merge branch 'main' into issue-2015-related-int-required-fields-setup
joepeeples Jun 14, 2022
a2cb3dc
Merge branch 'main' into issue-2015-related-int-required-fields-setup
joepeeples Jun 28, 2022
445ad4b
Merge branch 'main' into issue-2015-related-int-required-fields-setup
joepeeples Jun 29, 2022
0510f73
Merge branch 'main' into issue-2015-related-int-required-fields-setup
joepeeples Jun 30, 2022
3c05f5c
Re-adds RNs for 132409 and 131475
nastasha-solomon Jun 30, 2022
6ad45e3
Revise advanced setting description
joepeeples Jul 6, 2022
586b301
Create new section for related integrations etc.
joepeeples Jul 6, 2022
bb751a5
Add fields to API response examples
joepeeples Jul 6, 2022
93cd003
Update release notes
joepeeples Jul 6, 2022
8a58cd2
Add new section to What's New
joepeeples Jul 7, 2022
85a8409
Merge branch 'main' into issue-2015-related-int-required-fields-setup
joepeeples Jul 7, 2022
29dd38b
Additional revisions
joepeeples Jul 7, 2022
34d6992
Merge branch 'main' into issue-2015-related-int-required-fields-setup
joepeeples Jul 7, 2022
2b0162f
Merge branch 'main' into issue-2015-related-int-required-fields-setup
joepeeples Jul 21, 2022
6cfa822
Merge branch 'main' into issue-2015-related-int-required-fields-setup
joepeeples Aug 2, 2022
94cdec0
Merge branch 'main' into issue-2015-related-int-required-fields-setup
joepeeples Aug 3, 2022
b985848
Merge branch 'main' into issue-2015-related-int-required-fields-setup
joepeeples Aug 12, 2022
648f36c
Merge branch 'main' into issue-2015-related-int-required-fields-setup
joepeeples Aug 19, 2022
d4ba122
Complete main documentation
joepeeples Aug 19, 2022
a05e71a
Update release notes and What's New
joepeeples Aug 19, 2022
fb11b06
Revert changes to whats-new.asciidoc
joepeeples Aug 19, 2022
bf4051a
Merge branch 'main' into issue-2015-related-int-required-fields-setup
joepeeples Aug 22, 2022
f8fad08
Merge branch 'main' into issue-2015-related-int-required-fields-setup
joepeeples Aug 23, 2022
5948aeb
Merge branch 'main' into issue-2015-related-int-required-fields-setup
joepeeples Aug 23, 2022
f76644e
Merge branch 'main' into issue-2015-related-int-required-fields-setup
joepeeples Aug 23, 2022
726763b
Merge branch 'main' into issue-2015-related-int-required-fields-setup
joepeeples Aug 23, 2022
f449fb3
Merge branch 'main' into issue-2015-related-int-required-fields-setup
joepeeples Aug 23, 2022
43ac611
Update per new nav UI
joepeeples Aug 23, 2022
1b0a027
Apply suggestions from Nastasha's review
joepeeples Aug 23, 2022
62135cc
Apply suggestions from Ben's review
joepeeples Aug 24, 2022
8bcf023
Update docs/release-notes/8.3.asciidoc
joepeeples Aug 25, 2022
cc7ab89
Merge branch 'main' into issue-2015-related-int-required-fields-setup
joepeeples Aug 25, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions docs/detections/api/rules/rules-api-bulk-actions.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -392,6 +392,9 @@ For `enable`, `disable`, `delete`, `edit`, and `duplicate` actions, a JSON objec

],
"immutable":false,
"related_integrations": [], <1>
"required_fields": [], <1>
"setup": "", <1>
"type":"machine_learning",
"anomaly_threshold":50,
"machine_learning_job_id":[
Expand Down Expand Up @@ -432,7 +435,7 @@ For `enable`, `disable`, `delete`, `edit`, and `duplicate` actions, a JSON objec
}
--------------------------------------------------

<1> dev:[] `execution_summary` is under development and its schema may change.
<1> dev:[] These fields are under development and their usage or schema may change: `related_integrations`, `required_fields`, `setup`, and `execution_summary`.


For an `export` action, an `.ndjson` file containing exported rules.
Expand Down Expand Up @@ -558,6 +561,9 @@ If processing of any rule fails, a partial error outputs the ID and/or name of t
"version": 5,
"exceptions_list": [],
"immutable": false,
"related_integrations": [], <1>
"required_fields": [], <1>
"setup": "", <1>
"type": "query",
"language": "kuery",
"index": [
Expand Down Expand Up @@ -600,7 +606,7 @@ If processing of any rule fails, a partial error outputs the ID and/or name of t
}
--------------------------------------------------

<1> dev:[] `execution_summary` is under development and its schema may change.
<1> dev:[] These fields are under development and their usage or schema may change: `related_integrations`, `required_fields`, `setup`, and `execution_summary`.

*Example 3, Dry run*

Expand Down
31 changes: 26 additions & 5 deletions docs/detections/api/rules/rules-api-create.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -861,9 +861,14 @@ Example response for a query rule:
],
"throttle": "no_actions",
"query": "process.parent.name:EXCEL.EXE or process.parent.name:MSPUB.EXE or process.parent.name:OUTLOOK.EXE or process.parent.name:POWERPNT.EXE or process.parent.name:VISIO.EXE or process.parent.name:WINWORD.EXE",
"language": "kuery"
"language": "kuery",
"related_integrations": [], <1>
"required_fields": [], <1>
"setup": "" <1>
}
--------------------------------------------------
<1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`.


Example response for a {ml} job rule:

Expand Down Expand Up @@ -911,9 +916,13 @@ Example response for a {ml} job rule:
"status": "going to run",
"status_date": "2020-04-07T14:45:21.685Z",
"anomaly_threshold": 70,
"machine_learning_job_id": "linux_anomalous_network_activity_ecs"
"machine_learning_job_id": "linux_anomalous_network_activity_ecs",
"related_integrations": [], <1>
"required_fields": [], <1>
"setup": "" <1>
}
--------------------------------------------------
<1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`.

Example response for a threshold rule:

Expand Down Expand Up @@ -990,9 +999,13 @@ Example response for a threshold rule:
"threshold": {
"field": "source.ip",
"value": 20
}
},
"related_integrations": [], <1>
"required_fields": [], <1>
"setup": "" <1>
}
--------------------------------------------------
<1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`.

Example response for an EQL rule:

Expand Down Expand Up @@ -1033,9 +1046,13 @@ Example response for an EQL rule:
"actions": [],
"throttle": "no_actions",
"query": "sequence by process.entity_id with maxspan=2h [process where event.type in (\"start\", \"process_started\") and (process.name == \"rundll32.exe\" or process.pe.original_file_name == \"rundll32.exe\") and ((process.args == \"rundll32.exe\" and process.args_count == 1) or (process.args != \"rundll32.exe\" and process.args_count == 0))] [network where event.type == \"connection\" and (process.name == \"rundll32.exe\" or process.pe.original_file_name == \"rundll32.exe\")]",
"language": "eql"
"language": "eql",
"related_integrations": [], <1>
"required_fields": [], <1>
"setup": "" <1>
}
--------------------------------------------------
<1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`.

Example response for an indicator match rule:

Expand Down Expand Up @@ -1104,6 +1121,10 @@ Example response for an indicator match rule:
}
]
}
]
],
"related_integrations": [], <1>
"required_fields": [], <1>
"setup": "" <1>
}
--------------------------------------------------
<1> dev:[] These fields are under development and their usage may change: `related_integrations`, `required_fields`, and `setup`.
5 changes: 4 additions & 1 deletion docs/detections/api/rules/rules-api-find.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,9 @@ Example response:
"Windows"
],
"to": "now",
"related_integrations": [], <1>
"required_fields": [], <1>
"setup": "", <1>
"type": "query",
"threat": [
{
Expand Down Expand Up @@ -136,4 +139,4 @@ Example response:

--------------------------------------------------

<1> dev:[] `execution_summary` is under development and its schema may change.
<1> dev:[] These fields are under development and their usage or schema may change: `related_integrations`, `required_fields`, `setup`, and `execution_summary`.
5 changes: 4 additions & 1 deletion docs/detections/api/rules/rules-api-get.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,9 @@ Example response:
"immutable": false,
"interval": "1h",
"rule_id": "process_started_by_ms_office_user_folder",
"related_integrations": [], <1>
"required_fields": [], <1>
"setup": "", <1>
"language": "kuery",
"output_index": ".siem-signals-siem-test",
"max_signals": 100,
Expand Down Expand Up @@ -111,4 +114,4 @@ Example response:

--------------------------------------------------

<1> dev:[] `execution_summary` is under development and its schema may change.
<1> dev:[] These fields are under development and their usage or schema may change: `related_integrations`, `required_fields`, `setup`, and `execution_summary`.
5 changes: 4 additions & 1 deletion docs/detections/api/rules/rules-api-update.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -509,6 +509,9 @@ Example response:
"child process",
"ms office"
],
"related_integrations": [], <1>
"required_fields": [], <1>
"setup": "", <1>
"type": "query",
"threat": [
{
Expand Down Expand Up @@ -544,4 +547,4 @@ Example response:
}
--------------------------------------------------

<1> dev:[] `execution_summary` is under development and its schema may change.
<1> dev:[] These fields are under development and their usage or schema may change: `related_integrations`, `required_fields`, `setup`, and `execution_summary`.
Binary file modified docs/detections/images/all-rules.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/detections/images/monitor-table.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
23 changes: 21 additions & 2 deletions docs/detections/rules-ui-manage.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ On the Rules page, you can:
* <<load-prebuilt-rules>>
* <<select-all-prebuilt-rules>>
* <<download-prebuilt-rules>>
* <<rule-prerequisites>>
* <<edit-rules-settings>>
* <<manage-rules-ui>>
* <<import-export-rules-ui>>
Expand All @@ -36,7 +37,7 @@ NOTE: Searches for index patterns and MITRE ATT&CK tactics and techniques must m

[float]
[[load-prebuilt-rules]]
=== Load and activate prebuilt Elastic rules
=== Load and activate Elastic prebuilt rules

To load the {es-sec-app}'s <<prebuilt-rules, prebuilt rules>>, go to *Manage* -> *Rules* -> *Load Elastic prebuilt rules and Timeline templates*.

Expand All @@ -63,7 +64,7 @@ You can then modify the duplicated rules and, if required, delete the prebuilt o

[float]
[[download-prebuilt-rules]]
=== Download latest prebuilt Elastic rules
=== Download latest Elastic prebuilt rules

As of {stack} >=7.13.0, you can download the latest version of Elastic prebuilt rules outside of a regular release cycle. This feature ensures you have the latest detection capabilities before upgrading to the latest {stack}.

Expand All @@ -82,6 +83,24 @@ image::images/install-prebuilt-settings.png[]
[role="screenshot"]
image::images/install-prebuilt-rules.png[]

[float]
[[rule-prerequisites]]
=== Confirm rule prerequisites

Many Elastic prebuilt rules are designed to work with specific Elastic integrations and data fields. These prerequisites are identified in the *Related integrations* and *Required fields* fields on a rule's details page (*Manage* -> *Rules*, then click a rule's name). *Related integrations* also displays each integration's installation status and includes links for installing and configuring the listed integrations.

Additionally, the *Setup guide* section provides guidance on setting up the rule's requirements.

[role="screenshot"]
image::images/rule-details-prerequisites.png[Rule details page with Related integrations, Required fields, and Setup guide highlighted]

You can also check rules' related integrations in the *Rules* and *Rule Monitoring* tables. Click the integrations badge to display the related integrations in a popup.

[role="screenshot"]
image::images/rules-table-related-integrations.png[Rules table with related integrations popup,75%]

TIP: You can hide the related integrations badge in the rules tables. Go to *{kib}* -> *Stack Management* -> *Advanced Settings*, then turn off `securitySolution:showRelatedIntegrations`.
joepeeples marked this conversation as resolved.
Show resolved Hide resolved

[float]
[[edit-rules-settings]]
=== Modify existing rules settings
Expand Down
7 changes: 7 additions & 0 deletions docs/getting-started/advanced-setting.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ The advanced settings determine:
* Which IP reputation links appear on <<network-page-overview, IP detail>>
pages
* Whether cross-cluster search (CCS) privilege warnings are displayed
* Whether related integrations are displayed on the Rules page tables

You need `All` privileges for the *Advanced Settings* feature to change these
settings (refer to {kibana-ref}/kibana-privileges.html[Kibana privileges]).
Expand Down Expand Up @@ -161,3 +162,9 @@ Adds a link to \https://www.dnschecker.org on *IP detail* pages:
Each time a detection rule runs using a remote cross-cluster search (CCS) index pattern, it will return a warning saying that the rule may not have the required `read` privileges to the remote index. Because privileges cannot be checked across remote indices, this warning displays even when the rule actually does have `read` privileges to the remote index.

If you've ensured that your detection rules have the required privileges across your remote indices, you can use the `securitySolution:enableCcsWarning` setting to disable this warning and reduce noise.

[discrete]
[[show-related-integrations]]
== Show/hide related integrations in Rules page tables

By default, Elastic prebuilt rules in the *Rules* and *Rule Monitoring* tables include a badge showing how many related integrations have been installed. Turn off `securitySolution:showRelatedIntegrations` to hide this in the rules tables (related integrations will still appear on rule details pages).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe link to advanced settings here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@benironside This text is already on the advanced settings page. Do you mean add a link somewhere else that points TO this section?

2 changes: 2 additions & 0 deletions docs/release-notes/8.3.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,8 @@ On the next scheduled rule execution, the *Last Response* value for the rule wil
* Provides an option to select a severity level for a case ({pull}131626[#131626]).
* Adds the experimental *Alerts* tab to cases, which allows users to inspect attached alerts ({pull}131883[#131883]).
* Adds the *Average time to close* metric to the Cases page ({pull}131909[#131909]).
* Adds new fields to prebuilt detection rules' schemas: `related_integrations`, `required_fields`, and `setup` ({pull}132409[#132409]).
* Adds *Related integrations*, *Required fields*, and *Setup guide* sections to the rule details page, to help users identify, install, and configure a rule's prerequisites. Also adds related integrations badge to the rules table ({pull}131475[#131475]). Content for these new sections is delivered in a prebuilt rules update, independent of {stack} release versioning.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we going to update the 8.3 release notes? Am I understanding correctly that we'll do it only if rules with those fields will be released in one of the 8.3.x Out-Of-Band updates for the Prebuilt Detection Rules package?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@banderror Yes, this section will update the 8.3 release notes (the file name is version-specific).

For 8.4 release notes, we have a separate PR that's currently in progress, though it doesn't yet mention the new fields because they're not yet available in the UI. Once I can confirm that, I'll add the same wording from above to those 8.4 notes.

Basically before we merge anything related to the new fields, I'm waiting to confirm that the latest prebuilt rules package has been published and the new fields are available in both 8.3.x and 8.4.0.

joepeeples marked this conversation as resolved.
Show resolved Hide resolved

[discrete]
[[bug-fixes-8.3.0]]
Expand Down