[DOCS] Update documentation to include the new Rule fields: Related Integrations, Required Fields and Setup

[spong]

Description

With elastic/kibana#131475, elastic/kibana#132409 and elastic/kibana#132667, we've added three new fields (Related Integrations, Required Fields and Setup) to the Rule object to be initially supported in Prebuilt Rules (for 8.3), with support added for Custom Rules in the future.

With this initial implementation, these new fields will be used to show information about related integrations (and their install state) on the Rules Table and Rule Details, and other setup information on the Rule Details page like necessary ingest configuration or what fields are required for this rule to fire. This information will be displayed for any Prebuilt Rules that start including this information in 8.3.

As mentioned, these new fields are only visible with Prebuilt Rules, and so there is limited API support and currently no UI for editing them. If a Prebuilt Rule is duplicated, these fields are emptied (set to '' or []). When a Rule is exported these fields are included (as empty values), and it is possible to edit the ndjson and re-import and then see these fields for the Custom Rule (but still not editable in the UI). This is expected behavior, and is actually a nice and easy way to test.

Here is a sample export you can paste into an test.ndjson file and import to test this feature:

{"id":"6cc39c80-da3a-11ec-9fce-65c1a0bee904","updated_at":"2022-05-23T01:48:23.422Z","updated_by":"elastic","created_at":"2022-05-23T01:48:20.940Z","created_by":"elastic","name":"Testing #131475, don't mind me...","tags":["Elastic","Endpoint Security"],"interval":"5m","enabled":false,"description":"Generates a detection alert each time an Elastic Endpoint Security alert is received. Enabling this rule allows you to immediately begin investigating your Endpoint alerts.","risk_score":47,"severity":"medium","license":"Elastic License v2","output_index":".siem-signals-default","meta":{"from":"5m"},"rule_name_override":"message","timestamp_override":"event.ingested","author":["Elastic"],"false_positives":[],"from":"now-600s","rule_id":"2c66bf23-6ae9-4eb2-859e-446bea181ae9","max_signals":10000,"risk_score_mapping":[{"field":"event.risk_score","operator":"equals","value":""}],"severity_mapping":[{"field":"event.severity","operator":"equals","severity":"low","value":"21"},{"field":"event.severity","operator":"equals","severity":"medium","value":"47"},{"field":"event.severity","operator":"equals","severity":"high","value":"73"},{"field":"event.severity","operator":"equals","severity":"critical","value":"99"}],"threat":[],"to":"now","references":[],"version":7,"exceptions_list":[{"id":"endpoint_list","list_id":"endpoint_list","namespace_type":"agnostic","type":"endpoint"}],"immutable":false,"related_integrations":[{"package":"system","version":"1.6.4"},{"package":"aws","integration":"cloudtrail","version":"1.11.0"}],"required_fields":[{"ecs":true,"name":"event.code","type":"keyword"},{"ecs":true,"name":"message","type":"match_only_text"},{"ecs":false,"name":"winlog.event_data.AttributeLDAPDisplayName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.AttributeValue","type":"keyword"},{"ecs":false,"name":"winlog.event_data.ShareName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.RelativeTargetName","type":"keyword"},{"ecs":false,"name":"winlog.event_data.AccessList","type":"keyword"}],"setup":"## Config\\n\\nThe 'Audit Detailed File Share' audit policy must be configured (Success Failure).\\nSteps to implement the logging policy with with Advanced Audit Configuration:\\n\\n```\\nComputer Configuration > \\nPolicies > \\nWindows Settings > \\nSecurity Settings > \\nAdvanced Audit Policies Configuration > \\nAudit Policies > \\nObject Access > \\nAudit Detailed File Share (Success,Failure)\\n```\\n\\nThe 'Audit Directory Service Changes' audit policy must be configured (Success Failure).\\nSteps to implement the logging policy with with Advanced Audit Configuration:\\n\\n```\\nComputer Configuration > \\nPolicies > \\nWindows Settings > \\nSecurity Settings > \\nAdvanced Audit Policies Configuration > \\nAudit Policies > \\nDS Access > \\nAudit Directory Service Changes (Success,Failure)\\n```\\n","type":"query","language":"kuery","index":["logs-endpoint.alerts-*"],"query":"event.kind:alert and event.module:(endpoint and not endgame)\\n","filters":[],"throttle":"no_actions","actions":[]}
{"exported_count":1,"exported_rules_count":1,"missing_rules":[],"missing_rules_count":0,"exported_exception_list_count":0,"exported_exception_list_item_count":0,"missing_exception_list_item_count":0,"missing_exception_list_items":[],"missing_exception_lists":[],"missing_exception_lists_count":0}

Acceptance Test Criteria

• Add a documentation blurb about this new feature to the Rules Table and Rule Details
• No need to update API documentation till we add support for custom rules
• From 8.3 Detection Rules Area Checklist:
  • Associate prebuilt rules with Related Integrations:
    • New rule fields added to the REST API responses
    • UI for Related Integrations on the Rule Management page
    • UI for Related Integrations on the Rule Details page
    • UI for Required Fields on the Rule Details page
    • UI for Setup Guide on the Rule Details page
    • Kibana Advanced Setting for toggling the Related Integrations feature on the Rules/Monitoring tables

Notes

Please see myself or @banderror for any clarification here 🙂

[banderror]

No need to update API documentation till we add support for custom rules

@spong Alternatively, we could document these fields as dev as we did for execution_summary in #1900.

[spong]

Note, we'll also be adding a Kibana Advanced Setting for toggling the Related Integrations feature on the Rules/Monitoring tables so folks that are using their own custom ingest pipelines and aren't leveraging integrations can hide this from the UI. New setting is securitySolution:showRelatedIntegrations:

[joepeeples]

Note, we'll also be adding a Kibana Advanced Setting for toggling the Related Integrations feature on the Rules/Monitoring tables so folks that are using their own custom ingest pipelines and aren't leveraging integrations can hide this from the UI. New setting is securitySolution:showRelatedIntegrations:

@spong Thanks for the info! The linked PR has both 8.3.0 and 8.4.0 labels -- in which version is the Advanced Settings toggle being released? I don't see it in a recent 8.3.0-SNAPSHOT build (from Jun 2).

[spong]

@joepeeples -- shooting to get that PR merged early next week before BC3, so should start seeing it in the next build

[spong]

Just a heads up @joepeeples, this did not make it in for the BC today, so will have to wait till next week for these changes.

[joepeeples]

No need to update API documentation till we add support for custom rules

@spong Alternatively, we could document these fields as dev as we did for execution_summary in #1900.

@banderror @spong A few questions:

• Just to clarify, are we talking about using the dev tag only in the context of the fields being included in API responses? Or are the fields as they appear in the UI also considered dev at this time?

• When do we expect to start seeing these fields populated for prebuilt rules? I just spun up today's 8.3 snapshot and there doesn't seem to be any content -- the only way I can get the fields to appear is with a custom rule, using the test import/export ndjson mentioned above.

• What exactly changes the number in the indicator in the rules table? Is it # of integrations installed, or enabled/configured? Or something else?
  

[spong]

Just to clarify, are we talking about using the dev tag only in the context of the fields being included in API responses? Or are the fields as they appear in the UI also considered dev at this time?

Just for the API responses -- though since these fields are being exposed on an already public API we technically can't really change them going forward, but we could use that dev footnote to mention that these fields will start showing up in responses, but can't be updated via the public API just yet.

When do we expect to start seeing these fields populated for prebuilt rules? I just spun up today's 8.3 snapshot and there doesn't seem to be any content -- the only way I can get the fields to appear is with a custom rule, using the test import/export ndjson mentioned above.

The protections team is still working to get the prebuilt rules updated with these new fields, but they haven't pushed anything just yet. There's a chance they don't make it in 8.3 proper and will be published as an out-of-band update, but they've technically got one more week to try and get them in for the last BC. Till then, the import/export ndjsn will be the only way to test.

What exactly changes the number in the indicator in the rules table? Is it # of integrations installed, or enabled/configured? Or something else?

It'll be # of integrations enabled/configured, though for integrations that don't have additional inputs (e.g. System), installed is the same as configured and so will count. So if you install the System integration that badge should say 1/2 integrations since the System integration doesn't have additional inputs as they call them in fleet. Where-as if you installed the AWS integration, it'll still stay as 1/2 integrations until you go in and actually configure Cloudtrail on an integration policy (which can also be done by installing the AWS Cloudtrail integration from the UI).

This gets a little confusing, so let me know if this doesn't make sense and we can pair and go over these details. 🙂

[joepeeples]

Thanks @spong for the answers, I think it all makes sense now!

[spong]

Note @joepeeples -- the required privileges for this feature are outlined in this PR description elastic/kibana#134299. Essentially, in order to have access to integrations' data in Fleet, a user needs either of these 3 Kibana privileges:

• Integrations: Read or All
• Fleet: All
• Saved Objects Management: Read or All

If all of them are None, the Related Integrations feature will only show the integration name/link, with no 'installed state' badge.

[banderror]

@joepeeples Perhaps one more thing to document would be RBAC requirements for getting integrations data from Fleet. Users will need to have certain privileges to be able to see related integration's installation status, version mismatch, and correct link to the Integration Details page in Fleet. More details:

• in this PR
• and this test plan -- let me know if you have access and can find the RBAC section there

[banderror]

@spong 👍 we're working in sync 😀

[banderror]

Just to write up what we discussed today, we might not need to document any RBAC requirements if we push this PR before the last BC. In this case, we just won't have any special RBAC requirements.

[joepeeples]

This feature has been held back from the 8.3.0 release, but keeping the v8.3.0 label for now in case the feature becomes available soon after the release.

[joepeeples]

Docs updated in #2069 and associated PRs mentioned there.