Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Security Solution] [Elastic AI Assistant] Include acknowledged alert…
…s in the context sent to the LLM (Retrieval Augmented Generation (RAG) for Alerts) (#173121) ## [Security Solution] [Elastic AI Assistant] Include `acknowledged` alerts in the context sent to the LLM (Retrieval Augmented Generation (RAG) for Alerts) This PR updates the query used by [[Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts #172542](#172542) to include alerts with a `kibana.alert.workflow_status` value of `acknowledged`. The query previously only returned alerts with a status of `open`. This change ensures both `open` and `acknowledged` alerts are provided as context to the LLM. ### Updated Anonymization defaults Three fields, detailed below, were added as anonymization defaults because they improve the quality of responses from the LLM when it answers questions about alerts. For example, the LLM can refer to specific alerts by ID when the `_id` field is provided. This PR makes the following additive changes to the Assistant's `Anonymization` defaults: | Field | Allow by default | Anonymize by default | Value add | |--------------------------------|------------------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `_id` | ✅ | ✅ | An anonymized `_id` field enables responses from the LLM to refer to specific documents (but doesn't provide it the actual document IDs). | | `kibana.alert.risk_score` | ✅ | ❌ | The `getOpenAndAcknowledgedAlertsQuery` query sorts alerts by `kibana.alert.risk_score` to return the `n` riskiest alerts. Allowing this field (by default) enables the LLM to include actual alert risk scores in responses. | | `kibana.alert.workflow_status` | ✅ | ❌ | The `getOpenAndAcknowledgedAlertsQuery` query filters alerts by `kibana.alert.workflow_status` to ensure only `open` and `acknowledged` alerts are provided as context to the LLM. Allowing this field (by default) enables the LLM answer questions about workflow status, and echo the workflow status of alerts in responses. | - Clicking the `Reset` button shown in the screenshot below will reset the user's `Anonymization` defaults, such that they include the additive changes in the table above:  ### Updated settings text The text in the settings below was also updated:  ### Desk testing To desk test this change: - Enable the `assistantRagOnAlerts` feature flag described in [#172542](#172542) must be enabled, per the following example: ``` xpack.securitySolution.enableExperimental: ['assistantRagOnAlerts'] ``` - The `Alerts` feature must be enabled in the assistant settings, per the screenshot below:  1) Navigate to Security > Alerts 2) Click the `AI Assistant` button to open the assistant 3) Click the `Settings` gear to open the assistant settings 4) Click the `Anonymization` category 5) Click the `Reset` button, shown in the screenshot below  **Expected results** - `65` fields are allowed by default, per the screenshot above - `12` fields are anonymized by default, per the screenshot above - The `_id` field is allowed by default, per the screenshot above - The `_id` field is anonymized by default, per the screenshot above 6) Type `kibana.alert.risk` in the search box **Expected result** - The `kibana.alert.risk_score` field is allowed by default 7) Type `kibana.alert.workflow` in the search box **Expected result** - The `kibana.alert.workflow_status` field is allowed by default 8) Click `Save` 9) Click the `X` button to clear the conversation 10) Close the assistant 11) Add the following two fields as columns to the Alerts page table: - `kibana.alert.workflow_status` - `_id` 12) Sort the table, first by `kibana.alert.risk_score` from high to low, and then by `@timestamp` from new to old, per the screenshot below:  13) Filter the alerts page to only show `open` and `acknowledged` alerts **Expected result** - The alerts page has custom columns, sorting, and filtering, per the screenshot below:  14) Click the `AI Assistant` button to open the assistant 15) Ask the assistant: ``` What is the workflow status of my alerts? ``` **Expected result** - The assistant will report on the workflow status of alerts, per the example response below: ``` The workflow status for your alerts is currently 'open'. This status was observed on alerts related to processes started by Mimikatz, a known tool used in many cyberattacks, and sequences of processes that are often indicative of malicious activity. The severity of most of these alerts is 'high' or 'critical'. You may want to investigate these alerts further to ensure there's no ongoing threat to your system. ```  16) Close the assistant 17) Change the workflow status of an alert in the Alerts table from `open` to `acknowledged` **Expected result** - The alerts table shows the updated alert, per the screenshot below:  18) Once again, open the assistant 19) Once again, ask the (same) question: ``` What is the workflow status of my alerts? ``` **Expected result** - The response from the assistant makes reference to the alert who's workflow status was changed from `open` to `acknowledged`, per the example response below: ``` Based on the latest information, your alerts mainly show 'open' status, indicating that they have not been resolved yet. Some alerts have been acknowledged. Most of these unaddressed alerts have a critical severity rating and are primarily triggered by a Mimikatz process start and an EQL process sequence. You may want to prioritize these if the severity of the threat they pose is truly high or critical. It's also noteworthy that some alerts have a high severity rating. You should review all of these alerts as soon as possible to ensure your systems are secure. ```  20) Ask the assistant for details about the acknowledged alerts: ``` What are the details of the acknowledged alerts? ``` **Expected result** The assistant for details about the acknowledged alert that, for example, includes the `kibana.alert.risk_score`, per the example response below: ``` In response to your previous question, here are the details of the acknowledged alerts: 1. There is a 'mimikatz process started' alert, which is of 'critical' severity and 'acknowledged' status. It has a high risk score of 99. Its threat tactic is 'Command and Control'. The process involved was 'mimikatz.exe' running with arguments '--fo1'. 2. A 'Threshold rule' alert of 'critical' severity and 'open' status has also been detected with a risk score of 99 and threat tactic 'Collection'. 3. Lastly, there are several 'EQL process sequence' alerts of 'high' severity with 'open' status. These alerts involve execution of various processes including 'mimikatz.exe', 'lsass.exe', and 'notepad.exe'. Risk score for these alerts is 73 and the threat tactic involved is 'Execution'. Please, take appropriate action to address these alerts. ```  21) Ask the assistant for the `_id` of the acknowledged alert: ``` What is the id of the acknowledged alert? ``` **Expected results** - The response from the assistant contains the `_id` of the `acknowledged` alert, per the example response below: ``` The id of the acknowledged alert is 'db9e3dbaf40a37e3b7b95d8015e99c5721b416731e04b9140536675f6e4fd170'. This alert was for a 'mimikatz process started' event with a severity rating of 'critical' and a risk score of 99. The host name associated with this alert is 'Host-terkvbzvtj'. ```  - The `_id` shown in the assistant is the same `_id` of the acknowledged alert on the alerts page, per the screeenshot below:  22) Click the `Show anonymized` toggle in the assistant **Expected result** - The `_id` shown in the latest result is replaced with the actual anonymized value that was sent to the LLM, per the example screenshot below: 
- Loading branch information
1 parent
2f69f1b
commit 0d9c261