[Security Solution] - Security solution ES|QL configurable via advanced setting #181616

michaelolo24 · 2024-04-24T17:37:57Z

Summary

This PR links the ESQL functionality in security solution to the discover:enableESQL advanced setting. The advanced setting will only be present in ESS, but not serverless

The way this should work to maintain parity with the rest of Kibana such as discover and stack rules:

By default ES|QL will be enabled across all Kibana
When the ES|QL advanced setting is disabled:
- Timeline
  - ES|QL tab should not be accessible on any newly created timelines
  - Existing Timelines with an ES|QL query should still have the tab accessible
- Rules
  - New ES|QL rule should not be available to be created in the Rule Creation workflow
  - Existing ES|QL rules should still run and be able to be edited

Timeline Demo Video:

Screen.Recording.2024-04-29.at.3.52.51.PM.mov

Rules Demo Video:

Screen.Recording.2024-04-29.at.3.54.24.PM.mov

semd

Neat! tested locally, the Timeline all works as expected.

We will only need to add the setting to our Security project settings list for serverless (code).

Nevertheless, we will have to be careful and do all the testing the moment we add this setting in serverless, since it is defaulted to true, just by adding it to this list we'll be enabling all ESQL features by default.

vitaliidm · 2024-04-25T09:17:38Z

Thanks for improving ES|QL configuration in Kibana!

We register ES|QL rule type in plugin

https://github.com/elastic/kibana/blob/8.14/x-pack/plugins/security_solution/server/plugin.ts#L322-L324
depends on the settings

    if (config.settings.ESQLEnabled && !experimentalFeatures.esqlRulesDisabled) {
      plugins.alerting.registerType(securityRuleTypeWrapper(createEsqlAlertType(ruleOptions)));
    }

I think this new advanced setting should be accounted here too.

semd · 2024-04-25T11:16:39Z

@michaelolo24 there are a couple of places on the server side that will need to be updated:

kibana/x-pack/plugins/security_solution/server/plugin.ts

Lines 323 to 325 in 60c6cdb

    
           if (config.settings.ESQLEnabled && !experimentalFeatures.esqlRulesDisabled) { 
        
             plugins.alerting.registerType(securityRuleTypeWrapper(createEsqlAlertType(ruleOptions))); 
        
           }

kibana/x-pack/plugins/security_solution/server/lib/detection_engine/rule_preview/api/preview_rules/route.ts

Lines 433 to 435 in 60c6cdb

    
           if (!config.settings.ESQLEnabled || config.experimentalFeatures.esqlRulesDisabled) { 
        
             throw Error('ES|QL rule type is not supported'); 
        
           }

semd · 2024-04-25T11:18:26Z

x-pack/plugins/security_solution/common/config_settings.ts

@@ -22,7 +18,6 @@ export interface ConfigSettings {
 */
 export const defaultSettings: ConfigSettings = Object.freeze({
  ILMEnabled: true,
-  ESQLEnabled: true,


Remember to remove it from the serverless config

kibana/config/serverless.security.yml

Line 26 in 60c6cdb

ESQLEnabled: false, # ES|QL disabled, not supported by serverless Elasticsearch

michaelolo24 · 2024-04-25T13:41:59Z

Thanks @semd and @vitaliidm for the comments and letting me know about some spots left to update! I'll make the changes that you asked. I would like QA to take a look at this branch before it's merged, but will follow up with product about timelines for getting this in. Thanks again!

michaelolo24 · 2024-04-25T15:36:56Z

Thanks for improving ES|QL configuration in Kibana!

We register ES|QL rule type in plugin

https://github.com/elastic/kibana/blob/8.14/x-pack/plugins/security_solution/server/plugin.ts#L322-L324 depends on the settings
    if (config.settings.ESQLEnabled && !experimentalFeatures.esqlRulesDisabled) {
      plugins.alerting.registerType(securityRuleTypeWrapper(createEsqlAlertType(ruleOptions)));
    }
I think this new advanced setting should be accounted here too.

@vitaliidm afaik we don't have access to the uiSettings here as they haven't actually been initialized yet. Since ES|QL will be default enabled, I just removed the original config check, and this will only be disable-able via the feature flag.

michaelolo24 · 2024-04-25T15:47:19Z

@stratoula do you have any problems with security making use of discover:enableESQL. We're still considering whether or not to have a security:enableESQL setting, but it might be worth having a global ESQL one in general?

vitaliidm · 2024-04-25T15:54:10Z

@vitaliidm afaik we don't have access to the uiSettings here as they haven't actually been initialized yet. Since ES|QL will be default enabled, I just removed the original config check, and this will only be disable-able via the feature flag.

In that case new option should not affect rule type at all.
Otherwise it will just hide option to create new ES|QL rule from UI. But existing rules will be shown and executing.
Moreover, user even would be able to edit them - but not create.

stratoula · 2024-04-29T19:50:22Z

@michaelolo24 I am so sorry i missed the ping. I am def fine of using it. We decided to not rename it (as changing this will mean that the customers that have already set it will need to re-set it). But we changed the description so it doesnt mention only Discover now.

I think it would better to not create your own setting, I don't think we need so many settings for ESQL and will confuse the users even more. This setting doesnt provide any security against running ESQL queries, it just makes it harder from the UI.

With that being said let me bring again with my team the discussion on renaming and moving to a general section.

michaelolo24 · 2024-04-29T19:56:32Z

@michaelolo24 I am so sorry i missed the ping. I am def fine of using it. We decided to not rename it (as changing this will mean that the customers that have already set it will need to re-set it). But we changed the description so it doesnt mention only Discover now.

I think it would better to not create your own setting, I don't think we need so many settings for ESQL and will confuse the users even more. This setting doesnt provide any security against running ESQL queries, it just makes it harder from the UI.

With that being said let me bring again with my team the discussion on renaming and moving to a general section.

Great, thanks so much @stratoula . Makes sense. Yep, will wait on the final confirmation 😄

michaelolo24 · 2024-04-29T20:04:02Z

@vitaliidm afaik we don't have access to the uiSettings here as they haven't actually been initialized yet. Since ES|QL will be default enabled, I just removed the original config check, and this will only be disable-able via the feature flag.

In that case new option should not affect rule type at all. Otherwise it will just hide option to create new ES|QL rule from UI. But existing rules will be shown and executing. Moreover, user even would be able to edit them - but not create.

@vitaliidm and @semd after speaking with @paulewing we made the decision to align with how the rest of the stack currently works. When the advanced setting for ESQL is disabled, users are still able to edit old saved searches as well as any existing esql stack rules. They just are unable to create any new esql rules or new esql saved searches when the advanced setting is disabled. I had to make a minor change from the last update to get that logic to work for timeline, but the PR should match the expected scenario above.

stratoula · 2024-04-30T07:09:38Z

@michaelolo24 we decided to move to General section and rename to enableESQL for usage across kibana #182074

I will try to ask for fast review and also backport to 8.14 when CI is green

vitaliidm · 2024-04-30T08:34:38Z

. When the advanced setting for ESQL is disabled, users are still able to edit old saved searches as well as any existing esql stack rules. They just are unable to create any new esql rules or new esql saved searches when the advanced setting is disabled. I had to make a minor change from the last update to get that logic to work for timeline, but the PR should match the expected scenario above.

@michaelolo24, @paulewing users still would be able to create ES|QL rule through API. Is this fine?

kibanamachine · 2024-05-02T17:07:10Z

💚 All backports created successfully

Status	Branch	Result
✅	8.14

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…ed setting (elastic#181616) ## Summary This PR links the ESQL functionality in security solution to the `discover:enableESQL` advanced setting. The advanced setting will only be present in ESS, but not serverless The way this should work to maintain parity with the rest of Kibana such as discover and stack rules: - By default ES|QL will be enabled across all Kibana - When the ES|QL advanced setting is disabled: - Timeline - ES|QL tab should not be accessible on any newly created timelines - Existing Timelines with an ES|QL query should still have the tab accessible - Rules - New ES|QL rule should not be available to be created in the *Rule Creation* workflow - Existing ES|QL rules should still run and be able to be edited **Timeline Demo Video:** https://github.com/elastic/kibana/assets/17211684/d5429be9-de37-43e2-882d-687b3371beb4 **Rules Demo Video:** https://github.com/elastic/kibana/assets/17211684/7df2fd11-bd2b-4e50-ad97-b6e1d0f7867a --------- Co-authored-by: Vitalii Dmyterko <92328789+vitaliidm@users.noreply.github.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 963391e)

michaelolo24 · 2024-05-02T19:16:07Z

💚 All backports created successfully

Status	Branch	Result
✅	8.14

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

## Summary Follow up to this PR: #181616 Renaming `discover:enableEsql` to `enableESQL` according to this change: #182074

michaelolo24 · 2024-05-02T20:05:55Z

💚 All backports created successfully

Status	Branch	Result
✅	8.14

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…ed setting (elastic#181616) ## Summary This PR links the ESQL functionality in security solution to the `discover:enableESQL` advanced setting. The advanced setting will only be present in ESS, but not serverless The way this should work to maintain parity with the rest of Kibana such as discover and stack rules: - By default ES|QL will be enabled across all Kibana - When the ES|QL advanced setting is disabled: - Timeline - ES|QL tab should not be accessible on any newly created timelines - Existing Timelines with an ES|QL query should still have the tab accessible - Rules - New ES|QL rule should not be available to be created in the *Rule Creation* workflow - Existing ES|QL rules should still run and be able to be edited **Timeline Demo Video:** https://github.com/elastic/kibana/assets/17211684/d5429be9-de37-43e2-882d-687b3371beb4 **Rules Demo Video:** https://github.com/elastic/kibana/assets/17211684/7df2fd11-bd2b-4e50-ad97-b6e1d0f7867a --------- Co-authored-by: Vitalii Dmyterko <92328789+vitaliidm@users.noreply.github.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 963391e)

… advanced setting (#181616) (#182517) # Backport This will backport the following commits from `main` to `8.14`: - [[Security Solution] - Security solution ES|QL configurable via advanced setting (#181616)](#181616)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…ed setting (elastic#181616) ## Summary This PR links the ESQL functionality in security solution to the `discover:enableESQL` advanced setting. The advanced setting will only be present in ESS, but not serverless The way this should work to maintain parity with the rest of Kibana such as discover and stack rules: - By default ES|QL will be enabled across all Kibana - When the ES|QL advanced setting is disabled: - Timeline - ES|QL tab should not be accessible on any newly created timelines - Existing Timelines with an ES|QL query should still have the tab accessible - Rules - New ES|QL rule should not be available to be created in the *Rule Creation* workflow - Existing ES|QL rules should still run and be able to be edited **Timeline Demo Video:** https://github.com/elastic/kibana/assets/17211684/d5429be9-de37-43e2-882d-687b3371beb4 **Rules Demo Video:** https://github.com/elastic/kibana/assets/17211684/7df2fd11-bd2b-4e50-ad97-b6e1d0f7867a --------- Co-authored-by: Vitalii Dmyterko <92328789+vitaliidm@users.noreply.github.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

## Summary Follow up to this PR: elastic#181616 Renaming `discover:enableEsql` to `enableESQL` according to this change: elastic#182074

…hould disable ESQL Tab. (#182816) ## Summary handles #182798 Recently there was PR : #181616 which does not disable ES|QL tab in timeline even if feature flag is disabled when : - User has already created a ESQL Query in timeline and saved the timeline. This PR makes sure when below feature flag exists, then `ES|QL` tab will be definitely disabled even when user has a saved timeline with ES|QL Query in it. ```yaml xpack.securitySolution.enableExperimental: - timelineEsqlTabDisabled ``` ## Desk Testing Guidelines 1. Remove above Feature Flag 2. Go to Timeline and Create a Timeline with ESQL Query 3. Save the timeline. 4. Go To advanced Settings and search for `esql` and disable the `enableESQL` setting. 5. Go back to the timeline saved in step 3. 6. ✅ Assert that the `ES|QL` tab is still there. 7. ✅ Assert that the `ES|QL` tab is NOT there in a new timeline. 8. Add above feature flag - which disables the esql Tab. 9. Go back to the timeline saved in step 3. 10. ✅ Assert that the `ES|QL` tab is no longer there. 11. ✅ Assert that the `ES|QL` tab is NOT there in a new timeline. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

…hould disable ESQL Tab. (elastic#182816) ## Summary handles elastic#182798 Recently there was PR : elastic#181616 which does not disable ES|QL tab in timeline even if feature flag is disabled when : - User has already created a ESQL Query in timeline and saved the timeline. This PR makes sure when below feature flag exists, then `ES|QL` tab will be definitely disabled even when user has a saved timeline with ES|QL Query in it. ```yaml xpack.securitySolution.enableExperimental: - timelineEsqlTabDisabled ``` ## Desk Testing Guidelines 1. Remove above Feature Flag 2. Go to Timeline and Create a Timeline with ESQL Query 3. Save the timeline. 4. Go To advanced Settings and search for `esql` and disable the `enableESQL` setting. 5. Go back to the timeline saved in step 3. 6. ✅ Assert that the `ES|QL` tab is still there. 7. ✅ Assert that the `ES|QL` tab is NOT there in a new timeline. 8. Add above feature flag - which disables the esql Tab. 9. Go back to the timeline saved in step 3. 10. ✅ Assert that the `ES|QL` tab is no longer there. 11. ✅ Assert that the `ES|QL` tab is NOT there in a new timeline. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit c43da3e)

… flag should disable ESQL Tab. (#182816) (#182905) # Backport This will backport the following commits from `main` to `8.14`: - [[Security Solution] Timeline : Disabling Timeline ESQL feature flag should disable ESQL Tab. (#182816)](#182816)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Jatin Kathuria <jatin.kathuria@elastic.co>

michaelolo24 requested review from paulewing, vitaliidm and semd April 24, 2024 17:37

michaelolo24 changed the title ~~[Security Solution] - Security ESQL configurable via advanced setting~~ [Security Solution] - Security solution ES|QL configurable via advanced setting Apr 24, 2024

michaelolo24 added Team:Threat Hunting:Investigations Security Solution Investigations Team v8.14.0 release_note:enhancement labels Apr 24, 2024

semd approved these changes Apr 25, 2024

View reviewed changes

semd reviewed Apr 25, 2024

View reviewed changes

michaelolo24 requested a review from stratoula April 25, 2024 15:37

semd self-requested a review April 29, 2024 07:21

michaelolo24 force-pushed the link-security-to-esql-advanced-setting branch from 2e45cfe to 2ef799e Compare April 29, 2024 19:46

michaelolo24 added 4 commits April 30, 2024 10:45

link security esql configuration to advanced setting

02ae1c2

remove additional old config settings and update serverless yml

b7203e5

show esql tab for existing esql timelines

78acd00

remove esql toggle from security serverless settings

a4f8e9e

michaelolo24 force-pushed the link-security-to-esql-advanced-setting branch from 2ef799e to a4f8e9e Compare April 30, 2024 16:21

michaelolo24 marked this pull request as ready for review April 30, 2024 16:21

michaelolo24 requested a review from a team as a code owner April 30, 2024 16:21

kibanamachine mentioned this pull request May 2, 2024

[8.14] [Security Solution] - Security solution ES|QL configurable via advanced setting (#181616) #182430

Closed

michaelolo24 mentioned this pull request May 2, 2024

[Security Solution] - Rename esql setting #182432

Merged

michaelolo24 mentioned this pull request May 2, 2024

[8.14] [Security Solution] - Security solution ES|QL configurable via advanced setting (#181616) #182510

Closed

jbudz pushed a commit that referenced this pull request May 2, 2024

[Security Solution] - Rename esql setting (#182432)

c053208

## Summary Follow up to this PR: #181616 Renaming `discover:enableEsql` to `enableESQL` according to this change: #182074

michaelolo24 mentioned this pull request May 2, 2024

[8.14] [Security Solution] - Security solution ES|QL configurable via advanced setting (#181616) #182517

Merged

nastasha-solomon mentioned this pull request May 5, 2024

[Request][ESS] Document advanced setting that allows users to disable ES|QL in ESS elastic/security-docs#5163

Closed

banderror mentioned this pull request May 6, 2024

[Security Solution] Unskip tests broken by 181616 and fixed by 182432 #182682

Open

vitaliidm mentioned this pull request May 7, 2024

[Detection Engine][Serverless] Enable ES|QL rule in Serverless #181937

Closed

3 tasks

logeekal mentioned this pull request May 7, 2024

[Security Solution] Timeline : Disabling Timeline ESQL feature flag should disable ESQL Tab. #182816

Merged

1 task

szaffarano mentioned this pull request Jul 12, 2024

[8.14] [Telemetry][Security Solution] Use the proper index to query builtin alerts (#187859) #188233

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] - Security solution ES|QL configurable via advanced setting #181616

[Security Solution] - Security solution ES|QL configurable via advanced setting #181616

michaelolo24 commented Apr 24, 2024 •

edited by kibanamachine

Loading

semd left a comment •

edited

Loading

vitaliidm commented Apr 25, 2024

semd commented Apr 25, 2024

semd Apr 25, 2024

michaelolo24 commented Apr 25, 2024

michaelolo24 commented Apr 25, 2024

michaelolo24 commented Apr 25, 2024

vitaliidm commented Apr 25, 2024

stratoula commented Apr 29, 2024 •

edited

Loading

michaelolo24 commented Apr 29, 2024

michaelolo24 commented Apr 29, 2024

stratoula commented Apr 30, 2024 •

edited

Loading

vitaliidm commented Apr 30, 2024

kibanamachine commented May 2, 2024

michaelolo24 commented May 2, 2024

michaelolo24 commented May 2, 2024

[Security Solution] - Security solution ES|QL configurable via advanced setting #181616

[Security Solution] - Security solution ES|QL configurable via advanced setting #181616

Conversation

michaelolo24 commented Apr 24, 2024 • edited by kibanamachine Loading

Summary

semd left a comment • edited Loading

Choose a reason for hiding this comment

vitaliidm commented Apr 25, 2024

semd commented Apr 25, 2024

semd Apr 25, 2024

Choose a reason for hiding this comment

michaelolo24 commented Apr 25, 2024

michaelolo24 commented Apr 25, 2024

michaelolo24 commented Apr 25, 2024

vitaliidm commented Apr 25, 2024

stratoula commented Apr 29, 2024 • edited Loading

michaelolo24 commented Apr 29, 2024

michaelolo24 commented Apr 29, 2024

stratoula commented Apr 30, 2024 • edited Loading

vitaliidm commented Apr 30, 2024

kibanamachine commented May 2, 2024

💚 All backports created successfully

Questions ?

michaelolo24 commented May 2, 2024

💚 All backports created successfully

Questions ?

michaelolo24 commented May 2, 2024

💚 All backports created successfully

Questions ?

michaelolo24 commented Apr 24, 2024 •

edited by kibanamachine

Loading

semd left a comment •

edited

Loading

stratoula commented Apr 29, 2024 •

edited

Loading

stratoula commented Apr 30, 2024 •

edited

Loading