[Security Solution][Exceptions] Add lowercase normalizer for case-insensitivity + deprecate _tags field (new OS field) #77379

madirey · 2020-09-14T17:24:21Z

Summary

This PR ensures that *.caseless fields appearing in exceptions are translated correctly so that the endpoint can use them for case-insensitive comparisons.

A migration is added to convert *.text fields appearing in exceptions to *.caseless so that the detection engine will perform case-insensitive comparisons in the same way that the endpoint does (without tokenization).

Requires updating to a new endpoint package which contains the below fix:

Additionally, this PR deprecates usage of _tags in the lists plugin, in favor of os_types, which is more explicit. We were only using it for OS designation, so the relevant tags will be migrated. The field will remain in ES for now, but it has been removed from all schemas. This will prevent the confusion between _tags and tags and will prevent the user from inadvertently updating _tags to a value that breaks internal functionality.

Finally, this PR fixes the button text when adding an endpoint exception:

Checklist

Delete any items that are not applicable to this PR.

Documentation was added for features that require explanation or tutorials -- TODO
Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2020-09-14T17:24:23Z

Pinging @elastic/endpoint-response (Team:Endpoint Response)

elasticmachine · 2020-09-14T17:29:04Z

Pinging @elastic/siem (Team:SIEM)

elasticmachine · 2020-09-14T17:29:27Z

Pinging @elastic/endpoint-app-team (Feature:Endpoint)

…malizer

elasticmachine · 2020-09-23T11:51:01Z

Pinging @elastic/ingest-management (Team:Ingest Management)

…malizer

x-pack/plugins/security_solution/server/endpoint/lib/artifacts/lists.ts

marshallmain · 2020-09-24T20:48:36Z

x-pack/plugins/lists/server/saved_objects/migrations.test.ts

+      migration((doc as unknown) as SavedObjectUnsanitizedDoc<OldExceptionListSoSchema>)
+    ).toEqual({
+      attributes: {
+        _tags: ['1234', 'os:windows'],


With _tags deleted from the exceptionListItemSchema won't migrated exception list items fail to validate against it in update_exception_list_item_route and others if the items still contain _tags?

@marshallmain No, the validation is performed against an object that will already have _tags removed from the SO response.

marshallmain · 2020-09-24T20:51:04Z

x-pack/plugins/security_solution/server/endpoint/routes/trusted_apps/utils.ts

-  const { entries, description, created_by, created_at, name, _tags, id } = exceptionListItem;
-  const os = osFromTagsList(_tags);
+  const { entries, description, created_by, created_at, name, os_types, id } = exceptionListItem;
+  const os = os_types.length ? os_types[0] : 'unknown';


Trusted apps supports only 1 OS per entry, thus why we return a string rather than string[]. unknown should never happen (🤞 )

marshallmain · 2020-09-24T21:08:55Z

x-pack/plugins/lists/server/saved_objects/migrations.ts

+const migrateEntry = (entryToMigrate: EntryType): EntryType => {
+  const newEntry = entryToMigrate;
+  if (entriesNested.is(entryToMigrate) && entriesNested.is(newEntry)) {
+    newEntry.entries = entryToMigrate.entries.map((nestedEntry) =>
+      migrateEntry(nestedEntry)
+    ) as NonEmptyNestedEntriesArray;
+  }
+  newEntry.field = entryToMigrate.field.replace('.text', '.caseless');
+  return newEntry;


does making newEntry do anything here? it looks like a reference to the same object as entryToMigrate. The if conditions appear to be redundant as well.

@marshallmain Thanks, will fix!

@marshallmain Oh, this was actually to get around the no-param-reassign eslint rule. Leaving it as-is.

…malizer

marshallmain

LGTM once the unit test failures are fixed

marshallmain · 2020-09-26T01:04:52Z

One other note is that the exception migration from .text to .caseless is a breaking change for exceptions until the endpoint package is upgraded - we may need users to manually upgrade the endpoint package in ingest manager since the ingest manager does not (I think?) automatically install package upgrades as they become available.

madirey · 2020-09-26T01:38:41Z

@marshallmain I believe @jonathan-buttner is working on that...

jonathan-buttner · 2020-10-01T13:43:04Z

@marshallmain I believe @jonathan-buttner is working on that...

Oops sorry I had missed this previously. Yeah we implemented a check when the user is interacting with the security solution app and in the ingest manager app that checks for a new endpoint package and installs it:

#78081
#77498

…malizer

madirey · 2020-10-01T17:16:15Z

@elasticmachine merge upstream

madirey · 2020-10-01T18:30:39Z

@elasticmachine merge upstream

paul-tavares

Looks good from a Trusted Apps standpoint. I assume you are still working on getting the changes in for the .caseless change for trusted apps, right?

paul-tavares · 2020-10-02T14:39:56Z

x-pack/plugins/security_solution/server/endpoint/routes/trusted_apps/utils.ts

-  const { entries, description, created_by, created_at, name, _tags, id } = exceptionListItem;
-  const os = osFromTagsList(_tags);
+  const { entries, description, created_by, created_at, name, os_types, id } = exceptionListItem;
+  const os = os_types.length ? os_types[0] : 'unknown';


Trusted apps supports only 1 OS per entry, thus why we return a string rather than string[]. unknown should never happen (🤞 )

…malizer

kibanamachine · 2020-10-02T16:33:31Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 62d0362

Metrics [docs]

@kbn/optimizer bundle module count

id	before	after	diff
`securitySolution`	1989	1990	+1

async chunks size

id	before	after	diff
`securitySolution`	10.3MB	10.3MB	-1.3KB

distributable file count

id	before	after	diff
`default`	45824	45826	+2

page load bundle size

id	before	after	diff
`lists`	164.1KB	164.7KB	+658.0B
`securitySolution`	585.9KB	586.9KB	+996.0B
total			+1.6KB

Saved Objects .kibana field count

id	before	after	diff
`exception-list`	38	39	+1
`exception-list-agnostic`	38	39	+1
total			+2

History

💚 Build #78883 succeeded 6a70761
💔 Build #78853 failed 7921a1f
💔 Build #78832 failed 0753542
💔 Build #78768 failed 3815908
💔 Build #78723 failed b54ffc4

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

paul-tavares

Changes to Trusted Apps code for the .caseless field looks good 👍

…ensitivity + deprecate _tags field (new OS field) (#77379) (#79376) * Finish adding .lower to exceptionable fields * Add back migrations * .lower -> .caseless * Add separate field for os type * updates * Type updates * Switch over to osTypes * get rid of _tags * Add tests for schema validation * Remove remaining references to _tags * Another round of test fixes * DefaultArray tests * More test fixes * Fix remaining test failures * types / tests * more test updates * lowercase os values * Address feedback + fix test failure * tests * Fix integration test * process.executable.path -> process.executable.caseless Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

elasticmachine · 2021-09-23T14:25:57Z

Pinging @elastic/security-solution (Team: SecuritySolution)

madirey added v8.0.0 Team:Endpoint Response Endpoint Response Team v7.10 labels Sep 14, 2020

madirey requested review from a team as code owners September 14, 2020 17:24

madirey mentioned this pull request Sep 14, 2020

[Security Solution][Exceptions] Add lowercase normalized fields for case-insensitive matching elastic/endpoint-package#79

Merged

madirey added release_note:enhancement Team:SIEM labels Sep 14, 2020

madirey added the Feature:Endpoint Elastic Endpoint feature label Sep 14, 2020

jen-huang added v7.10.0 and removed v7.10 labels Sep 14, 2020

Finish adding .lower to exceptionable fields

e7db631

madirey force-pushed the lowercase-normalizer branch from 153eb41 to e7db631 Compare September 18, 2020 15:20

madirey added 9 commits September 18, 2020 15:23

Merge branch 'master' of github.com:elastic/kibana into lowercase-nor…

4219411

…malizer

Add back migrations

3693404

Merge branch 'master' of github.com:elastic/kibana into lowercase-nor…

e36002d

…malizer

Merge branch 'master' of github.com:elastic/kibana into lowercase-nor…

614d504

…malizer

.lower -> .caseless

af305c5

Add separate field for os type

04f4b42

Merge branch 'master' of github.com:elastic/kibana into lowercase-nor…

b2888b5

…malizer

updates

dd355ed

Merge branch 'master' of github.com:elastic/kibana into lowercase-nor…

d838ec1

…malizer

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Sep 23, 2020

madirey added 2 commits September 23, 2020 14:41

Type updates

1d658c6

Merge branch 'master' of github.com:elastic/kibana into lowercase-nor…

fce8899

…malizer

madirey requested a review from a team September 23, 2020 14:42

marshallmain reviewed Sep 24, 2020

View reviewed changes

madirey added 2 commits September 25, 2020 22:43

Address feedback + fix test failure

288b05b

Merge branch 'master' of github.com:elastic/kibana into lowercase-nor…

8ae05b9

…malizer

marshallmain approved these changes Sep 26, 2020

View reviewed changes

marshallmain mentioned this pull request Sep 28, 2020

[Security Solution][Detections] Endpoint exceptions do not filter by OS in detection engine #78599

Closed

peluja1012 added the needs_docs label Sep 30, 2020

Merge master, fix conflicts

b54ffc4

madirey added 4 commits October 1, 2020 14:30

tests

70c0cd1

Merge branch 'master' of github.com:elastic/kibana into lowercase-nor…

3815908

…malizer

Fix integration test

f03fada

Merge branch 'master' of github.com:elastic/kibana into lowercase-nor…

0753542

…malizer

Merge branch 'master' into lowercase-normalizer

7921a1f

Merge branch 'master' into lowercase-normalizer

6a70761

madirey changed the title ~~[Security Solution][Exceptions] Add lowercase normalizer for more accurate case-insensitive exceptions~~ [Security Solution][Exceptions] Add lowercase normalizer for case-insensitivity + deprecate _tags field (new OS field) Oct 2, 2020

paul-tavares reviewed Oct 2, 2020

View reviewed changes

madirey added 2 commits October 2, 2020 14:56

Merge branch 'master' of github.com:elastic/kibana into lowercase-nor…

24ced09

…malizer

process.executable.path -> process.executable.caseless

62d0362

paul-tavares approved these changes Oct 2, 2020

View reviewed changes

madirey merged commit c456f64 into elastic:master Oct 2, 2020

madirey deleted the lowercase-normalizer branch October 2, 2020 19:54

madirey mentioned this pull request Oct 2, 2020

[7.x] [Security Solution][Exceptions] Add lowercase normalizer for case-insensitivity + deprecate _tags field (new OS field) (#77379) #79376

Merged

MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Sep 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution][Exceptions] Add lowercase normalizer for case-insensitivity + deprecate _tags field (new OS field) #77379

[Security Solution][Exceptions] Add lowercase normalizer for case-insensitivity + deprecate _tags field (new OS field) #77379

madirey commented Sep 14, 2020 •

edited

Loading

elasticmachine commented Sep 14, 2020

elasticmachine commented Sep 14, 2020

elasticmachine commented Sep 14, 2020

elasticmachine commented Sep 23, 2020

marshallmain Sep 24, 2020

madirey Sep 25, 2020

marshallmain Sep 24, 2020

madirey Sep 25, 2020

paul-tavares Oct 2, 2020

marshallmain Sep 24, 2020

madirey Sep 25, 2020

madirey Sep 25, 2020

marshallmain left a comment

marshallmain commented Sep 26, 2020

madirey commented Sep 26, 2020

jonathan-buttner commented Oct 1, 2020

madirey commented Oct 1, 2020

madirey commented Oct 1, 2020

paul-tavares left a comment

paul-tavares Oct 2, 2020

kibanamachine commented Oct 2, 2020

paul-tavares left a comment

elasticmachine commented Sep 23, 2021

[Security Solution][Exceptions] Add lowercase normalizer for case-insensitivity + deprecate _tags field (new OS field) #77379

[Security Solution][Exceptions] Add lowercase normalizer for case-insensitivity + deprecate _tags field (new OS field) #77379

Conversation

madirey commented Sep 14, 2020 • edited Loading

Summary

Checklist

For maintainers

elasticmachine commented Sep 14, 2020

elasticmachine commented Sep 14, 2020

elasticmachine commented Sep 14, 2020

elasticmachine commented Sep 23, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marshallmain left a comment

Choose a reason for hiding this comment

marshallmain commented Sep 26, 2020

madirey commented Sep 26, 2020

jonathan-buttner commented Oct 1, 2020

madirey commented Oct 1, 2020

madirey commented Oct 1, 2020

paul-tavares left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibanamachine commented Oct 2, 2020

💚 Build Succeeded

Metrics [docs]

@kbn/optimizer bundle module count

async chunks size

distributable file count

page load bundle size

Saved Objects .kibana field count

History

paul-tavares left a comment

Choose a reason for hiding this comment

elasticmachine commented Sep 23, 2021

madirey commented Sep 14, 2020 •

edited

Loading