[RAC][Rule Registry] Generate ECS fieldmap from ECS 8.0 #123012

rylnd · 2022-01-14T00:23:44Z

Summary

This is the result of running the generate_ecs_fieldmap script against ECS' 8.0 branch.

Checklist

Unit or functional tests were updated or added to match the most common scenarios

This is the result of running the generate_ecs_fieldmap script against ECS' 8.0 branch.

rylnd · 2022-01-14T01:04:31Z

Scanning CI it looks like this is the offending error, currently:

│ proc [kibana] [2022-01-14T00:59:57.161+00:00][ERROR][plugins.ruleRegistry] ResponseError: mapper_parsing_exception: [mapper_parsing_exception] Reason: Failed to parse mapping: Field [scaling_factor] is required

│ proc [kibana] [2022-01-14T00:54:43.051+00:00][ERROR][plugins.ruleRegistry] Error: Failure installing common resources shared between all indices. mapper_parsing_exception: [mapper_parsing_exception] Reason: Failed to parse mapping: Field [scaling_factor] is required

I'm not sure which field that pertains to, but will continue to investigate.

marshallmain · 2022-01-14T04:39:05Z

x-pack/plugins/rule_registry/common/assets/field_maps/ecs_field_map.ts

+  'host.cpu.usage': {
+    type: 'scaled_float',
+    array: false,
+    required: false,
+  },


Looks like this is the source of the mapping error, the script that generates this file doesn't pick up the scaling_factor from the ECS file (https://github.com/elastic/ecs/blob/main/generated/ecs/ecs_flat.yml#L4646) and scaling_factor is a required property for scaled_float fields.

This is a required field for e.g. scaled_float fields, so we need to reflect its value in our field map.

It does not appear that this value was ever being set, nor does this value appear in ECS' flat output, so I'm removing it for now to keep our types as accurate as possible.

This is a required field for type: alias fields.

This now exceeds the default of 1000.

Apparently 1300 wasn't enough, either.

rylnd · 2022-01-14T21:09:57Z

For posterity: the errors related to these changes look to be due to #108941 and the resulting PR.

Makes this field optional, since the technical component template doesn't currently use it.

Including the newest ECS fields, this index now exceeds 1600 fields. This value should probably be derived from the composed template's limits, but for now this allows the template to be created.

rylnd · 2022-01-18T06:30:41Z

@elasticmachine merge upstream

kibana-ci · 2022-01-18T07:49:46Z

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`apm`	2.8MB	2.8MB	+33.4KB
`observability`	357.8KB	391.2KB	+33.4KB
total			+66.7KB

History

💚 Build #17645 succeeded 6313f82
💔 Build #17638 failed 4dcfd86
💔 Build #17631 failed a632b55
💔 Build #17609 failed c113d26
💔 Build #17417 failed 1009642

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @rylnd

ecezalp · 2022-01-18T18:59:34Z

threat fields w/ both threat.enrichments.* and threat.indicator.* LGTM, thanks

elasticmachine · 2022-01-18T21:25:39Z

Pinging @elastic/response-ops (Team:ResponseOps)

kibanamachine · 2022-01-19T01:48:06Z

The following labels were identified as gaps in your version labels and will be added automatically:

v8.1.0

If any of these should not be on your pull request, please manually remove them.

* Generate ECS fieldmap from ECS 8.0 This is the result of running the generate_ecs_fieldmap script against ECS' 8.0 branch. * Account for scaling_factor property from ECS This is a required field for e.g. scaled_float fields, so we need to reflect its value in our field map. * Remove unused, unset property from FieldMap It does not appear that this value was ever being set, nor does this value appear in ECS' flat output, so I'm removing it for now to keep our types as accurate as possible. * Add path back to FieldMap definition This is a required field for type: alias fields. * Try upping the fields limit on our ECS component template This now exceeds the default of 1000. * Bump our field limit a bit more Apparently 1300 wasn't enough, either. * Fix type error Makes this field optional, since the technical component template doesn't currently use it. * Bump the field limit of our composed template Including the newest ECS fields, this index now exceeds 1600 fields. This value should probably be derived from the composed template's limits, but for now this allows the template to be created. Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 8737691)

kibanamachine · 2022-01-19T01:50:43Z

💚 All backports created successfully

Status	Branch	Result
✅	8.0

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…23334) * Generate ECS fieldmap from ECS 8.0 This is the result of running the generate_ecs_fieldmap script against ECS' 8.0 branch. * Account for scaling_factor property from ECS This is a required field for e.g. scaled_float fields, so we need to reflect its value in our field map. * Remove unused, unset property from FieldMap It does not appear that this value was ever being set, nor does this value appear in ECS' flat output, so I'm removing it for now to keep our types as accurate as possible. * Add path back to FieldMap definition This is a required field for type: alias fields. * Try upping the fields limit on our ECS component template This now exceeds the default of 1000. * Bump our field limit a bit more Apparently 1300 wasn't enough, either. * Fix type error Makes this field optional, since the technical component template doesn't currently use it. * Bump the field limit of our composed template Including the newest ECS fields, this index now exceeds 1600 fields. This value should probably be derived from the composed template's limits, but for now this allows the template to be created. Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 8737691) Co-authored-by: Ryland Herrick <ryalnd@gmail.com>

elasticmachine · 2022-01-19T15:28:59Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@madirey

## Summary New ECS FieldMap was generated in #123012, however since it only contained changes to `Rule Registry` code the `Security Solution` Cypress tests were not run, and thus did not catch this field change. See #122661 (comment) for details. Confirmed w/ @madirey that expected value is indeed `5` now that `host.geo.continent_code` has been [added](https://github.com/elastic/kibana/pull/123012/files#diff-a1647ccb73ef26c8c8b6aefd87084504b146af72fcb088ccacad93fcaad15b69R1524-R1528). Some failing PR's from `main`: #123357 #121644 #123352 ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

@madirey

…123429) ## Summary New ECS FieldMap was generated in elastic#123012, however since it only contained changes to `Rule Registry` code the `Security Solution` Cypress tests were not run, and thus did not catch this field change. See elastic#122661 (comment) for details. Confirmed w/ @madirey that expected value is indeed `5` now that `host.geo.continent_code` has been [added](https://github.com/elastic/kibana/pull/123012/files#diff-a1647ccb73ef26c8c8b6aefd87084504b146af72fcb088ccacad93fcaad15b69R1524-R1528). Some failing PR's from `main`: elastic#123357 elastic#121644 elastic#123352 ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit d6917fc)

@madirey

…#123433) ## Summary New ECS FieldMap was generated in #123012, however since it only contained changes to `Rule Registry` code the `Security Solution` Cypress tests were not run, and thus did not catch this field change. See #122661 (comment) for details. Confirmed w/ @madirey that expected value is indeed `5` now that `host.geo.continent_code` has been [added](https://github.com/elastic/kibana/pull/123012/files#diff-a1647ccb73ef26c8c8b6aefd87084504b146af72fcb088ccacad93fcaad15b69R1524-R1528). Some failing PR's from `main`: #123357 #121644 #123352 ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios (cherry picked from commit d6917fc) Co-authored-by: Garrett Spong <spong@users.noreply.github.com>

* Generate ECS fieldmap from ECS 8.0 This is the result of running the generate_ecs_fieldmap script against ECS' 8.0 branch. * Account for scaling_factor property from ECS This is a required field for e.g. scaled_float fields, so we need to reflect its value in our field map. * Remove unused, unset property from FieldMap It does not appear that this value was ever being set, nor does this value appear in ECS' flat output, so I'm removing it for now to keep our types as accurate as possible. * Add path back to FieldMap definition This is a required field for type: alias fields. * Try upping the fields limit on our ECS component template This now exceeds the default of 1000. * Bump our field limit a bit more Apparently 1300 wasn't enough, either. * Fix type error Makes this field optional, since the technical component template doesn't currently use it. * Bump the field limit of our composed template Including the newest ECS fields, this index now exceeds 1600 fields. This value should probably be derived from the composed template's limits, but for now this allows the template to be created. Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

@madirey

…123429) ## Summary New ECS FieldMap was generated in elastic#123012, however since it only contained changes to `Rule Registry` code the `Security Solution` Cypress tests were not run, and thus did not catch this field change. See elastic#122661 (comment) for details. Confirmed w/ @madirey that expected value is indeed `5` now that `host.geo.continent_code` has been [added](https://github.com/elastic/kibana/pull/123012/files#diff-a1647ccb73ef26c8c8b6aefd87084504b146af72fcb088ccacad93fcaad15b69R1524-R1528). Some failing PR's from `main`: elastic#123357 elastic#121644 elastic#123352 ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

Generate ECS fieldmap from ECS 8.0

1009642

This is the result of running the generate_ecs_fieldmap script against ECS' 8.0 branch.

rylnd added release_note:enhancement v8.0.0 Feature:RAC label obsolete labels Jan 14, 2022

rylnd self-assigned this Jan 14, 2022

rylnd requested review from marshallmain, jasonrhodes and kobelb January 14, 2022 00:23

marshallmain reviewed Jan 14, 2022

View reviewed changes

rylnd added 5 commits January 14, 2022 12:48

Account for scaling_factor property from ECS

ba8b93d

This is a required field for e.g. scaled_float fields, so we need to reflect its value in our field map.

Remove unused, unset property from FieldMap

c113d26

It does not appear that this value was ever being set, nor does this value appear in ECS' flat output, so I'm removing it for now to keep our types as accurate as possible.

Add path back to FieldMap definition

6696b2b

This is a required field for type: alias fields.

Try upping the fields limit on our ECS component template

a632b55

This now exceeds the default of 1000.

Bump our field limit a bit more

4dcfd86

Apparently 1300 wasn't enough, either.

rylnd added 2 commits January 14, 2022 15:33

Fix type error

a028046

Makes this field optional, since the technical component template doesn't currently use it.

Bump the field limit of our composed template

6313f82

Including the newest ECS fields, this index now exceeds 1600 fields. This value should probably be derived from the composed template's limits, but for now this allows the template to be created.

rylnd marked this pull request as ready for review January 15, 2022 01:59

Merge branch 'main' into update_ecs_mappings_8_0

c7fd66b

rylnd added the Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) label Jan 18, 2022

kobelb approved these changes Jan 18, 2022

View reviewed changes

rylnd added the auto-backport Deprecated - use backport:version if exact versions are needed label Jan 19, 2022

rylnd merged commit 8737691 into elastic:main Jan 19, 2022

kibanamachine added the v8.1.0 label Jan 19, 2022

kibanamachine mentioned this pull request Jan 19, 2022

[8.0] [RAC][Rule Registry] Generate ECS fieldmap from ECS 8.0 (#123012) #123334

Merged

rylnd deleted the update_ecs_mappings_8_0 branch January 19, 2022 02:09

KOTungseth added the Team:Detections and Resp Security Detection Response Team label Jan 19, 2022

spong mentioned this pull request Jan 20, 2022

[Security Solution] Fixes broken cypress test after 8.0 ECS update to Rule Registry #123429

Merged

1 task

nkhristinin mentioned this pull request Sep 21, 2022

Auto generate ECS fields #141152

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RAC][Rule Registry] Generate ECS fieldmap from ECS 8.0 #123012

[RAC][Rule Registry] Generate ECS fieldmap from ECS 8.0 #123012

rylnd commented Jan 14, 2022

rylnd commented Jan 14, 2022 •

edited

Loading

marshallmain Jan 14, 2022

rylnd commented Jan 14, 2022

rylnd commented Jan 18, 2022

kibana-ci commented Jan 18, 2022

ecezalp commented Jan 18, 2022

elasticmachine commented Jan 18, 2022

kibanamachine commented Jan 19, 2022

kibanamachine commented Jan 19, 2022

elasticmachine commented Jan 19, 2022

[RAC][Rule Registry] Generate ECS fieldmap from ECS 8.0 #123012

[RAC][Rule Registry] Generate ECS fieldmap from ECS 8.0 #123012

Conversation

rylnd commented Jan 14, 2022

Summary

Checklist

rylnd commented Jan 14, 2022 • edited Loading

marshallmain Jan 14, 2022

Choose a reason for hiding this comment

rylnd commented Jan 14, 2022

rylnd commented Jan 18, 2022

kibana-ci commented Jan 18, 2022

💚 Build Succeeded

Metrics [docs]

Async chunks

History

ecezalp commented Jan 18, 2022

elasticmachine commented Jan 18, 2022

kibanamachine commented Jan 19, 2022

kibanamachine commented Jan 19, 2022

💚 All backports created successfully

Questions ?

elasticmachine commented Jan 19, 2022

rylnd commented Jan 14, 2022 •

edited

Loading