Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fleet] Generate dynamic template mappings for field with wildcard in name #137978

Conversation

nchaulet
Copy link
Member

@nchaulet nchaulet commented Aug 3, 2022

Description

Resolve #129344

Fleet create datastream mappings based on fields yaml provided by the integration, it was decided that the fields that contains a * in the name should be installed as dynamic template and not property mappings that PR change that.

If a field name contains a * the field will be transformed to be of type object with ${object_type:field.type}, for example the followed field will be transformed like this

- name: test.*.toto
  type: scaled_float
  description: test

# will become
- name: test.*.toto
  type: object
  object_type: scaled_float
  description: test

And with the work introduced in #137772 this will result in dynamic_templates in the final mappings.

@jsoriano Does it make sense to you?

@nchaulet nchaulet added release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v8.5.0 labels Aug 3, 2022
@nchaulet nchaulet force-pushed the feature-generate-dynaminc-mapping-field-wildcard branch from 9a937a9 to 6255698 Compare August 3, 2022 13:19
@nchaulet nchaulet marked this pull request as ready for review August 3, 2022 14:12
@nchaulet nchaulet requested a review from a team as a code owner August 3, 2022 14:12
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@nchaulet nchaulet added the v8.4.0 label Aug 3, 2022
Copy link
Member

@kpollich kpollich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor refactor suggestion otherwise LGTM

export function processFieldsWithWildcard(fields: Fields): Fields {
const newFields: Fields = [];
for (const field of fields) {
if (field.name.includes('*') && (field.type !== 'object' || !field.object_type)) {
Copy link
Member

@kpollich kpollich Aug 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This condition may be complex enough to warrant breaking out into a few variables, e.g.

const hasWildcard = field.name.includes('*');
const isObjectField = field.type === 'object' || field.object_type;

if (hasWildcard && !isObjectField) {
  // ...
} else {
  // ...
}

Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks!

const newFields: Fields = [];
for (const field of fields) {
const hasWildcard = field.name.includes('*');
const hasNotObjectType = !field.object_type;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would probably be a bit more sensible as a "positive" condition that's negated below

const hasObjectType = !!field.object_type

if (hasWildcard && !hasObjectType) ...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes it make sense, having a bad time naming variables on this one

@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@nchaulet nchaulet merged commit 88eae3e into elastic:main Aug 3, 2022
@nchaulet nchaulet deleted the feature-generate-dynaminc-mapping-field-wildcard branch August 3, 2022 18:04
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Aug 3, 2022
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
8.4

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v8.4.0 v8.5.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Fleet] dynamic_template mappings for wildcard field names are not installed
6 participants