Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove support for dynamic mappings in SavedObject types #69985

Closed
joshdover opened this issue Jun 25, 2020 · 2 comments
Closed

Remove support for dynamic mappings in SavedObject types #69985

joshdover opened this issue Jun 25, 2020 · 2 comments
Labels
Feature:New Platform Feature:Saved Objects Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc

Comments

@joshdover
Copy link
Contributor

As we continue to add more SavedObject types, we are starting to push up against the default field limit of 1000 mapped fields in Elasticsearch.

One common culprit that causes this is the usage of dynamic mappings in SavedObject schemas that cause the mappings to grow over time and eventually hit the 1000 limit. This failure scenario mappings are hard to test for since it depends on the shape of the data being ingested which can vary based on user input.

In most cases, dynamic mappings are not needed since we do not need to aggregate or search on these fields. Instead, we should either be using index: false or dynamic: false in order to support documents with dynamic key-value pairs without needing to index these keys in Elasticsearch.

An audit needs to be performed, but we may be able to remove support for this altogether in order to eliminate this problem.

Related #69869

@joshdover joshdover added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Feature:New Platform Feature:Saved Objects labels Jun 25, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-platform (Team:Platform)

@pgayvallet
Copy link
Contributor

  1. We're now discouraging usage of dynamic mapping via our TS mapping type that does not accept dynamic: true

export type SavedObjectsFieldMapping = EsMappingProperty & {
/**
* The dynamic property of the mapping, either `false` or `'strict'`. If
* unspecified `dynamic: 'strict'` will be inherited from the top-level
* index mappings.
*
* Note: To limit the number of mapping fields Saved Object types should
* *never* use `dynamic: true`.
*/
dynamic?: false | 'strict';

  1. The index split we did last year reduced the risk of mapping explosions

This has been addressed, closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature:New Platform Feature:Saved Objects Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
Projects
None yet
Development

No branches or pull requests

3 participants