Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allow forcing dynamic object type mapping #6691

Merged
merged 3 commits into from
Mar 30, 2018

Conversation

graphaelli
Copy link
Member

@graphaelli graphaelli commented Mar 28, 2018

Currently dynamic mappings are tolerant of unexpected data types. For example, this definition:

- name: test
    type: object
    object_type: long
    dynamic: true

produces this dynamic template:

"test": {
  "path_match": "test.*",
  "match_mapping_type": "long",
  "mapping": {
    "type": "long"
  }
}

If the first value seen is a string, say indexing this document:

"test": {
      "somelong": 16,
      "somestring": "foo"
}

the resulting mapping is actually:

"test": {
  "dynamic": "true",
  "properties": {
    "test": {
      "properties": {
        "somelong": {
          "type": "long"
        },
        "somestring": {
          "type": "keyword",
          "ignore_above": 1024
        }
      }
    }
  }
}

In apm-server, data is already validated before being sent along to elasticsearch, so barring a bug, a string will not be sent where a long is required. This change allows forcing the data for a dynamic field to be a specific type via field config, eg for:

- name: test
    type: object
    object_type: long
    object_type_mapping_type: "*"
    dynamic: true

The resulting dynamic template would be:

"test": {
  "path_match": "test.*",
  "match_mapping_type": "*",
  "mapping": {
    "type": "long"
  }
}

Any value that can't be mapped to long will result in an indexing error. Continuing this example:

"error": {
  "root_cause": [
    {
      "type": "mapper_parsing_exception",
       "reason": "failed to parse [test.somestring]"
    }
  ]
}

@ruflin
Copy link
Member

ruflin commented Mar 29, 2018

Should we introduce a config match_mapping_type instead to be closer to the naming in Elasticsearch. Instead of setting the force config it would allow to set the type that it should actually map, also *.

Can you add a CHANGELOG?

For the PR description it would be nice if it would also contain the new functionality not only the way it works now which would make it easy to compare the two without having to go into the code.

@graphaelli
Copy link
Member Author

Should we introduce a config match_mapping_type instead to be closer to the naming in Elasticsearch. Instead of setting the force config it would allow to set the type that it should actually map, also *.

Works for me, is that your preference then?

Can you add a CHANGELOG?

Will do once this is takes shape.

For the PR description it would be nice if it would also contain the new functionality not only the way it works now which would make it easy to compare the two without having to go into the code.

Hey, I thought the original description was pretty good! I'll update for future readers.

@graphaelli
Copy link
Member Author

@ruflin implemented object_type_mapping_type for your consideration, changelog updated.

Copy link
Member

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. TBH the original PR message was already really good. It is probably the reason you teased me in requesting the second part about how the solution looks like without having to look at the code. I really appreciate the very good PR descriptions ❤️

@ruflin ruflin merged commit cfe008e into elastic:master Mar 30, 2018
@graphaelli graphaelli deleted the strict-object-type branch March 30, 2018 12:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants