Mappings: disallow exotic options on meta fields

[jpountz]

We have some mapping options that sound interesting but are actually almost useless or dangerous. I propose to remove them:

_type: { index: no }

Not indexing the _type sounds appealing since documentation mentions everything will keep on working, so that should just save space. Except that elasticsearch will internally run a prefix query on the _uid field instead, which is going to be super slow. I think we should remove this option.

_type: { store: yes } and _id: { store: yes }

Storing the type and _id is useless since we already enforce the _uid to be stored, and the _uid contains these informations.

_id: { index: not_analyzed }

The _id field is the same for all documents so we should not need to index, store or doc-value it. (Can be done now thanks to #6073 and #7965)

In general I'm wondering if we shouldn't go further and completely lock down how data is indexed/stored/docvalued for meta fields. There would just remain high-level configuration options such as enabled on the _timestamp mapper or type on _parent.

Relates to #8870

[clintongormley]

+1 I get the feeling a number of these options were added just in case the current settings didn't work out so well, but I think we can safely declare them battle tested now

[rjernst]

In general I'm wondering if we shouldn't go further and completely lock down how data is indexed/stored/docvalued for meta fields.

+1

[jpountz]

Removed the discuss label, let's do it!

[rjernst]

I started work on this, but realized the branch was becoming massive with all the test fixes for different meta fields. I'm now attempting to have a separate PR for each meta field. First one is simple, just _uid: #9836

[rjernst]

_timestamp and _all are the only meta fields left that can have their field type modifed. I'm going to close this issue as _timestamp has its own discussion going in #9679 and _all I'm not sure what we can do there to limit it down (more discussion is probably needed), but we probably want to lock out some settings or improve error messages (right now, if you try to set it to not indexed, it just silently indexes it anyways...)

[jpountz]

Thanks @rjernst these changes are already awesome!

[djschny]

In regards to the _type disabling, I've used this historically when I already had a field on my document that was the same value as the _type and therefore indexing it was not necessary since would craft the queries to filter on that field instead. For folks that have billions of very small documents, it was my understanding that not indexing _type was helpful as the ratio of that data compared to overall document size was larger.

[rjernst]

@djschny if there is a field that duplicates a meta field, why can't the user not send that field? _type is essentially a virtual field on _uid, so it is actually not using any more memory than would be used nornally.

[djschny]

@rjernst Sure a user could not send that field, but that may not be ideal for them as they want their document to contain that field explicitly for when pulling data back out of ES. I understand that _type is essentially a virtual field on _uid, but it is my understanding that unless it is indexed separately then queries to find all docs of a particular type use a prefix query on _id which is not as ideal as a filter.

Why I bring this up is there are valid use cases for the configuration of _type to not be indexed and it's not an "exotic" use case. I think its very important to draw a distinction between a configuration that is invalid or not necessary (like the "store" on _type) vs. ones that have a practical use case.

[rjernst]

I said "essentially" because it wasn't quite true. _type is indexed (the virtual field over _uid is for returning type as a stored field), but this is the cost of having the type system. Having types not indexed is not a valid use case, it is necessary for the basic operation of elasticsearch (doing a search like GET /myindex/type1,type2,type3/_search).

If a user doesn't want to pay that cost, they can not use types. By that I mean: send all their documents with the same ES type (the type will still be indexed, but the posting list will be highly compressed because all docs will have the same value). Then they can make their type field (notice no underscore) and do with it as they please.