Should the term query apply normalizers to the search text when searching keyword fields

[colings86]

This issue was inspired by https://discuss.elastic.co/t/term-request-with-normalized-keyword-behave-strangely/91120

At the moment if you run a term query on a text field the terms in your query are not passed through the analyzer for that field and instead are considered as the raw terms to search in the index.

In contrary, for keyword fields, the term text is passed through the normalizer for the field, which seems a bit unexpected to me as I had always considered the term query to be used to run raw un-modified searches and the match query to be used when you wanted to applied analysis to the search terms.

[jpountz]

Discussed in FixitFriday, we will make the term query unaware of normalization.

[talevy]

Pinging @elastic/es-search-aggs

[talevy]

removing normalization feels like a breaking change, is it one?

[wmellouli]

@jpountz & @colings86 I think, this is a very good new feature introduced in elasticsearch. Otherwise, users should apply manually the same normalizer before searching data with term-level query. I updated the documentation to cover this new feature here.

Related documentation pages are:
https://www.elastic.co/guide/en/elasticsearch/reference/master/term-level-queries.html
https://www.elastic.co/guide/en/elasticsearch/reference/master/normalizer.html

So I don't know if removing normalization (before searching with term-level query) is still relevant ?
I hope no.

[mayya-sharipova]

We had a team discussion today, but still have not reached conclusion on this topic.
Some expressed ideas in favour of keeping normalization:

• normalization is also used in query_string query and terms query
• normalization is also used in terms aggregation. We don't want to have inconsistencies between results of terms agg and term query.

There was also an idea to make terms queries on text fields apply normalization as well.
As an alternative to terms queries that apply normalization, we are considering introducing exact_match query that doesn't apply any normalization, and fails on text fields.

[mayya-sharipova]

We have made a decision NOT to do this and keep the existing behaviour when term queries apply normalizers for keyword fields.
We may still revisit in the future the behaviour of term queries on text fields, where currently normalization is not applied.