support case_insensitive in terms query

[zhuming]

Elasticsearch version (bin/elasticsearch --version):
7.10.2

Plugins installed: []
none

JVM version (java -version):
any

OS version (uname -a if on a Unix-like system):
any

Description of the problem including expected versus actual behavior:
Trying add case_insensitive:true to a terms query,

get alias_player/_search
{
  "query": {
    "terms": {
      "email": [
        "Diego.Maradona@fifa.com",
        "robert.BAGGIO@FIFA.com"
      ],
      "case_insensitive": true
    }
  }
}

and it failed with the below error info

{
    "type" : "parsing_exception",
    "reason" : "[terms] query does not support [case_insensitive]",
    "line" : 8,
    "col" : 27
}

case_insensitive works for term/query/wildcard, but not for terms

Steps to reproduce:

Please include a minimal but complete recreation of the problem,
including (e.g.) index creation, mappings, settings, query etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

1. create a simple index with a keyword field, e.g. email, in the above example
2. execute the above query
3. elasticsearch fails to parse this request with case_insensitive flag

Provide logs (if relevant):

[cbuescher]

Hi @zhuming, I'm relabeling this as an enhancement request because I believe this parameter hasn't been supported for this type of query so far. We might decide to add it but I don't see that we did this in any previously released version. Please let me know if I'm mistaken on this.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[davendu]

For anyone interested: This was planned in #61162, and the last comment before issue closed already stated case_insensitive for terms is not implemented due to possible complexity explosion.

[SudoNova]

A workaround is using using a text field with

Record -> keyword tokenizer -> lowercase token filter

both as index and search analyzer. Performance and storage is blackbox here (for me, not the devs ofc ).
You may use a multiplexer before lowercase to preserve original keywords (text tokens).

[javanna]

Relates to #82825

[javanna]

We discussed this with the team and summarized the reasons why the functionality is not currently supported: TermsInSetQuery that the terms query is translated to in Lucene land does not support case insensitive queries. We would then have to automatically rewrite the query to a boolean query made of many low-level term queries, and there is a limit to the number of clauses that a boolean query can have.

Also, we would like to hear more about real usecases that would require terms to be case insensitive. Very often ids are looked up and they don't require analysis.

We would be happy to enable this feature once TermsInSetQuery supports it natively.

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)