Aggs and constant_keyword

[nik9000]

Description

It'd be lovely is a filters or filter agg that touches a constant_keyword would properly rewrite itself to match_all and match_none and "go fast", at least when it's at the root of the aggs tree. Theoretically it should "just work" but it doesn't seem to. Also! The terms agg should be able to act on there being only a single possible value. Maybe it can even act on the match_all/match_none nature of the constant_keyword. I'm not sure how, but it does feel like there are optimizations in there. Particularly when terms is the top level agg.

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[nik9000]

We should also have a look at filters when the constant_keyword is deep in a bool query inside the filter. It should get rewritten and then merged into a simpler query.

[AkisPanagiotopoulos]

Hello @nik9000 , can i take this issue as a first time contributor?

[utkarsh-shrivastav77]

Hi is this issue still open

[afn7081]

Can i take this up if its still open?

[kkrik-es]

This issue is fairly involved and has some weird interactions with other ongoing efforts in aggregations. We prefer to keep this assigned as is, hope to make progress in the coming weeks.

[atript8]

@kkrik-es please remove the good first issue tag then.

[kkrik-es]

A single setup with one table containing a const_keywork shows that the query gets actually rewritten to match_all or match_none, e.g. (YAML)

  - do:
      indices.create:
        index:  bicycles
        body:
          settings:
            number_of_shards: 1
          mappings:
            properties:
              cycle_type:
                type: constant_keyword
                value: bicycle
              id:
                type: integer
              status:
                type: keyword
              price:
                type: integer

  - do:
      bulk:
        refresh: true
        body:
          - index:
              _index: bicycles
              _id: 1
          - id: 1
            status: up
            price: 100
          - index:
              _index: bicycles
              _id: 2
          - id: 2
            status: up
            price: 150
          - index:
              _index: bicycles
              _id: 3
          - id: 3
            status: down
            price: 200

  - do:
      search:
        index: bicycles
        body:
          size: 0
          aggs:
            agg:
              histogram:
                field: price
                interval: 50
              aggs:
                0-bucket:
                  filter:
                    bool:
                      filter:
                        - bool:
                            filter:
                              - bool:
                                  filter:
                                    - bool:
                                        should:
                                          - term:
                                              id: 1
                                        minimum_should_match: 1
                                    - bool:
                                        should:
                                          - term:
                                              # Ignored.  Setting to 'bicycles' turns the filter to MatchNone.
                                              cycle_type: bicycle   
                                        minimum_should_match: 1
                              - bool:
                                  should:
                                    - term:
                                        status: up
                                  minimum_should_match: 1

I wonder if the synthetic source or the use of timestamp filtering over many indexes contribute to the root cause.

[salvatore-campagna]

I think that behaviour might be a result of this change #96161.

It is not clear to me if you are trying to test this avoiding the rewrite. In case you would like to skip rewriting the query to test the behaviour without rewriting, you can act on the pre-filter shard size.

See SearchRequest::setPreFilterShardSize.

Basically by setting a pre-filter shard size that is large enough you will skip rewriting of the query.

[kkrik-es]

After further discussion with the team, this is actually a legit repro: the filter is converted to a MatchNone but we still process every doc. We need to investigate if we can either move the filter into the query, or just skip doc processing in there's no possible match.

[martijnvg]

I think the terms agg part hasn't been implemented yet of this issue?

[kkrik-es]

I checked simple term aggregations on const_keyword, the TermsAggregator uses StringTermsAggregatorFromFilters that is adapter agg wrapping FilterByFilterAggregator. In this case, the submitted fix should also apply - though I'm not sure how this can get accelerated since the query gets rewritten to MatchAllDocsQuery.

Do we have a good use case here, where the terms aggregation can be further improved?

[kkrik-es]

Closing for now, we can reopen if we have good use cases to cover.