Fail to create aggregate on a field with a mixed numeric data type

[stacey-gammon]

To Repro:

Create an index with an int data type:

POST test-int/type
{
  "duration": 5,
  "timestamp": "2017-06-11T17:12:52.128Z"
}

Create another index with a float data type:

POST test-double/type
{
  "duration": 5.5,
  "timestamp": "2017-06-11T18:12:52.128Z"
}

In kibana, create the index pattern that picks up a single one: test-int as well as one that picks up both: test-*.
You can see that the field duration in test-int is marked as aggregatable and searchable but the same field duration in test-* is not marked as such.

The issue is that the field comes back as two different types depending on the index:
GET /test-*/_field_caps?fields=duration yeilds:

{
  "fields": {
    "duration": {
      "float": {
        "type": "float",
        "searchable": true,
        "aggregatable": true,
        "indices": [
          "test-double"
        ]
      },
      "long": {
        "type": "long",
        "searchable": true,
        "aggregatable": true,
        "indices": [
          "test-int"
        ]
      }
    }
  }
}

This is not actually a problem for elasticsearch, as you can see it has no problem with the aggregation:

GET /test-*/type/_search 
{
  "size": 0,
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "*",
            "analyze_wildcard": true
          }
        },
        {
          "range": {
            "timestamp": {
              "gte": 1497200746003,
              "lte": 1497201520102,
              "format": "epoch_millis"
            }
          }
        }
      ],
      "must_not": []
    }
  },
  "_source": {
    "excludes": []
  },
  "aggs": {}
}

but for some reason, Kibana can't handle it and won't allow you to create a visualization using that field.

Background: https://discuss.elastic.co/t/variation-of-the-field-is-a-required-parameter-error/89081

I suspect this is the cause for some of the more frequent encounters in 5.4 we are seeing of the "field is a required parameter" error mentioned in #9571

[Bargs]

What does field_stats and/or field_caps say about the field? searchable/aggregatable come straight from ES, so it may be a false negative coming from ES.

[stacey-gammon]

Look at the discuss ticket linked, it had field_caps response. Send back true twice looks like, for each type, so maybe we aren't interpreting that response correctly

[Bargs]

Ah I see, it does look that way. @spalger do you know why that might be the case?

[Bargs]

Errr... I don't know what I was thinking yesterday. The switch to field_caps didn't happen until 5.5. So if this is happening in 5.4, it's because field_stats is responding with a conflict (as shown in the discuss ticket)

[Bargs]

Ok, so I tested both 5.4.1 and 5.5.0 to be sure. In 5.4.1 _field_stats is definitely reporting a conflict for that field, so we don't treat it as aggregatable. In 5.5.0 there is no "conflict" reported by _field_caps, but since it reports multiple types we're treating it as a conflict. I'm not 100% sure what the correct fix is here. @spalger can you provide any background on the algorithm for parsing field_caps? I'm guessing there's no way for us to know whether two given types will conflict or not based on the info field_caps returns?

[chrisronline]

I'm fairly certain this was fixed earlier.

cc @Bargs @stacey-gammon

Can you confirm?

[Bargs]

I haven't tested the fix, but this is definitely the same issue that was being addressed in that PR.

[chrisronline]

K. I'm going to close this for now. Please reopen if it wasn't addressed