[Management] Field properties don't seem to match es API response

[stacey-gammon]

I have an index with fields system.fsstat.total_size.free which is showing up in Kibana, under management, as a field that is not searchable or aggregatable, but when I query es, it doesn't have index: false.

Below, compare system.fsstat.total_size.free with system.filesystem.used.bytes. Notice .bytes is searchable and aggregatable but .free is not.

Now compare the es responses from both:

GET metricbeat-2016.11.22/_mapping/metricsets?filter_path=*.mappings.*.properties.system.*.fsstat.*.total_size.properties.free
returns:

  "metricbeat-2016.11.22": {
    "mappings": {
      "metricsets": {
        "properties": {
          "system": {
            "properties": {
              "fsstat": {
                "properties": {
                  "total_size": {
                    "properties": {
                      "free": {
                        "type": "long"
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

and GET metricbeat-2016.11.22/_mapping/metricsets?filter_path=*.mappings.*.properties.system.*.filesystem.*.used.*.bytes also returns something very similar:

  "metricbeat-2016.11.22": {
    "mappings": {
      "metricsets": {
        "properties": {
          "system": {
            "properties": {
              "filesystem": {
                "properties": {
                  "used": {
                    "properties": {
                      "bytes": {
                        "type": "long"
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

I tried to compare this with a couple new fields I created via

POST test-index/_mapping/type
{
  "properties": {
    "age": {
      "type": "long"
    },
    "noindexnum": {
      "type": "long",
      "index": "no"
    }
  }
}

and when querying the mapping via GET test-index/_mapping/type , the es response returns:

{
  "test-index": {
    "mappings": {
      "type": {
        "properties": {
          "age": {
            "type": "long"
          },
          "noindexnum": {
            "type": "long",
            "index": false
          }
        }
      }
    }
  }
}

and kibana correctly shows:

@djschny

[stacey-gammon]

I've tracked this down to the _field_stats api, the query GET metricbeat-2016.11.22/_field_stats?fields=system.f* is not returning anything for system.fsstat fields, but does return the system.filesystem fields.

[stacey-gammon]

I think this is because there is no data in my index for the system.fsstat fields, so the field_stats api doesn't return anything. This is somewhat misleading, so I still think it's a bug.

[jimczi]

@stacey-gammon the field_stats api does not return anything for fields that are not in the index yet. You need to index at least one document for that field to get information about it using the field_stats API. Maybe a small icon saying that this field is currently empty could help ?

[stacey-gammon]

I think the right fix is for Kibana to use the field_mapping api to get information like aggregatgable, searchable, etc on the fields, since this does return information on non-indexed fields. Then perhaps fallback to the field_stats api if more information is required (I talked a little bit with @djschny about this, but forget now what information is required from field_stats that field_mapping doesn't have).

[Bargs]

@stacey-gammon is_searchable and is_aggregatable are computed properties that only field stats provide, they're not returned by the field mapping api. We asked the ES team to add these so that Kibana doesn't have to determine what it means to be "searchable" or "aggregatable" based on mapping info.

[Bargs]

Our plan way back when was actually to remove the use of the field mapping api completely and rely only on field stats (see step 3 here) so that we wouldn't have to do the messy normalization and de-duplication ourselves.

[stacey-gammon]

Related: elastic/elasticsearch#22438

[tsg]

I added the :Beats label to this, because this can make the sample dashboards fail if the user refreshes the index and doesn't have values for all fields.

[chrisronline]

I think this is addressed in the change to using field_caps.

PUT test-index
{
  "mappings": {
    "doc": {
      "properties": {
        "age": {
          "type": "long"
        },
        "height": {
          "type": "long"
        }
      }
    }
  }
}

POST test-index/doc/1
{
  "age": 31
}

@stacey-gammon Can you confirm when you have the chance?