terms stats facet not accurate when dealing with high cardinality fields

[lokivog]

I've experience a scenario where the terms stats facet was not calculating totals accurately. After investigation, I found that this facets has an additional property named shard_size which will determine how many term entries will be requested from each shard. The terms stats facet documentation states:
"When dealing with field with high cardinality (at least higher than the requested size) The greater shard_size is - the more accurate the result will be (and the more expensive the overall facet computation will be). shard_size is there to enable you to increase accuracy yet still avoid returning too many terms_stats entries back to the client."

I would like to add the shard_size property as an option to the terms facet panel to account for data that requires accurate calculations.

[lokivog]

Here is a trivial example that demonstrates the issue. When given the following data-set

{ "name": "Ten", "value": 1}
{ "name": "Ten", "value": 1}
{ "name": "Ten", "value": 1}
{ "name": "Ten", "value": 1}
{ "name": "Ten", "value": 1}
{ "name": "Ten", "value": 1}
{ "name": "Ten", "value": 1}
{ "name": "Ten", "value": 1}
{ "name": "Ten", "value": 1}
{ "name": "Ten", "value": 1}
{ "name": "Nine", "value": 1}
{ "name": "Nine", "value": 1}
{ "name": "Nine", "value": 1}
{ "name": "Nine", "value": 2}
{ "name": "Nine", "value": 2}
{ "name": "Nine", "value": 2}
{ "name": "Eight", "value": 1}
{ "name": "Eight", "value": 1}
{ "name": "Eight", "value": 1}
{ "name": "Eight", "value": 1}
{ "name": "Eight", "value": 2}
{ "name": "Eight", "value": 2}
{ "name": "Seven", "value": 1}
{ "name": "Seven", "value": 1}
{ "name": "Seven", "value": 1}
{ "name": "Seven", "value": 1}
{ "name": "Seven", "value": 1}
{ "name": "Seven", "value": 1}
{ "name": "Seven", "value": 1}
{ "name": "Six", "value": 6}
{ "name": "Five", "value": 3}
{ "name": "Five", "value": 2}
{ "name": "Four", "value": 1}
{ "name": "Four", "value": 1}
{ "name": "Four", "value": 1}
{ "name": "Four", "value": 1}
{ "name": "Three", "value": 3}
{ "name": "Two", "value": 1}
{ "name": "Two", "value": 1}
{ "name": "One", "value": 1}

I've made the changes in a fork and here is an screen-shot depicting the data-set with the following:
size = 10, shard_size not set
size = 3, shard_size not set
size = 3, shard_size = 10

[lokivog]

Here is the edit panel. I've added tool tips to the size and accuracy settings to help explain the meanings of them to the user.

[lokivog]

Please let me know if you need anymore information regarding this issue. If not, is it possible to have the pull request merged?

[rashidkpc]

This is a factor of the way elasticsearch does the distributed count. Unfortunately there is no way for Kibana to fix this. Elasticsearch issue here: elastic/elasticsearch#1305

[EmilyZhangHui]

When updating the length to 10, the terms pannel can't show the correct mean value.