Show ignore_above in the Indices settings

[timroes]

In the indices settings there is only a checkmark if a field is indexed. That looks like: "This field is indexed, I can search for stuff in it."

I have seen several cases where people wondered why some of the searches on this field didn't find the documents they were looking for and the problem was, that ignore_above was set for this field, and the content of that specific document(s) they were looking for, was longer than the limit and not indexed (most often talking about stacktraces stored in ES).

It would be nice if the settings screen make this somehow more clear to the user, that some values (above a specific length) are not indexed in that field and won't be found by Elasticsearch when searching for something in that fields.

If you don't know that behaviour it is very hard to understand why e.g. I can search for a class name in my not so long stacktrace and it will find the stacktraces, but if I search for a classname that only appears in longer stacktraces it won't find the documents.

Distinction from #4801: That issue wants to warn when the user searches for something longer than ignore_above, I would like to show the user in the index settings, that some values might not be indexed and therefor not findable (even if she searches for something shorter than the ignore_above limit).

Small idea how it could look: showing a superscript info or warning icon with the number of ignore_above behind it and show a tooltip with information when the user hover over it.

[epixa]

This seems totally reasonable to me. Since the ignore_above value can be changed in the mappings at any time, we would need to make sure we craft the warning message to make it clear that the configuration was present at the time the mappings were cached.

[jimgoodwin]

@w33ble did we decide this was more complex to do than as stated ?

[djptek]

Hi @timroes, saw this was a candidate for Kibana 7 then not implemented. @elastic/ecs were discussing how exposing this field in the UI might be useful in helping users make the right storage choice for their data in e.g. Security (unset ignore_above) vs Logging (set ignore_above) use cases.

Would this be an appropriate time to revisit?

[djptek]

@jimczi ⬆️

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[timroes]

@djptek I am not sure if we would still want to persue the issue as it's outlined here. We're planning to move warning for not indexed too long field values into Discover (see discussion in #101232). So users will be able when viewing their data directly in Discover see if specific values got ignored. Do you feel there is an additional usage in showing that value still in the index pattern UI?

[djptek]

Hi @timroes this has come up because formerly ECS has automatically added a value for this field under certain circumstances, to encourage best practice for the Logging use case, however, it turns out this is potentially a very bad thing for the Security use case so we're removing that see elastic/ecs#1483

So, more than showing the value of ignore_above the goal is to give the user visibility of this field and agency should they wish to set/unset the field according to their use case.

[elasticmachine]

Pinging @elastic/kibana-stack-management (Team:Stack Management)

[cjcenizal]

@timroes Can you specify which UI you're referring to? Is it the Index Patterns UI, Index Management, or something else?

[timroes]

I was referring to the index patterns UI (the only thing that existed 2016 :D) Thus the previous appServices label was imho correct.

That said, as in my above comment mentioned, I am not sure if that's really anything we still want to do that. ignore_above can be changed in the mapping thus showing it in the index pattern UI might not be that good, since it's not necessarily reflecting the state of all indexed documents.

We're currently working on showing ignored values in Discover (since with the switch to the fields API this has become mandatory), see #101232 and elastic/elasticsearch#74121

At that point we'd show wawrnings on values in Discover directly if they weren't indexed, and thus are not searchable (since it theoretically can even apply to just one value within an array of values in one document). I think that is the more desired approach to go with, so from my side there is no need for this feature anymore. Please feel free to close this if you agree.