[Fleet] Investigate usage of prefix queries in Fleet setup

[joshdover]

Kibana version: 8.4.3

Elasticsearch version: 8.4.3

Elasticsearch has a setting called search.allow_expensive_queries which when set to false will disable several types of queries from being executed: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html

We've had reports that when this setting is enabled, Fleet's setup process fails with an error like:

Kibana Fleet setup failed: http POST request to http://XXXX.containerhost:9244/api/fleet/setup fails: 
[search_phase_execution_exception: [exception] Reason: [prefix] queries cannot be executed when 'search.allow_expensive_queries' is set to false.
 For optimised prefix queries on text fields please enable [index_prefixes].]: all shards failed: %!w(<nil>). 
Response: {"statusCode":400,"error":"Bad Request","message":"[search_phase_execution_exception: [exception] Reason: [prefix] queries cannot be executed when 'search.allow_expensive_queries' is set to false. For optimised prefix queries on text fields please en... (truncated)

This will block the Integrations Server from starting up in ESS & ECE.

Fleet should continue to work when expensive queries are disabled. We should investigate where we're using prefix queries and see if we can either eliminate this usage pattern or use better mappings (eg. wildcard) to avoid this issue. First priority should be fixing the setup process, but more generally we should avoid this pattern entirely.

I did a quick search and didn't find anything obvious in the setup code. One area to investigate further would be the usage of KQL/kuery. It's possible a KQL query like field:"foo*" produces a prefix query when transpiled to ES Query DSL.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[hop-dev]

In trying to reproduce this I got a slightly different error:

[2022-10-18T11:58:49.342+01:00][ERROR][plugins.fleet] all shards failed: search_phase_execution_exception: [exception] Reason: Cannot search on field [epm-packages.keep_policies_up_to_date] since it is not indexed and 'search.allow_expensive_queries' is set to false.

Coming from this query:

kibana/x-pack/plugins/fleet/server/services/managed_package_policies.ts

Line 41 in 277201e

filter: `${PACKAGES_SAVED_OBJECT_TYPE}.attributes.install_status:installed AND ${PACKAGES_SAVED_OBJECT_TYPE}.attributes.keep_policies_up_to_date:true`,

[joshdover]

Interesting, it's likely that should be fixed too 😄

[javanna]

That last error looks like it's due to querying a field that has index:false but doc_value:true. Can you double check? If the field is queried often, it would make sense to index it, as doc_value only fields are slower to query.

[jlind23]

@joshdover is this still relevant or should I close that issue?

[joshdover]

It may still be an issue, but there's been further discussion in elastic/elasticsearch#90898 (comment) which points to this ES setting being reconsidered overall.

I think we should probably still look at the issue that Mark found above though, as that is likely a query and it affects Fleet setup which is performance sensitive (blocks Kibana startup and rendering of the Fleet UI).

[ThomThomson]

Has there been any movement on this? Currently the new Controls work well with allow_expensive_queries off, but we're considering migrating to use prefix queries instead of terms include for prefix searching within Keyword mapped fields. I'm getting the sense from ES folks that terms include is roughly as expensive as prefix, and that it can be a little arbitrary which query types require that setting to be on.

Switching to prefix queries would greatly reduce our code complexity. The only risk is that this might prevent the Dashboard UI from loading when the allow_expensive_queries is off - just like the Fleet UI fails to load. Have users complained about fleet not loading when this setting is off? Is turning off this setting common at all?

Additionally, I've been unable to find any methods which can tell Kibana plugins the value of this setting - is it even possible to check whether or not it's on in order to change pieces of the UI?

[joshdover]

@ThomThomson I think your feedback here may be better suited on the ES issue to get the right eyes on this: elastic/elasticsearch#90898

[ThomThomson]

Good point, I will copy my response over there.