search.allow_expensive_queries + wildcard application privs can fail authorization

[jakelandis]

Elasticsearch Version

Tested against 8.6.0

Installed Plugins

No response

Java Version

bundled

OS Version

Ubuntu

Problem Description

When search.allow_expensive_queries is set to false some queries may fail since they are considered expensive. Typically there is an obvious connection between the action you are performing and any errors raised due to expensive queries. For example, if you explicitly issue a prefix query and search.allow_expensive_queries is set to false there is a natural correlation between your request and the error.

It has been found that when the following conditions are met

• The API key or role has application level privileges
• One of the application level privileges has a trailing wildcard in the application name (i.e. myapp*)
• The API key or role is not cached

then authentication will fail due to an error [1] related to the expensive query. This because internally the code issues a prefix query to for the application name. The end result is the API key can longer be used for authentication for any action.

Workaround include :

• set search.allow_expensive_queries is set to true
• remove the * from the application privilege name

It is also possible to set search.allow_expensive_queries to true make a call with the API key or user with the role so that it gets cached, then set search.allow_expensive_queries back to false . That will cause the API key or role to be cached, and it will work after that, but it can also start to fail again at what would seem a random time if ever evicted from the cache or the node gets restarted. For this reason, that work around is not suggested.

[1] Example exception. Note - other similar exceptions can occur but they all reference search.allow_expensive_queries and the .security-* index

Caused by: [.security-7/9mbNjDCNRzSzrDtj2YEeKQ] org.elasticsearch.index.query.QueryShardException: failed to create query: [prefix] queries cannot be executed when 'search.allow_expensive_queries' is set to false. For optimised prefix queries on text fields please enable [index_prefixes].
	at org.elasticsearch.index.query.SearchExecutionContext.toQuery(SearchExecutionContext.java:548)
	at org.elasticsearch.index.query.SearchExecutionContext.toQuery(SearchExecutionContext.java:531)
	at org.elasticsearch.search.SearchService.parseSource(SearchService.java:1174)
	at org.elasticsearch.search.SearchService.createContext(SearchService.java:986)
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:630)
	at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:495)
	at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50)
	at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47)
	at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:917)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.lang.Thread.run(Thread.java:1589)

Steps to Reproduce

PUT _cluster/settings
{
    "persistent": {
        "search.allow_expensive_queries": "false"
    }
}

POST _security/api_key
{
  "name": "my_api_key",
  "role_descriptors": {
    "role1": {
      "cluster": ["all"],
      "indices": [
        {
          "names": ["test"],
          "privileges": ["write"]
        }
      ],
      "applications": [
        {
          "application": "app*",
          "privileges": ["read", "write"],
          "resources": ["*"]
        }
      ]
    }
  },
  "expiration": "1d"
}

using API key just created...

POST _bulk 
{ "index" : { "_index" : "test", "_id" : "4" } }
{ "field1" : "value1" }

EDIT: Updated text to suggest this can happen with an API key or a custom role

[elasticsearchmachine]

Pinging @elastic/es-security (Team:Security)

[n1v0lg]

I think this applies more broadly than just to API keys, i.e., regular users (and roles) as well. As long as there is a role with the right application privilege descriptors, we will run into this. I haven't tested it, just basing this on what I've seen in the code.

There are other examples where turning off expensive queries breaks Security APIs (e.g., the QueryApiKey API may not work because we use a runtime field to flatten metadata, possibly token refresh functionality too (?)), and likely other places outside of security.

I think we should document this limitation, e.g., here: https://www.elastic.co/guide/en/elasticsearch/reference/current/security-limitations.html and also more broadly here: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html. If we want to solve this beyond better docs, I think we might want to aim for something general rather than specific to just application privileges (e.g., along the lines of #53607). I'll leave more thoughts on this tomorrow.

[ywangd]

This is a long standing issue. As @n1v0lg said, it affects roles as well, not just API keys. More importantly, we have reserved roles that use wildcard for application names, e.g.:

elasticsearch/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/store/ReservedRolesStore.java

Line 398 in 1cc1d12

.application("kibana-*")

So users can hit this issue even without creating any custom roles or API keys. We talked about this couple times in the past. I think we all agree it is an issue because the breakage is unintuitve. But haven't really allocate time to address it partly because we are not sure whether it should be addressed in a broader scope of how allow_expensive_queries setting should work. There have been different proposals including the one Nikolaj linked and more recently #90898

I do think we should fix the issue in the narrow context of security usage because it is unknown when the broader expensive query issue can be resolved. The solution we discussed before is to filter it in memory when allow_expensive_queries is disabled. It is not going to be "inexpensive" but it does avoid the problem.

There are other examples where turning off expensive queries breaks Security APIs (e.g., the QueryApiKey API may not work because we use a runtime field to flatten metadata, possibly token refresh functionality too (?)), and likely other places outside of security.

AFAIK, this is the only place where security internally uses expensive queries. API Key metadata is flattened field type which is not expensive. Users can query API keys with expensive queries. But in that case, they explicitly request the expensive operations. It is not the same as the case for NativePrivilegesStore where we internally uses prefix queries for role resolution without users being aware of it.