Add support for API keys to access Elasticsearch

[bizybot]

X-Pack security supports built-in authentication service token-service that allows
access tokens to be used to access Elasticsearch without using Basic authentication.
The tokens are generated by token-service based on OAuth2 spec. The access token
is a short-lived token (defaults to 20m) and refresh token with a lifetime of 24 hours,
making them unsuitable for long-lived or recurring tasks where the system might go
offline thereby failing refresh of tokens.

This commit introduces a built-in authentication service api-key-service that
adds support for long-lived tokens aka API keys to access Elasticsearch. The api-key-service
is consulted after token-service in the authentication chain. By default, if TLS is enabled
then api-key-service is also enabled. The service can be disabled using the configuration
setting.

The API keys:-

• by default do not have an expiration but expiration can be configured
  where the API keys need to be expired after a certain amount of time.
• when generated will keep authentication information of the user that
  generated them.
• can be defined with a role describing the privileges for accessing
  Elasticsearch and will be limited by the role of the user that generated
  them
• can be invalidated via invalidation API
• information can be retrieved via a get API
• that have been expired or invalidated will be retained for 1 week before
  being deleted. The expired API keys remover task handles this.

Following are the API key management APIs:-

1. Create API Key - PUT/POST /_security/api_key
2. Get API key(s) - GET /_security/api_key
3. Invalidate API Key(s) DELETE /_security/api_key

The API keys can be used to access Elasticsearch using Authorization
header, where the auth scheme is ApiKey and the credentials, is the
base64 encoding of API key Id and API key separated by a colon.
Example:-

curl -H "Authorization: ApiKey YXBpLWtleS1pZDphcGkta2V5" http://localhost:9200/_cluster/health

Closes #34383

[elasticmachine]

Pinging @elastic/es-security

[tvernum]

I haven't reviewed enough of the API Key PRs to be able to say whether this is ready or not, but I have no objections.

[jaymode]

LGTM

[jkakavas]

I didn't go through the whole changeset here but I've reviewed enough of the individual PRs so LGTM

[karmi]

I was recently adding support for the API Key authentication to the Go client (elastic/go-elasticsearch@3b68c0f), and I think there's a small, but nice usability optimization: it would help if the API would return also the base64-encoded value for the key, so the users (or the client) doesn't have to perform the encoding themselves.

[albertzaharovits]

@karmi I see in the ES code that the key (api_key field in the response to POST _security/api_key) is base64 url encoded. Are we talking about the same thing?

[karmi]

@albertzaharovits , what I mean is the output (ie. result) of the "Create API Key" API:

docker run \
  --env "discovery.type=single-node" \
  --env "xpack.security.enabled=true" \
  --env "xpack.security.authc.api_key.enabled=true" \
  --env "xpack.license.self_generated.type=trial" \
  --env "ELASTIC_PASSWORD=elastic" \
  --publish 9200:9200 \
  --rm -it \
  docker.elastic.co/elasticsearch/elasticsearch:8.0.0-SNAPSHOT

curl -s -X POST \
        -H 'Content-Type: application/json' \
        http://elastic:elastic@localhost:9200/_security/api_key -d \
        '{"name":"test-key", "role_descriptors":{}}' \
| jq '.'

Output:

{
  "id": "ZquG-WsBx1ePiPP47oig",
  "name": "test-key",
  "api_key": "ipRxILiWT7udFhKgb69O4Q"
}

Ie. the API returns the id and api_key value, but not the base64-encoded version which has to be passed in the Authorization header. Is it more clear?

[albertzaharovits]

Thank you @karmi .
I understand that in this case you have to concatenate id with : and api_key , encode base64, and then append to ApiKey to get the value of the Authorization header. And you are asking why isn't there a value in the response that I can put in the header as is.

I was under the impression that ApiKey concatenated with $api_key was the way to build the credentials in this case. But I am wrong.

@bizybot Do you think it would be feasible to achieve what @karmi is asking for? Either by adding another field in the response or make ES not require the id to be concatenated to the api_key, and only use the api_key?

[karmi]

And you are asking why isn't there a value in the response that I can put in the header as is.

Yes, exactly — with the goal of increasing the usability. If the response contains the fully-encoded value, the user can just copy&paste that value into the client in a single step.

Either by adding another field in the response (...)

Yes, that's what I had in mind — adding a property like value, encoded, credentials (not sure about the naming in this case...) in addition to existing properties.

[bizybot]

Hi @karmi and @albertzaharovits,
I think this is feasible, we could indeed send out encoded_credentials, I do not see a problem there.
I think the same would be useful in case of native user creation as well, though we never sent credentials out. I like this as it could be useful in our REST integration test cases where we can just parse the value from the response and use it later in the next calls. We can create a discuss issue and take it forward if everyone agrees, wdyt?
Thank you for bringing up @karmi and thanks for the reminder for it @albertzaharovits. I had seen this on mobile and then I could not remember where the message came from as this was a closed PR.

[karmi]

We can create a discuss issue and take it forward if everyone agrees, wdyt?

Yes, please! Thanks for catching it, @bizybot !

[tvernum]

make ES not require the id to be concatenated to the api_key, and only use the api_key

This part is not feasible. The id is needed so that we can find the document in the index so that we can check the hash. (We used salted hashes when storing the api_key, so it is not possible to do a search using only the key, and not the id).