Long lived api keys/tokens

[jaymode]

The security codebase already supports the use of access tokens for
interactive access. These tokens are based on the OAuth2 spec; there is
a short-lived access token and a refresh token with a lifetime of 24 hours.
These tokens are not suitable for Hadoop jobs or recurring scheduled tasks.

Hadoop jobs typically require a token that may need to be refreshed but
the token needs to remain the same after refresh; our current tokens
always generate a new access and refresh token upon refresh. Another
pattern of Hadoop jobs is that once they are complete the tokens are
not invalidated but instead rely on a expiration time of something like
a week.

The security tokens are not a fit for recurring/scheduled jobs as they
can only be refreshed for 24 hours. If the system that would refresh the
token is down for greater than 24 hours, which could be possible over a
weekend or holiday, the jobs would require updating to function again.

After discussion, we've settled upon adding long lived tokens that would
be akin to API tokens. The properties of these tokens are as follows:

• Tokens will store the authentication information of the user that
  created them
• By default, tokens have no expiration. A cluster level setting will
  be available to specify a maximum lifetime
• When creating a token, a maximum lifetime can be specified for cases
  where the token is expected to stop working after a certain amount of
  time

In terms of APIs, we will be adding:

Create a token with optional expiration, which is a TimeValue.

POST /_security/token
{
  "expiration": "7d"
}

List security tokens. Optionally restrict by username and/or realm.

GET /_security/token
{
  "username": "john.doe",
  "realm": "ldap1"
}

List specific token

GET /_security/token/<token>

Remove a token.

DELETE /_security/token/<token>

In order to use these tokens, they will be passed over HTTP using the
Authorization header. The value must be prefixed with Bearer .

In order to use this token, security must take the value and find the
document with this id. If the document does not exist or the search fails,
the token may not be used. Once security has retrieved a document with
this ID, it must make sure to verify the token has not expired.

Invalidation or a removal of a token should remove the token document
from the security index.

Tokens must store enough information to re-build authentication and
support determining if the token has expired.

Future considerations:

• The ability to limit the scope of a token
• keep track of last used time

[elasticmachine]

Pinging @elastic/es-security

[jaymode]

@jbaiera and @chrisdavies FYI. Let me know if you see any issues with this proposal.

[jasontedor]

@martijnvg @ywelsch Let us watch this issue in the context of CCR where we would consider using these tokens instead of the current approach of associating authorization headers with the persistent task.

[jbaiera]

• By default, tokens have no expiration. A cluster level setting will
  be available to specify a maximum lifetime
• When creating a token, a maximum lifetime can be specified for cases
  where the token is expected to stop working after a certain amount of
  time

Are we thinking that the cluster setting for maximum token lifetime will be the absolute maximum even if a client requests a longer lifetime? So if the maximum TTL for a token is set at 7 days in the cluster settings, and a client requests a token that lives for 1 month, will the client receive a token that expires in 7 days or 1 month?

Not sure if this was already assumed, but when we receive a token, it should probably denote its expiration date and time. I didn't see any example responses, so I figured I would put that here.

Final thought - It might make sense to allow clients to name the tokens to make interacting with and managing responses from the token api easier.

[jaymode]

Are we thinking that the cluster setting for maximum token lifetime will be the absolute maximum even if a client requests a longer lifetime?

Yes.

So if the maximum TTL for a token is set at 7 days in the cluster settings, and a client requests a token that lives for 1 month, will the client receive a token that expires in 7 days or 1 month?

I'd prefer for this to be an error condition, but given the variability in what a user could define the client would receive a token that expires in 7 days.

Not sure if this was already assumed, but when we receive a token, it should probably denote its expiration date and time

Makes sense.

Although the more I think about this, I am contemplating leaving out this ability in the first iteration and see what kind of feedback we get.

Final thought - It might make sense to allow clients to name the tokens to make interacting with and managing responses from the token api easier.

++

[jaymode]

The ability to define scoped access is allowed by providing a role definition with the token but is not necessarily a role that can retrieved using another mechanism. In order to support this, we need to think about how this role is handled in a few different scenarios:

1. Same node
2. Node to node in the same cluster
3. Node to remote node

The current thinking for 1 is that the role would be loaded by the service that loads the token. We need a way to get this role into the authorization service. The ThreadContext is the quick way out but I'd like to avoid using this as it feels like a mistake. We may be able to make use of metadata on the AuthenticationResult object that was recently added.

For the node to node scenario, there is a question of whether the receiving node should perform the loading of the role or if it should be serialized with the request and token. We've never serialized a role like this before, so we would be adding to the data that one nodes trusts from the other.

In terms of cross cluster, we've always just sent the role names and relied on the remote cluster looking this up. I think we set a dangerous precedent if we decide to serialize the role. However, if we do not serialize the role then these tokens cannot be used for cross cluster jobs, which hinders their use significantly.

[tvernum]

Invalidation or a removal of a token should remove the token document from the security index.

It's an implementation detail, but I think we should keep the tokens in an "invalidated" state for a short period (~24h?).
My two reasons for that are:

• Forensics. If invalidating a token removes all record of it, then it might be used by someone to cover their tracks. 24h is long enough for an admin to generate audit logs of whatever they think they need to retain.
• Simple debugging. It's nicer to be able to diagnose (in the server logs) that a job attempted to use a token after it was "deleted" as opposed to using a token that never existed.

[kobelb]

Is it safe to assume that we'd be able to use an existing token in the situation of SAML to get a long-lived token?

[jkakavas]

I would add a future consideration for the possibility to invalidate all long lived tokens of user or realm as we've discussed for the oAuth tokens ( currently tracked in #34556 )

[jaymode]

Is it safe to assume that we'd be able to use an existing token in the situation of SAML to get a long-lived token?

Yes. Given how we use the short lived tokens, I think we need this

[bizybot]

By default, tokens have no expiration. A cluster level setting will
be available to specify a maximum lifetime

I think instead of no expiration, we can go with a pre-defined expiration ( like 7 days or something) and the configuration setting allows option to say no expiration if one wants to use it.

[jaymode]

Can you add details about why you feel that way? My current thinking is to leave the setting out completely. There are use cases where some users may not want an expiration; an example of existing work would be github personal access tokens. The reason I am considering that the setting can be left out initially is because it can cause issues with Kibana task manager and hadoop if the duration is too short; additionally leaving it out also simplifies the implementation.

[bizybot]

Well, the reason I have is mostly that these are,not API keys but API tokens which we are using for authentication. For me giving out expiring tokens by defaults seems to minimize the threat on leaks and theft. Even if we plan to add setting this in the later version, the customer's who will be using the version with non-expiring API tokens can be at risk. I agree the different scenarios demand different expiration, and we would want all of these scenarios to be supported, so if possible I would go with a simple setting that allows it to be controlled.

[albertzaharovits]

In terms of cross cluster, we've always just sent the role names and relied on the remote cluster looking this up. I think we set a dangerous precedent if we decide to serialize the role. However, if we do not serialize the role then these tokens cannot be used for cross cluster jobs, which hinders their use significantly.

I think it's fine to confine the API key to the local cluster. Sure it would be wonderful to generate an API key and use it over a group of clusters, similar to cross-cluster search. But this is stretching cross-cluster search. With cross-cluster search you have to think about the roles on the remote cluster: What roles make sense in the remote cluster and what is to be their definition. This is important because index schema could differ on the remote cluster so role definition should also differ.
Whereas with scoped api we would effectively enforce the role definition on the remote cluster.

Moreover I view these api keys on the same rank with users as they have permissions attached to them. Generating an api key usable for cross cluster searches would be akin to generating a user on the remote cluster, which the user generating the api key might not have privileges to do (not that we want to do this in the first place, or have the means to do so).

If we really wish that long running jobs be capable to cross-cluster search, then we should add the capability to pair api keys, such that an api key from remote is translated to an api key from local.

[bizybot]

Subtasking so we can track the progress:-

• Create an API key
  • API Key service generates API key and stores it in the index.
  • Rest API
  • HLRC changes for the API
• Authenticate using the API key
  new Authentication scheme to handle authentication of the API key
• Authorization for the API key
• Invalidate API key
  Rest API to invalidate key(s) and HLRC changes for the API.
• Get API key
  • Rest API
  • HLRC changes for the API