Implement Server-Side sessions

.github/CODEOWNERS

@@ -174,6 +174,17 @@
 /x-pack/plugins/security/  @elastic/kibana-security
 /x-pack/plugins/security/**/*.scss @elastic/kibana-core-ui-designers
 /x-pack/test/api_integration/apis/security/ @elastic/kibana-security
+/x-pack/test/encrypted_saved_objects_api_integration/ @elastic/kibana-security
+/x-pack/test/functional/apps/security/ @elastic/kibana-security
+/x-pack/test/kerberos_api_integration/ @elastic/kibana-security
+/x-pack/test/login_selector_api_integration/ @elastic/kibana-security
+/x-pack/test/oidc_api_integration/ @elastic/kibana-security
+/x-pack/test/pki_api_integration/ @elastic/kibana-security
+/x-pack/test/saml_api_integration/ @elastic/kibana-security
+/x-pack/test/security_api_integration/ @elastic/kibana-security
+/x-pack/test/security_functional/ @elastic/kibana-security
+/x-pack/test/spaces_api_integration/ @elastic/kibana-security
+/x-pack/test/token_api_integration/ @elastic/kibana-security
 
 # Kibana Localization
 /src/dev/i18n/  @elastic/kibana-localization

docs/settings/security-settings.asciidoc

@@ -96,16 +96,13 @@ The valid settings in the `xpack.security.authc.providers` namespace vary depend
 `<provider-type>.<provider-name>.showInSelector`
 | Flag that indicates if the provider should have an entry on the Login Selector UI. Setting this to `false` doesn't remove the provider from the authentication chain.
 
-|===
-
+2+a|
+[TIP]
 [NOTE]
 ============
 You are unable to set this setting to `false` for `basic` and `token` authentication providers.
 ============
 
-[cols="2*<"]
-|===
-
 | `xpack.security.authc.providers.`
 `<provider-type>.<provider-name>.accessAgreement.message`
 | Access agreement text in Markdown format. For more information, refer to <<xpack-security-access-agreement>>.
@@ -125,8 +122,8 @@ In addition to <<authentication-provider-settings,the settings that are valid fo
 | SAML realm in {es} that provider should use.
 
 | `xpack.security.authc.providers.`
-`saml.<provider-name>.maxRedirectURLSize`
-| The maximum size of the URL that {kib} is allowed to store during the authentication SAML handshake. For more information, refer to <<security-saml-and-long-urls>>.
+`saml.<provider-name>.useRelayStateDeepLink`
+| Determines if the provider should treat the `RelayState` parameter as a deep link in {kib} during Identity Provider initiated log in. By default, this setting is set to `false`. The link specified in `RelayState` should be a relative, URL-encoded {kib} URL. For example, the `/app/dashboards#/list` link in `RelayState` parameter would look like this: `RelayState=%2Fapp%2Fdashboards%23%2Flist`.
 
 |===
 
@@ -164,19 +161,27 @@ There is a very limited set of cases when you'd want to change these settings. F
 |===
 
 [float]
-[[login-selector-settings]]
-===== Login Selector UI settings
+[[login-ui-settings]]
+===== Login user interface settings
+
+You can configure the following settings in the `kibana.yml` file.
 
 [cols="2*<"]
 |===
+| `xpack.security.loginAssistanceMessage`
+| Adds a message to the login UI. Useful for displaying information about maintenance windows, links to corporate sign up pages, and so on.
+
+| `xpack.security.loginHelp`
+| Adds a message accessible at the login UI with additional help information for the login process.
+
 | `xpack.security.authc.selector.enabled`
-| Determines if the Login Selector UI should be enabled. By default, this setting is set to `true` if more than one authentication provider is configured.
+| Determines if the login selector UI should be enabled. By default, this setting is set to `true` if more than one authentication provider is configured.
 
 |===
 
 [float]
-[[security-ui-settings]]
-==== User interface security settings
+[[security-session-and-cookie-settings]]
+==== Session and cookie security settings
 
 You can configure the following settings in the `kibana.yml` file.
 
@@ -186,8 +191,7 @@ You can configure the following settings in the `kibana.yml` file.
   | Sets the name of the cookie used for the session. The default value is `"sid"`.
 
 | `xpack.security.encryptionKey`
-  | An arbitrary string of 32 characters or more that is used to encrypt credentials
-  in a cookie. It is crucial that this key is not exposed to users of {kib}. By
+  | An arbitrary string of 32 characters or more that is used to encrypt session information. Do **not** expose this key to users of {kib}. By
   default, a value is automatically generated in memory. If you use that default
   behavior, all sessions are invalidated when {kib} restarts.
   In addition, high-availability deployments of {kib} will behave unexpectedly
@@ -205,42 +209,33 @@ You can configure the following settings in the `kibana.yml` file.
   This is *not set* by default, which modern browsers will treat as `Lax`. If you use Kibana embedded in an iframe in modern browsers, you might need to set it to `None`. Setting this value to `None` requires cookies to be sent over a secure connection by setting `xpack.security.secureCookies: true`.
 
 | `xpack.security.session.idleTimeout`
-  | Sets the session duration. By default, sessions stay active until the
-  browser is closed. When this is set to an explicit idle timeout, closing the
-  browser still requires the user to log back in to {kib}.
-
-|===
+  | Ensures that user sessions will expire after a period of inactivity. This and `xpack.security.session.lifespan` are both
+highly recommended. By default, this setting is not set.
 
+2+a|
 [TIP]
 ============
-The format is a string of `<count>[ms|s|m|h|d|w|M|Y]`
-(e.g. '70ms', '5s', '3d', '1Y').
+The format is a string of `<count>[ms\|s\|m\|h\|d\|w\|M\|Y]` (e.g. '20m', '24h', '7d', '1w').
 ============
 
-[cols="2*<"]
-|===
-
 | `xpack.security.session.lifespan`
-  | Sets the maximum duration, also known as "absolute timeout". By default,
-  a session can be renewed indefinitely. When this value is set, a session will end
-  once its lifespan is exceeded, even if the user is not idle. NOTE: if `idleTimeout`
-  is not set, this setting will still cause sessions to expire.
-
-|===
+  | Ensures that user sessions will expire after the defined time period. This behavior also known as an "absolute timeout". If
+this is _not_ set, user sessions could stay active indefinitely. This and `xpack.security.session.idleTimeout` are both highly
+recommended. By default, this setting is not set.
 
+2+a|
 [TIP]
 ============
-The format is a
-string of `<count>[ms|s|m|h|d|w|M|Y]` (e.g. '70ms', '5s', '3d', '1Y').
+The format is a string of `<count>[ms\|s\|m\|h\|d\|w\|M\|Y]` (e.g. '20m', '24h', '7d', '1w').
 ============
 
-[cols="2*<"]
-|===
+| `xpack.security.session.cleanupInterval`
+| Sets the interval at which {kib} tries to remove expired and invalid sessions from the session index. By default, this value is 1 hour. The minimum value is 10 seconds.
 
-| `xpack.security.loginAssistanceMessage`
-  | Adds a message to the login screen. Useful for displaying information about maintenance windows, links to corporate sign up pages etc.
-
-| `xpack.security.loginHelp`
-  | Adds a message accessible at the Login Selector UI with additional help information for the login process.
+2+a|
+[TIP]
+============
+The format is a string of `<count>[ms\|s\|m\|h\|d\|w\|M\|Y]` (e.g. '20m', '24h', '7d', '1w').
+============
 
 |===

docs/setup/production.asciidoc

@@ -132,7 +132,7 @@ server.port
 
 Settings that must be the same:
 --------
-xpack.security.encryptionKey //decrypting session cookies
+xpack.security.encryptionKey //decrypting session information
 xpack.reporting.encryptionKey //decrypting reports
 xpack.encryptedSavedObjects.encryptionKey // decrypting saved objects
 --------

docs/user/security/authentication/index.asciidoc

@@ -59,8 +59,6 @@ For more information, refer to <<authentication-security-settings, authenticatio
 
 To successfully log in to {kib}, basic authentication requires a username and password. Basic authentication is enabled by default, and is based on the Native, LDAP, or Active Directory security realm that is provided by {es}. The basic authentication provider uses a {kib} provided login form, and supports authentication using the `Authorization` request header `Basic` scheme.
 
-The session cookies that are issued by the basic authentication provider are stateless. Therefore, logging out of {kib} when using the basic authentication provider clears the session cookies from the browser, but does not invalidate the session cookie for reuse.
-
 NOTE: You can configure only one Basic provider per {kib} instance.
 
 For more information about basic authentication and built-in users, see
@@ -71,8 +69,6 @@ For more information about basic authentication and built-in users, see
 
 Token authentication allows users to log in using the same {kib} provided login form as basic authentication, and is based on the Native security realm or LDAP security realm that is provided by {es}. The token authentication provider is built on {es} token APIs.
 
-The session cookies that are issued by the token authentication provider are stateful, and logging out of {kib} invalidates the session cookies for reuse.
-
 Prior to configuring {kib}, ensure token support is enabled in {es}. See the {ref}/security-api-get-token.html[{es} token API] documentation for more information.
 
 To enable the token authentication provider in {kib}, set the following value in your `kibana.yml`:
@@ -181,26 +177,6 @@ Basic authentication is supported _only_ if the `basic` authentication provider
 
 To support basic authentication for the applications like `curl` or when the `Authorization: Basic base64(username:password)` HTTP header is included in the request (for example, by reverse proxy), add `Basic` scheme to the list of supported schemes for the <<http-authentication,HTTP authentication>>.
 
-[float]
-[[security-saml-and-long-urls]]
-===== SAML and long URLs
-
-At the beginning of the SAML handshake, {kib} stores the initial URL in the session cookie, so it can redirect the user back to that URL after successful SAML authentication.
-If the URL is long, the session cookie might exceed the maximum size supported by the browser--typically 4KB for all cookies per domain. When this happens, the session cookie is truncated,
-or dropped completely, and you might experience sporadic failures during SAML authentication.
-
-To remedy this issue, you can decrease the maximum
-size of the URL that {kib} is allowed to store during the SAML handshake. The default value is 2KB.
-
-[source,yaml]
---------------------------------------------------------------------------------
-xpack.security.authc.providers:
-  saml.saml1:
-    order: 0
-    realm: saml1
-    maxRedirectURLSize: 1kb
---------------------------------------------------------------------------------
-
 [[oidc]]
 ==== OpenID Connect single sign-on
 
@@ -263,11 +239,11 @@ The following sections apply both to <<saml>> and <<oidc>>
 ===== Access and refresh tokens
 
 Once the user logs in to {kib} Single Sign-On, either using SAML or OpenID Connect, {es} issues access and refresh tokens
-that {kib} encrypts and stores them in its own session cookie. This way, the user isn't redirected to the Identity Provider
-for every request that requires authentication. It also means that the {kib} session depends on the <<security-ui-settings,
+that {kib} encrypts and stores as a part of its own session. This way, the user isn't redirected to the Identity Provider
+for every request that requires authentication. It also means that the {kib} session depends on the <<security-session-and-cookie-settings,
 `xpack.security.session.idleTimeout` and `xpack.security.session.lifespan`>> settings, and the user is automatically logged
-out if the session expires. An access token that is stored in the session cookie can expire, in which case {kib} will
-automatically renew it with a one-time-use refresh token and store it in the same cookie.
+out if the session expires. An access token that is stored in the session can expire, in which case {kib} will
+automatically renew it with a one-time-use refresh token and store it in the same session.
 
 {kib} can only determine if an access token has expired if it receives a request that requires authentication. If both access
 and refresh tokens have already expired (for example, after 24 hours of inactivity), {kib} initiates a new "handshake" and
@@ -280,8 +256,7 @@ indicates that both access and refresh tokens are expired. Reloading the current
 [float]
 ===== Local and global logout
 
-During logout, both the {kib} session cookie and access/refresh token pair are invalidated. Even if the cookie has been
-leaked, it can't be re-used after logout. This is known as "local" logout.
+During logout, both the {kib} session and {es} access/refresh token pair are invalidated. This is known as "local" logout.
 
 {kib} can also initiate a "global" logout or _Single Logout_ if it's supported by the external authentication provider and not
 explicitly disabled by {es}. In this case, the user is redirected to the external authentication provider for log out of

docs/user/security/securing-kibana.asciidoc

@@ -56,34 +56,16 @@ xpack.security.encryptionKey: "something_at_least_32_characters"
 For more information, see <<security-settings-kb,Security settings in {kib}>>.
 --
 
-. Optional: Set a timeout to expire idle sessions. By default, a session stays
-active until the browser is closed. To define a sliding session expiration, set
-the `xpack.security.session.idleTimeout` property in the `kibana.yml`
-configuration file. The idle timeout is formatted as a duration of
-`<count>[ms|s|m|h|d|w|M|Y]` (e.g. '70ms', '5s', '3d', '1Y'). For example, set
-the idle timeout to expire idle sessions after 10 minutes:
+. Configure {kib}'s session expiration settings. Set both the idle timeout and lifespan settings:
 +
 --
 [source,yaml]
 --------------------------------------------------------------------------------
-xpack.security.session.idleTimeout: "10m"
+xpack.security.session.idleTimeout: "1h"
+xpack.security.session.lifespan: "30d"
 --------------------------------------------------------------------------------
---
 
-. Optional: Change the maximum session duration or "lifespan" -- also known as
-the "absolute timeout". By default, a session stays active until the browser is
-closed. If an idle timeout is defined, a session can still be extended
-indefinitely. To define a maximum session lifespan, set the
-`xpack.security.session.lifespan` property in the `kibana.yml` configuration
-file. The lifespan is formatted as a duration of `<count>[ms|s|m|h|d|w|M|Y]`
-(e.g. '70ms', '5s', '3d', '1Y'). For example, set the lifespan to expire
-sessions after 8 hours:
-+
---
-[source,yaml]
---------------------------------------------------------------------------------
-xpack.security.session.lifespan: "8h"
---------------------------------------------------------------------------------
+For more information, see <<xpack-security-session-management, Session management in {kib}>>.
 --
 
 . Optional: <<configuring-tls,Configure {kib} to encrypt communications>>.
@@ -146,3 +128,4 @@ include::securing-communications/index.asciidoc[]
 include::securing-communications/elasticsearch-mutual-tls.asciidoc[]
 include::audit-logging.asciidoc[]
 include::access-agreement.asciidoc[]
+include::session-management.asciidoc[]

docs/user/security/session-management.asciidoc

@@ -0,0 +1,49 @@
+[role="xpack"]
+[[xpack-security-session-management]]
+=== Session management
+
+When you log in, {kib} creates a session that is used to authenticate subsequent requests to {kib}. A session consists of two components: an encrypted cookie that is stored in your browser, and an encrypted document in a dedicated {es} hidden index. By default, the name of that index is `.kibana_security_session_1`, where the prefix is derived from the primary `.kibana` index. If either of these components are missing, the session is no longer valid.
+
+When your session expires, or you log out, {kib} will invalidate your cookie and remove session information from the index. {kib} also periodically invalidates and removes any expired sessions that weren't explicitly invalidated.
+
+[[session-idle-timeout]]
+==== Session idle timeout
+
+You can use `xpack.security.session.idleTimeout` to expire sessions after a period of inactivity. This and `xpack.security.session.lifespan` are both highly recommended.
+By default, sessions don't expire because of inactivity. To define a sliding session expiration, set the  property in the `kibana.yml` configuration file. The idle timeout is formatted as a duration of `<count>[ms|s|m|h|d|w|M|Y]` (e.g. '20m', '24h', '7d', '1w'). For example, set the idle timeout to expire sessions after 1 hour of inactivity:
+
+--
+[source,yaml]
+--------------------------------------------------------------------------------
+xpack.security.session.idleTimeout: "1h"
+--------------------------------------------------------------------------------
+--
+
+[[session-lifespan]]
+==== Session lifespan
+
+You can use `xpack.security.session.lifespan` to configure the maximum session duration or "lifespan" -- also known as the "absolute timeout". This and `xpack.security.session.idleTimeout` are both highly recommended. By default, sessions don't have a fixed lifespan, and if an idle timeout is defined, a session can still be extended indefinitely. To define a maximum session lifespan, set the property in the `kibana.yml` configuration file. The lifespan is formatted as a duration of `<count>[ms|s|m|h|d|w|M|Y]` (e.g. '20m', '24h', '7d', '1w'). For example, set the lifespan to expire sessions after 30 days:
+
+--
+[source,yaml]
+--------------------------------------------------------------------------------
+xpack.security.session.lifespan: "30d"
+--------------------------------------------------------------------------------
+--
+
+[[session-cleanup-interval]]
+==== Session cleanup interval
+
+[IMPORTANT]
+============================================================================
+If you specify neither session idle timeout nor lifespan, then {kib} will not automatically remove session information from the index unless you explicitly log out. This might lead to an infinitely growing session index. Configure the idle timeout and lifespan settings for the {kib} sessions so that they can be cleaned up even if you don't explicitly log out.
+============================================================================
+
+You can configure the interval at which {kib} tries to remove expired and invalid sessions from the session index. By default, this value is 1 hour and cannot be less than 10 seconds. To define another interval, set the `xpack.security.session.cleanupInterval` property in the `kibana.yml` configuration file. The interval is formatted as a duration of `<count>[ms|s|m|h|d|w|M|Y]` (e.g. '20m', '24h', '7d', '1w'). For example, schedule the session index cleanup to perform once a day:
+
+--
+[source,yaml]
+--------------------------------------------------------------------------------
+xpack.security.session.cleanupInterval: "1d"
+--------------------------------------------------------------------------------
+--

src/dev/build/tasks/os_packages/docker_generator/resources/bin/kibana-docker

@@ -237,6 +237,7 @@ kibana_vars=(
     xpack.security.sessionTimeout
     xpack.security.session.idleTimeout
     xpack.security.session.lifespan
+    xpack.security.session.cleanupInterval
     xpack.security.loginAssistanceMessage
     xpack.security.loginHelp
     xpack.spaces.enabled

x-pack/plugins/security/kibana.json

@@ -3,7 +3,7 @@
   "version": "8.0.0",
   "kibanaVersion": "kibana",
   "configPath": ["xpack", "security"],
-  "requiredPlugins": ["data", "features", "licensing"],
+  "requiredPlugins": ["data", "features", "licensing", "taskManager"],
   "optionalPlugins": ["home", "management"],
   "server": true,
   "ui": true,