Implement Server-Side sessions

[azasypkin]

Summary

In this PR we're changing the approach we use to handle users sessions. Previously all session information was stored in the client-side cookie, but client-side cookies cannot cover all the session management requirements we have these days and we'd like to store some of the session information in the Elasticsearch index (see details and discussion in #17870).

Below I'm trying to outline the major changes that this PR makes.

Changes

Storage

The session value is now split between two sources: session cookie and session index. Fields in bold are NOT available to consumers of the Session APIs and considered as an internal implementation detail of the Session.

• Session cookie value fields:

  • sid - unique identifier of the session used to match session cookie with session info stored in Elasticsearch index;
  • aad - additional authenticated data (AAD) used to encrypt\decrypt sensitive part of the session value stored in the index. See Security section below for details;
  • path - base path of the Kibana instance that created the session. Knowing this allows us to detect old cookies that may belong to the session that was created when/if Kibana was hosted under another base path or without it (it may happen during initial configuration or development). Having this in the cookie allows us to avoid querying the index.
  • idleTimeoutExpiration - as name implies, this field tells us when the session should be considered expired. It's the most frequently updated field of the session value, having this in the cookie allows us to significantly reduce number of costly index writes. See Optimizations section for details.
  • lifespanExpiration - this field tells us the max total lifespan of the session. This value never changes unless xpack.security.session.lifespan is reconfigured. Again, having this in the cookie allows us to avoid querying the index just to figure out that session is expired.

• Session index value fields:

  • sid - unique identifier of the session used to match session cookie with session info stored in Elasticsearch index;
  • username_hash - hash digest of the username associated with the session. When combined with the provider field can be used to uniquely identify session of a particular user within a particular realm. This field is only present in authenticated sessions, unauthenticated sessions (e.g. intermediate sessions used during SSO handshake), for obvious reasons, don't have this field.
  • tenant - it's a hash of the Kibana index name, we rely on this to only cleanup session values that belong to a specific tenant
  • provider - type and name of the authentication provider used to create session. When combined with the username_hash field can be used to uniquely identify session of a particular user within a particular realm;
  • idleTimeoutExpiration - same as idleTimeoutExpiration described in the session cookie section above. We need to store this in index to know which sessions are still active so that we can automatically clean up expired ones and e.g. figure out how many active session particular user has at the moment (Limit the number of concurrent user sessions #18162). Unlike session cookie, here this value doesn't reflect exact expiration time, see Optimizations section for details.
  • lifespanExpiration - same as lifespanExpiration described in the session cookie section above. And as well as idleTimeoutExpiration this field allow us know which sessions are still active so that we can automatically clean up expired ones and e.g. figure out how many active session particular user has at the moment (Limit the number of concurrent user sessions #18162);
  • accessAgreementAcknowledged - indicates whether user acknowledged access agreement or not. There is no need to store this field in the cookie.
  • content - it's an encrypted JSON string that includes a portion of the session value that is sensitive and should be hidden. Currently this JSON includes username and session state returned by the authentication provider (e.g. username/password for basic provider, access/refresh token pair for saml, client certificate fingerprint for pki etc.). The decrypted fields are available to the consumers of Session APIs.
  • metadata - currently it contains only primaryTerm and sequenceNumber we need to deal with concurrent update/extend requests.

Note: I don't include roles and ip in this iteration as these can easily be added separately when we think we need them.

For now the plan is to use new .kibana_security_session_1 index under .kibana_security_session alias (we may want to use another numbering scheme, e.g. ES uses 7 to indicate ES version in which particular index template was introduced/updated) to store session info. Since it's the first time we add session index I didn't add the code that would handle versions and possible migrations.

Currently I'm using index template, to be a little bit more resilient to the cases when index is deleted (so that the first session create operation would properly resurrect it), but I'm not completely sure we need this. I'm copying index settings from ES's security-tokens index template plus marking index as hidden:

settings: {
  index: {
    number_of_shards: 1,
    number_of_replicas: 0,
    auto_expand_replicas: '0-1',
    priority: 1000,
    refresh_interval: '1s',
    hidden: true,
  },
},

Security

As discussed in #17870 we're implementing a number of security-related measures, namely:

• sid and aad are generated using a secure PRNG (256 bits)
• aad is only stored in the session cookie, so one need to have access to both the session cookie and session index to decrypt sensitive session information
• username is encrypted, and only exposed to the querying via username_hash field that is sha3-256 digest of the username

Optimizations

Since querying index on every user request will inevitably affect performance I'm trying to implement a number of optimizations to reduce the impact:

• We duplicate path, lifespanExpiration and idleTimeoutExpiration in the session cookie so that we can do basic session validity checks without querying index
• We update session index only in the following cases:
  • When session is created
  • When username changes (+ old session will be removed)
  • When provider decides to update its state (e.g. at the end of SSO handshake we replace unauthenticated intermediate session with the authenticated final one, or when provider refreshes tokens, or when user changes password)
  • When xpack.security.session.lifespan or xpack.security.session.idleTimeout is reconfigured
  • When xpack.security.session.idleTimeout is specified we regularly update idleTimeoutExpiration only in the session cookie and eventually update session index only when a difference between idleTimeoutExpiration in a session cookie and in the index exceeds 2 * idleTimeout (we can play with the factor). Automatic cleanup routine will remove sessions from the index only if difference between current time and idleTimeoutExpiration exceeds 3 * idleTimeout. This way we reduce a number of index updates, but session may be considered valid for a longer than it's in reality. It should not make us less secure since we rely on idleTimeoutExpiration from the cookie to decide whether session is still valid or not before we perform any action that requires authentication. But this is relevant for automatic cleanup routine and any future checks when we would like to know how many active sessions particular user has at the moment. That sounds like a reasonable trade-off to me, but I'd like to know what you think?
• We run a cleanup routine using a periodic task, interval is configured via xpack.security.session.cleanupInterval (see all covered cases in SessionIndex.cleanUp)

I did a quick performance test as well:

Input

I created ~115000 of sessions in the .kibana_security_session index and querying /internal/security/me endpoint. The basic autocannon snippet I used looks like this (100000 requests in total with 100 simultaneous connections):

#!/usr/bin/env bash

CONNECTIONS=100
REQUESTS=100000
COOKIE="sid=xxxxxx"
EXPECTED_BODY="{\"username\":\"elastic\",\"roles\":[\"superuser\"],\"full_name\":null,\"email\":null,\"metadata\":{\"_reserved\":true},\"enabled\":true,\"authentication_realm\":{\"name\":\"reserved\",\"type\":\"reserved\"},\"lookup_realm\":{\"name\":\"reserved\",\"type\":\"reserved\"},\"authentication_provider\":\"basic1\"}"

npx autocannon \
  --connections "$CONNECTIONS" \
  --amount "$REQUESTS" \
  --headers Cookie="$COOKIE" \
  --headers Accept="application/json" \
  --headers Content-Type="application/json" \
  --expectBody="$EXPECTED_BODY" \
  "https://localhost:5601/internal/security/me"

Results

Before the change (session is extracted from the cookie)

Running 100000 requests test @ https://localhost:5601/internal/security/me
100 connections

┌─────────┬───────┬────────┬────────┬────────┬───────────┬──────────┬────────────┐
│ Stat    │ 2.5%  │ 50%    │ 97.5%  │ 99%    │ Avg       │ Stdev    │ Max        │
├─────────┼───────┼────────┼────────┼────────┼───────────┼──────────┼────────────┤
│ Latency │ 96 ms │ 105 ms │ 136 ms │ 200 ms │ 107.84 ms │ 27.24 ms │ 1643.51 ms │
└─────────┴───────┴────────┴────────┴────────┴───────────┴──────────┴────────────┘
┌───────────┬────────┬────────┬─────────┬─────────┬─────────┬─────────┬────────┐
│ Stat      │ 1%     │ 2.5%   │ 50%     │ 97.5%   │ Avg     │ Stdev   │ Min    │
├───────────┼────────┼────────┼─────────┼─────────┼─────────┼─────────┼────────┤
│ Req/Sec   │ 606    │ 608    │ 936     │ 1012    │ 917.44  │ 83.29   │ 501    │
├───────────┼────────┼────────┼─────────┼─────────┼─────────┼─────────┼────────┤
│ Bytes/Sec │ 696 kB │ 698 kB │ 1.08 MB │ 1.16 MB │ 1.05 MB │ 95.6 kB │ 575 kB │
└───────────┴────────┴────────┴─────────┴─────────┴─────────┴─────────┴────────┘

Req/Bytes counts sampled once per second.

100k requests in 109.08s, 115 MB read

After the change (session is extracted from the cookie and index)

Running 100000 requests test @ https://localhost:5601/internal/security/me
100 connections

┌─────────┬────────┬────────┬────────┬────────┬───────────┬──────────┬────────────┐
│ Stat    │ 2.5%   │ 50%    │ 97.5%  │ 99%    │ Avg       │ Stdev    │ Max        │
├─────────┼────────┼────────┼────────┼────────┼───────────┼──────────┼────────────┤
│ Latency │ 263 ms │ 287 ms │ 356 ms │ 395 ms │ 291.67 ms │ 29.23 ms │ 1346.63 ms │
└─────────┴────────┴────────┴────────┴────────┴───────────┴──────────┴────────────┘
┌───────────┬────────┬────────┬────────┬────────┬────────┬─────────┬────────┐
│ Stat      │ 1%     │ 2.5%   │ 50%    │ 97.5%  │ Avg    │ Stdev   │ Min    │
├───────────┼────────┼────────┼────────┼────────┼────────┼─────────┼────────┤
│ Req/Sec   │ 244    │ 300    │ 344    │ 372    │ 341.3  │ 21.74   │ 159    │
├───────────┼────────┼────────┼────────┼────────┼────────┼─────────┼────────┤
│ Bytes/Sec │ 162 kB │ 199 kB │ 228 kB │ 247 kB │ 226 kB │ 14.4 kB │ 105 kB │
└───────────┴────────┴────────┴────────┴────────┴────────┴─────────┴────────┘

Req/Bytes counts sampled once per second.

100k requests in 293.13s, 66.3 MB read

The average difference is ~2.5-3 times, same result for almost empty session index. I'll need to investigate where we spend most of the time, it's either network (loopback) or decryption.

Limitations and unresolved questions

• Session cookie is still encrypted. As @jportner mentioned in Kibana Security to use Server Side Sessions #17870 it allows to query index only if we can decrypt cookie which is good, we don't need to make any changes in the core that handles cookie encryption which is also good, but we'll spend time on decryption/encryption. That sounds reasonable to me, what do you think?
• Another question is that we use the same encryption key for the cookie and the value we store in the index. It's probably not recommended, but at the same time having separate keys may bring unnecessary complexity into the mix.
  * Currently we don't regenerate sid and aad if username changes or if user changes their own password. IIRC the recommendation is to re-generate these things when such an event happens, but it increases implementation complexity a bit (still need to evaluate). I'd keep it as is at the initial stage, but don't have a strong opinion. What do you think?
• When neither xpack.security.session.lifespan nor xpack.security.session.idleTimeout is configured and encryption key changes, old sessions may stuck in the index indefinitely. I'm not sure how I feel about that yet, but should we store some sort of hash of the encryption key in the index so that we can detect that change and remove old sessions?
• Should we use _bulk query for session creation to run cleanup routine together with the _create query? Theoretically login action should be more frequent than logout.

Checklist

• Consolidate session handling code.
• Use ES index to store session information
• Fix Properly handle very long URL fragments captured during SAML handshake #53478
• Fix Use Overwritten Session view if an active session is implicitly overriden by a different provider #61217
• Fix Merge /internal/security/login_with endpoint into /internal/security/login #61222
• Test how implementation behaves when we have many concurrent requests that updates session index
• Write unit and integration tests
• Mention xpack.security.session.cleanupInterval in the docs

Fixes: #17870

---

Release Note: from now on Kibana will be storing users session information in a dedicated Elasticsearch index. By default expired and invalid sessions are cleaned up from the index every hour. The cleanup interval can be configured with xpack.security.session.cleanupInterval setting. Note that after upgrade all existing sessions will become invalid and users will have re-log in to Kibana. Another notable side effect of this change is that from now on it will no longer be possible to host different Kibana tenants on the different ports of the same host. Even though such setup worked well in the past it was always discouraged since browsers share cookie across all applications hosted using the same host name ignoring ports. With this change cookies will be strictly tied to a particular tenant.

[azasypkin]

@legrego @jportner I believe PR is at the stage when we can play with it and discuss the implementation. In #68117 (comment) I tried to explain all the decisions I've made so far and I'd really appreciate if you can take a look at the questions I left for you whenever you have time.

I'll try to approach you next week for the quick sync to handle any concerns you may have 🙂

P.S. PR doesn't include any tests yet, I haven't performed performance/concurrency tests as well, but it includes the fix for #53478 (to have at least one real benefit from this work 🙂)

Thanks!

[kobelb]

Is my understanding correct that this doesn't use saved-objects to access the documents in the .kibana_security_session index?

[azasypkin]

Is my understanding correct that this doesn't use saved-objects to access the documents in the .kibana_security_session index?

Yes, that's correct, but not set in stone yet. My initial reasoning behind having a dedicated index was to have a full control over index (e.g. to not accidentally expose session objects to other parts of Kibana that use SO client heavily even if it's marked as hidden) and more importantly the performance we'll need to squeeze from index related operations (e.g. support for delete_by_query).

It's all theoretical though, happy to discuss alternatives, pros and cons (here or over Zoom)!

[legrego]

path - base path of the Kibana instance that created the session. This information is stored in the index to allow us to automatically cleanup sessions with a path that is different from the current Kibana base path;

For now the plan is to use new .kibana_security_session index (must probably be alias with .kibana_security_session-1 index) to store session info, but I haven't thoroughly thought it through yet. Since it's the first time we add session index I didn't add the code that would handle versions and possible migrations.

For at least the remainder of 7.x, and potentially longer, we'll probably want to take the current tenant into account (kibana.index value). Otherwise we'll risk one tenant deleting the sessions of another tenant if they are hosted on different paths.

[azasypkin]

For at least the remainder of 7.x, and potentially longer, we'll probably want to take the current tenant into account (kibana.index value). Otherwise we'll risk one tenant deleting the sessions of another tenant if they are hosted on different paths.

That's a good point! I'll make sure we take this into account.

[legrego]

Currently we don't regenerate sid and aad if username changes or if user changes their own password. IIRC the recommendation is to re-generate these things when such an event happens, but it increases implementation complexity a bit (still need to evaluate). I'd keep it as is at the initial stage, but don't have a strong opinion. What do you think?

I think we should take the time to regenerate these for at least the username change. Otherwise, it's possible to hijack existing sessions. For example, if alice has an active session, but then walks away from her workstation, someone else could copy the cookie value, and send that cookie to Kibana's /internal/security/login endpoint with a different set of credentials.

Now, the attacker is authenticated as whoever they just authenticated as, but more importantly, Alice's existing session cookie is no longer associated with alice, but instead with the second set of credentials.

[azasypkin]

Now, the attacker is authenticated as whoever they just authenticated as, but more importantly, Alice's existing session cookie is no longer associated with alice, but instead with the second set of credentials.

Well, I'd say it's a risk, but it doesn't seem like a new one. I believe if currently you can access someone's else cookie you most likely can replace it with your own too, in one way or another. That would lead to a similar state.

But irrespective to whether it's new or existing risk we have a chance to improve the state of things and it's a good practice anyway. I'll see how that would look like.

[kobelb]

Another question is that we use the same encryption key for the cookie and the value we store in the index. It's probably not recommended, but at the same time having separate keys may bring unnecessary complexity into the mix.

Have you considered storing the encryption key for the server-side session in the client-side cookie? This way, if an attacker was to compromise the .kibana_security_session index, they wouldn't be able to decrypt the values with the encryption key which is stored on the Kibana filesystem.

[azasypkin]

Have you considered storing the encryption key for the server-side session in the client-side cookie? This way, if an attacker was to compromise the .kibana_security_session index, they wouldn't be able to decrypt the values with the encryption key which is stored on the Kibana filesystem.

We mitigate this risk via storing AAD (random 256 bit string) only in the client cookie (see Security section in #68117 (comment)). Do you have any concerns regarding this approach?

[kobelb]

We mitigate this risk via storing AAD (random 256 bit string) only in the client cookie (see Security section in #68117 (comment)). Do you have any concerns regarding this approach?

Nope! That seems completely reasonable. I forgot the AAD would prevent the decryption entirely, and initially thought it was acting more like a checksum, but that's wrong...

[azasypkin]

Hey @elastic/siem, can anyone please let me know what this code does? Is it only for tests or you use this for anything else? We're changing our internal login endpoint and I want to make sure I properly adjusted your code that depends on it.

[spong]

Correct @azasypkin, this is just for our e2e cypress tests. These tests run as part of CI, so we should see a failure then if there are any issues. Thanks for the ping! 🙂

cc @MadameSheema

[azasypkin]

Correct @azasypkin, this is just for our e2e cypress tests. These tests run as part of CI, so we should see a failure then if there are any issues. Thanks for the ping! slightly_smiling_face

Thanks for confirming! Security solution Cypress Tests suite is green, so we should be good.

[azasypkin]

note: removed this cookie-related sections, I think they don't make any sense now - it's should be assumed by default that if user logs out session cannot be reused.

[jportner]

Minor nits below. Other than that, there's one last admonition in the "Security settings" page that is breaking up a table:

[jportner]

Suggested change

	===== Login user interface settings
	[cols="2*<"]
	===== Login user interface settings
	You can configure the following settings in the `kibana.yml` file.
	[cols="2*<"]

[jportner]

I like how you modified this tip for cleanupInterval to provide more realistic values. We should do the same here 😁

Suggested change

	The format is a string of `<count>[ms\|s\|m\|h\|d\|w\|M\|Y]` (e.g. '70ms', '5s', '3d', '1Y').
	The format is a string of `<count>[ms\|s\|m\|h\|d\|w\|M\|Y]` (e.g. '20m', '24h', '7d', '1w').

[jportner]

This too

Suggested change

	The format is a string of `<count>[ms\|s\|m\|h\|d\|w\|M\|Y]` (e.g. '70ms', '5s', '3d', '1Y').
	The format is a string of `<count>[ms\|s\|m\|h\|d\|w\|M\|Y]` (e.g. '20m', '24h', '7d', '1w').

[jportner]

It feels odd to see lifespan before idleTimeout, even though this is the idle timeout section.

How about rewording it like this?

Suggested change

	You can expire sessions after a period of inactivity. This and `xpack.security.session.lifespan` are both
	highly recommended. By default, session doesn't expire because of inactivity. To define a sliding session expiration, set
	the `xpack.security.session.idleTimeout` property in the `kibana.yml`
	configuration file. The idle timeout is formatted as a duration of
	`<count>[ms|s|m|h|d|w|M|Y]` (e.g. '70ms', '5s', '3d', '1Y'). For example, set
	the idle timeout to expire sessions after 1 hour of inactivity:
	You can use `xpack.security.session.idleTimeout` to expire sessions after a period of inactivity. This and `xpack.security.session.lifespan`
	are both highly recommended.
	By default, sessions don't expire because of inactivity. To define a sliding session expiration, set the property in the `kibana.yml`
	configuration file. The idle timeout is formatted as a duration of `<count>[ms|s|m|h|d|w|M|Y]` (e.g. '20m', '24h', '7d', '1w'). For example,
	set the idle timeout to expire sessions after 1 hour of inactivity:

Note 1: changed wording "session doesn't expire" -> "sessions don't expire"
Note 2: changed duration example to match that in prior comments

[jportner]

Same as above -- it looked weird to see idleTimeout in this section before lifespan. How about this?

Suggested change

	You can configure the maximum session duration or "lifespan" -- also known as
	the "absolute timeout". This and `xpack.security.session.idleTimeout` are both highly
	recommended. By default, session doesn't have a fixed lifespan, and if an idle timeout is defined, a session can still be extended
	indefinitely. To define a maximum session lifespan, set the
	`xpack.security.session.lifespan` property in the `kibana.yml` configuration
	file. The lifespan is formatted as a duration of `<count>[ms|s|m|h|d|w|M|Y]`
	(e.g. '70ms', '5s', '3d', '1Y'). For example, set the lifespan to expire
	sessions after 30 days:
	You can use `xpack.security.session.lifespan` to configure the maximum session duration or "lifespan" -- also known as the "absolute
	timeout". This and `xpack.security.session.idleTimeout` are both highly recommended.
	By default, sessions don't have a fixed lifespan, and if an idle timeout is defined, a session can still be extended indefinitely. To define
	a maximum session lifespan, set the property in the `kibana.yml` configuration file. The lifespan is formatted as a duration of
	`<count>[ms|s|m|h|d|w|M|Y]` (e.g. '20m', '24h', '7d', '1w'). For example, set the lifespan to expire sessions after 30 days:

Note 1: changed wording "session doesn't have" -> "sessions don't have"
Note 2: changed duration example to match that in prior comments

[jportner]

Another duration example for consistency 🙂

Suggested change

You can configure the interval at which {kib} tries to remove expired and invalid sessions from the session index. By default, this value is 1 hour and cannot be less than 10 seconds. To define another interval, set the `xpack.security.session.cleanupInterval` property in the `kibana.yml` configuration file. The interval is formatted as a duration of `<count>[ms|s|m|h|d|w|M|Y]` (e.g. '70000ms', '50s', '3d', '1Y'). For example, schedule the session index cleanup to perform once a day:

You can configure the interval at which {kib} tries to remove expired and invalid sessions from the session index. By default, this value is 1 hour and cannot be less than 10 seconds. To define another interval, set the `xpack.security.session.cleanupInterval` property in the `kibana.yml` configuration file. The interval is formatted as a duration of `<count>[ms|s|m|h|d|w|M|Y]` (e.g. '20m', '24h', '7d', '1w'). For example, schedule the session index cleanup to perform once a day:

[azasypkin]

Minor nits below. Other than that, there's one last admonition in the "Security settings" page that is breaking up a table:

Handled, thanks!

[gchaps]

A few minor comments, but otherwise LGTM.

[gchaps]

Suggested change

	| Determines if the provider should treat the `RelayState` parameter as a deep link in {kib} during Identity Provider initiated log in. By default, this setting is set to `false`. The link specified in `RelayState` should be a relative, URL-encoded {kib} URL. For example, the `/app/dashboards#/list` link in `RelayState` parameter would look like this `RelayState=%2Fapp%2Fdashboards%23%2Flist`.
	| Determines if the provider should treat the `RelayState` parameter as a deep link in {kib} during Identity Provider initiated log in. By default, this setting is set to `false`. The link specified in `RelayState` should be a relative, URL-encoded {kib} URL. For example, the `/app/dashboards#/list` link in `RelayState` parameter would look like this: `RelayState=%2Fapp%2Fdashboards%23%2Flist`.

[gchaps]

Suggested change

	| Adds a message to the login UI. Useful for displaying information about maintenance windows, links to corporate sign up pages etc.
	| Adds a message to the login UI. Useful for displaying information about maintenance windows, links to corporate sign up pages, and so on.

[gchaps]

Suggested change

	| This setting ensures that user sessions will expire after a period of inactivity. This and `xpack.security.session.lifespan` are both
	| Ensures that user sessions will expire after a period of inactivity. This and `xpack.security.session.lifespan` are both

[gchaps]

Suggested change

	| This setting ensures that user sessions will expire after the defined time period. This behavior also known as an "absolute timeout". If
	| Ensures that user sessions will expire after the defined time period. This behavior also known as an "absolute timeout". If

[gchaps]

Suggested change

	that {kib} encrypts and stores them as a part of its own session. This way, the user isn't redirected to the Identity Provider
	that {kib} encrypts and stores as a part of its own session. This way, the user isn't redirected to the Identity Provider

[gchaps]

Suggested change

	. Configure {kib}'s session expiration settings. We strongly recommend setting both idle timeout and lifespan settings:
	. Configure {kib}'s session expiration settings. Set both the idle timeout and lifespan settings:

[jportner]

Docs changes LGTM!

[azasypkin]

7.x/7.10.0: 2bf5a2c

[kibanamachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request
• Commit: 8ea6939
• Pipeline Steps (look for red circles / failed steps)
• Interpreting CI Failures

Failed CI Steps

• Execute xpack-kibana-ciGroup8
• Execute oss-firefox
• Execute xpack-kibana-ciGroup5
• Execute kibana-ciGroup1
• Execute kibana-ciGroup8
• Execute xpack-kibana-ciGroup2
• Execute kibana-ciGroup10
• Execute xpack-kibana-ciGroup10
• Execute kibana-ciGroup7
• Execute xpack-firefox
• Execute kibana-ciGroup9
• Execute xpack-kibana-ciGroup4
• Execute kibana-ciGroup2
• Execute kibana-ciGroup6
• Execute xpack-kibana-ciGroup3
• Execute xpack-kibana-ciGroup6
• Execute kibana-ciGroup4
• Execute xpack-kibana-ciGroup1
• Execute kibana-ciGroup12
• Execute xpack-kibana-ciGroup9
• Execute oss-accessibility
• Execute oss-accessibility
• API Integration Tests
• Execute kibana-ciGroup5
• Execute xpack-savedObjectsFieldMetrics
• Execute kibana-ciGroup3
• Execute xpack-accessibility
• Execute kibana-ciGroup11
• Execute xpack-kibana-ciGroup7
• Execute oss-pluginFunctional
• Execute xpack-securitySolutionCypress

Metrics [docs]

‼️ ERROR: metrics for 7d3fe58 were not reported

History

• 💚 Build #68540 succeeded 8ea6939
• 💚 Build #68351 succeeded 3ded572
• 💚 Build #68137 succeeded a4cdb60
• 💛 Build #68111 was flaky 378c239

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream