[Discuss] A way for plugins to store and retrieve their own data from the server side session

[jgr]

Kibana recently moved to using a server-side session store. We (the Workplace Search team) would like a way to store plugin specific data in this server side session so that it follows the user throughout their login session.

Background

In the course of setting up the product operators go through the OAuth flows of third party services to connect them to Workplace Search. While an operator is going through these flows the Workplace Search product currently saves some temporary tokens and an authenticity token in it's own login session. As we transition this functionality to the Enterprise Search plugin we'd like to store these tokens in Kibana's session instead, as we do not maintain an Enterprise Search session when using the product via the Kibana plugin.

Something as durable as a login session store is preferable because these tokens need to persist between a redirect. As part of the OAuth flow the plugin redirects the user to the third party to be authenticated, and the third party redirects the user back to the Kibana plugin.

An Example

I'm envisioning something fairly simple that takes a KibanaRequest (to find the session), a key to set in the session, and a string payload. Something along the lines of:

setSessionValue(request, 'xpack.enterpriseSearch.authenticityToken', payload)

Retrieval would be fairly similar.

const authenticityToken = getSessionValue(request, 'xpack.enterpriseSearch.authenticityToken')

I realize we might need to do more here to scope this to just a single plugin, but hopefully that example illustrates our need.

I also realize that there could be a session when there is no user logged in. This isn't really a use case for Workplace Search (as our plugin only really makes sense when a user is authenticated and has the appropriate access), but I understand I may need to guard these calls with a check that there is a currently logged in user.

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[jgr]

ping @ioanatia @jportner @azasypkin

[azasypkin]

The ability to store custom data in a user session sounds reasonable to me in general. I think it can be useful for other plugins in the future too. Here is just a very rough idea on how that could look like, please comment if you have any concerns, or questions, or ideas.

////// -----------SETUP-----------

// To avoid name collisions and prevent other plugins from accessing values they don't own you have
// to register a session storage container with a unique name at the `SETUP` stage. Container name will
// be used as a prefix for all session storage values your plugin is going to deal with. You're not allowed
// and don't have a way to retrieve or store any values in the session at this stage.
// @returns Session storage container instance that you can exchange to a session storage in `START`.
// @throws If this method is invoked after `setup` or if any other plugin already registered session storage
// container with such a name error will be thrown.
const sessionStorageContainer = setupDeps.security.sessionStorage.registerContainer(
  'xpack.entSearch'
);

////// -----------START-----------

// Then at the `START` stage you can retrieve a session storage that is bound to a particular container.
// @returns Session storage that can be used to store or retrieve session values.
// @throws If container isn't valid for whatever reason this method will throw.
const sessionStorage = startDeps.security.sessionStorage.getStorage(sessionStorageContainer);

////// -----------USER REQUEST-----------

// Method is asynchronous as it may cause a request to Elasticsearch.
// @returns Returns value in the same form as it was stored (deserialized).
// @throws Method doesn't swallow any exceptions, consumers should handle errors on their own,
// e.g. when Elasticsearch isn't available or if request isn't authenticated.
const value = await sessionStorage.get(request, 'my-key');

// Method is asynchronous as it causes request to Elasticsearch to persist a session value.
// @throws Method doesn't swallow any exceptions, consumers should handle errors on their own, 
// e.g. when Elasticsearch isn't available or request isn't authenticated. In addition to that consumers
// should make sure that `value` is JSON serializable.
await sessionStorage.set(request, 'my-key', value);

// Method is asynchronous as it causes request to Elasticsearch to persist a session value.
// @throws Method doesn't swallow any exceptions, consumers should handle errors on their own, e.g. when
// Elasticsearch isn't available or if request isn't authenticated.
await sessionStorage.remove(request, 'my-key');

Sessions are retrieved from Elasticsearch on every request and I assume the values that plugins will store will have to be encrypted/decrypted. So the size of the session content is critical. If Security plugin notices a very large value, it may issue a warning log record, but consumers should be responsible for a proper cleanup and should remove the value as soon as they don't need it anymore, that's especially important in the case of long-lived sessions.

@jgr when do you need this? Do you have a workaround that you're happy with for the time being?

[jgr]

@azasypkin I'm still confirming a few things, but overall this looks great! It should satisfy the need we have.

Sessions are retrieved from Elasticsearch on every request and I assume the values that plugins will store will have to be encrypted/decrypted. So the size of the session content is critical.

This makes sense to me. I'm working now to only send what we need for the current OAuth flow, rather than the entire session. I can figure out the size of the maximum payload we'll be storing, if you're curious. I'm currently encrypting/decrypting it on the Enterprise Search side.

consumers should be responsible for a proper cleanup and should remove the value as soon as they don't need it anymore

We should be able to remove these values while processing a successful source connection 👍

@jgr when do you need this?

We're aiming to have our Kibana plugin usable stand alone in 7.14. It would be nice to have it by then.

Do you have a workaround that you're happy with for the time being?

I should know this by the end of the week. I'm proving out a workaround we've been discussing. I may ask for suggestions if I run into further problems using a cookie.

[azasypkin]

I can figure out the size of the maximum payload we'll be storing, if you're curious.

Yes, please share when you have the numbers. We'd like to better understand the needs of the consumers of this API.

We're aiming to have our Kibana Plugin usable stand alone in 7.14. It would be nice to have it by then.

Good, 7.14 sounds doable to me.

I should know this by the end of the week. I'm proving out some a workaround we've been discussing. I may ask for suggestions if I run into further problems using a cookie.

Got it, let us know if you're stuck with anything.

[jgr]

An update: we have a workaround that we're comfortable with (using a cookie) and it is nearly complete.

I also spent some time generating payloads. I believe the largest string we would store is 789 characters. Here is an example:

Y2xSVE1qbDBNWGRDYVhCTWMwZFlhbE5rYjBnM2RtdHpWelU1ZVdSaVZTODVORXcyVFRWMFkwZ3ZVR2czT0hsak5YcFZhV3BsWTNsblp6SXlabmRUWTBSMVJXaE5SMUZ6Ym1RM2QwdFplRWx4V2tOM1QzTlFNbEpMTVU4MGExUXdUVXd4V20xd2RYcGhUV2h3ZEdFNFpYVlZiMDFNY0VOSFptOUlhbkJuS3pnM2JrOVRTbEZoVEcwd05FVlZWSGMzUmtGTVJuWkVVMUZoUlVoaE5raFNka1JpUzFKV1ZFNVpTSGh1ZW5KQ1ZrZEdUMVp6Y1ZOTGRtRXJVMkpZYnpJM1dqTkxiMmM0SzJScVRqTlBkVU53VmxKeVZrWnpLMnhuVVhGVGRHNTVVa1pZWWxCUVdFSjROM0JaWkZSUk1EWlZVM1pvY2xweE4zaDFWalZFUWt0MGJtY3lhV0ozZERCWU1WTlFia1pMUkZOdWRYZFhObWt4ZURKVFMyNTNTbk01VWxSRVQzUk1ibEp2Y1c1VFJXZ3hWR292YW14VWQxaFdPRTFpYlRWRGVuVjZabEZOYWtWV1NEaEZOalFyZERoSWRVMVdhRmd3Tkc4MlluSjVaMkpQYXpkcFRXeFFUMVJyVUZONmFIZFdlVm9yTjJGWFVHWlVSVk56VEhGb2NrSkVMUzFIWWpJMVIyc3hOSGhGTm1WMmIyTnZZemxPU0d4blBUMD0tLWIyY2RmMjY1ZTIxOTk3ZDM0OWEwMDYxZGNiNTYzMjM0NTM3NDk3NGI=

[azasypkin]

I also spent some time generating payloads. I believe the largest string we would store is 789 characters. Here is an example:

Looks good, thanks for the update 👍

[azasypkin]

@jgr, while working on #98049 I realized that there is one limitation that you'll hit if you start storing your data in the user session:

We don't maintain sessions for the users who're authenticated via Authorization: xxx HTTP header directly. E.g. if Kibana is hosted behind a reverse proxy that tries to emulate anonymous access and attaches Authorization: xxx HTTP header to every request before forwarding it to Kibana, then such users won't have sessions. With the introduction of the "native" anonymous access in Kibana we started to discourage reverse-proxy based authentication, but many setups still rely on it.

I doubt that such setups are used with the Workplace Search though, but I wanted to call it out anyway.

[jgr]

@azasypkin thanks for pointing that out. While I'm not familiar with this technique to emulate anonymous access, I agree that it seems unlikely that it will effect users of Enterprise Search.

[azasypkin]

@azasypkin thanks for pointing that out. While I'm not familiar with this technique to emulate anonymous access, I agree that it seems unlikely that it will effect users of Enterprise Search.

Good, thanks for confirming. For the sake of completeness, emulating of the anonymous access is just one example, our users rely on reverse proxies + Kibana HTTP authentication for other cases too, e.g. custom authentication via ouath2-proxy + Kibana Basic HTTP authentication is also a setup that we may see in a wild. But I still think it's safe to assume that the absolute majority of your users won't rely on these custom setups.

[azasypkin]

It doesn't seem there is enough demand to warrant this change, closing for now.