Interactive setup mode

[legrego]

The Kibana and ES Security teams are seeing what it would take to enable security features by default, rather than requiring administrators to configure security on their own.

As part of this initiative, we would like to explore creating an interactive setup mode within Kibana. We would like to present an interface to the user on first launch, if we detect that they haven't explicitly configured their own connection to Elasticsearch.

This interface would allow users to provide their ES connection information. Once verified, we would write this configuration to disk, and allow the rest of Kibana to start.

Broadly speaking, this setup mode should be able to configure settings under server.ssl.* and elasticsearch.*

Here is a rough mockup to illustrate what we are thinking about. The options are only meant to illustrate possibilities, and do not dictate requirements or final design:

---

@thomheymann and I met with @joshdover today, and he proposed extending the "Not Ready Server" to accomodate this. This server currently runs after all plugins have completed their setup phase, but before they are allowed to start.

Since we are looking to change the ES configuration as a result of the config UI, we might need to launch the "Not Ready Server" before plugins are allowed to run through setup (in case they obtain a reference to config that will go stale).

There is a PR in flight which extends the "Not Ready Server" to support the /api/status endpoint, and allows other routes to be served via this server as well. There is potential for us to leverage this work in order to present a setup UI to the end user.

Related: #102538, #104068

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[pgayvallet]

A few questions/remarks after our discussions in our weekly about the proposition to do that using the notReady server:

Regarding the UI:

• We won't be able to bootstrap a real react app here, and as we still need some dynamic interactions in the UI, just rendering on the server-side wouldn't be an option either, so we would have to just use plain old html+javascript for the page
• We won't be able to use translation files, so this page will only be available in a single language for all our customers (unless we recode a re-implement i18n label fetching just for this page). I would check with Product if this is acceptable.

Regarding the backend:

• (Correct me if I'm wrong) We will have to implements a few new endpoints on the 'notReady' server, to manage the various actions (such as 'generate certificate'). That will be adding additional routes only relying on vanilla HAPI, without core's http framework capabilities.
• The 'notReady' server is initialized very early in the lifecycle. We won't have any way to easily query ES for example, or use most of core's services (not sure if we need to)

Regarding 'writing' down the config:

• if we want to write in the user's configuration file
  • We will not necessarily have the permission to do so (access may be in readonly mode for our user)
  • User may have multiple config files (not sure why we allowed that in the first place tbh, but that's technically possible). In that case, which config should we write in?
• if we want to use/add our own, additional configuration file
  • What should be the loading/priority order? should our config should be the first or last of the list?
  • What if the user manually edits its configuration file later? Even if we decide that their config prevails, we could easily have scenarios where they only overrides some settings, creating conflicts with our owns (can easily be done with some of elasticsearch.ssl options imho)

[legrego]

Thanks for your questions/remarks @pgayvallet, this is a great starting point for discussions:

Regarding the UI:

We won't be able to bootstrap a real react app here, and as we still need some dynamic interactions in the UI, just rendering on the server-side wouldn't be an option either, so we would have to just use plain old html+javascript for the page

I'm confused by this limitation. Can you explain why we wouldn't be able to bootstrap a real React app? I understand that we won't have Core services on the front-end, but why can't the notReady server serve a react app?

We won't be able to use translation files, so this page will only be available in a single language for all our customers (unless we recode a re-implement i18n label fetching just for this page). I would check with Product if this is acceptable.

I don't understand the reasons for the limitation, but I'll accept that's true for now 🙂

Regarding the backend:

(Correct me if I'm wrong) We will have to implements a few new endpoints on the 'notReady' server, to manage the various actions (such as 'generate certificate'). That will be adding additional routes only relying on vanilla HAPI, without core's http framework capabilities.

That's right, and I think that's an acceptable tradeoff.

(Knowing nothing about the plugin system), could we make the notReady server extensible via the existing plugin system, and allow plugins to declare which "system" they belong to? Would that give us the necessary infrastructure to better support real React apps (potentially with i18n)? I realize this is probably a lot of effort for little benefit, as we would have completely separate contracts (different core services, for example), but I wanted to at least raise this as an idea.

The 'notReady' server is initialized very early in the lifecycle. We won't have any way to easily query ES for example, or use most of core's services (not sure if we need to)

We will need to communicate with Elasticsearch, but perhaps we can do so by constructing our own client, rather than using the clients exposed by Core?

Regarding 'writing' down the config:

I share all of your concerns here. We'll need to take some time to discuss our options, and see what makes the most sense. For what it's worth, Elasticsearch will have a similar set of challenges, as it will also need to update its own configuration, while dealing with potential conflicts and permission constraints. I'd like to table this for now and instead focus on what (if anything) we can do to enable the experience. We'll collaborate with the ES team to see if there's an approach for config management that makes sense for both systems.

[pgayvallet]

I understand that we won't have Core services on the front-end, but why can't the notReady server serve a react app?

What I mean is that I don't think our optimizer / bundler can work with 'custom' webpack entrypoints atm. I think we would need to manually bundle this application using our own specific webpack config, which is something we may want to avoid. But we will need @spalger opinion here.

(i18) I don't understand the reasons for the limitation

Same reasons as previous point, and would also be solved if previous point is solved I think. Even if I don't exactly know how/where/when the translations are loaded from the server, if we have our client-side bundle, we can have i18n.

could we make the notReady server extensible via the existing plugin system, and allow plugins to declare which "system" they belong to

Joke aside, that would be doing a mini-core inside core. The notReady server is just a vanilla HAPI server atm. This thing was meant to display a plain string to the user 😄 .

Plus, no core services are available when this noReady server starts, meaning that the gains would be minimals. Only pro I see (please tell me if you see more) is that the server-side code would be able to be elsewhere than inside core. But we could eventually just have it inside package imported by core instead.

We will need to communicate with Elasticsearch, but perhaps we can do so by constructing our own client, rather than using the clients exposed by Core?

The config service is loaded before http, so accessing the ES config would be doable. The elasticsearch service is not though, so yea, we would probably have to hack around to call configureClient manually (src/core/server/elasticsearch/client/configure_client.ts)

We'll collaborate with the ES team to see if there's an approach for config management that makes sense for both systems.

SGTM

I realize this is probably a lot of effort for little benefit

That is, to be honest, my summary of doing anything else than a hacky solution for this feature.

But don't get me wrong. I hate what we are suggesting here. And I would love to see more similar needs to have a good reason to implement something cleaner. Unfortunately, I don't think we do atm?

Also I'm searching for a way to have this application be a 'normal' one. But realistically, I don't think we'll be able to graceful restart all the plugins once the configuration has been updated. We are just not ready for full configuration hot reload (or are we @joshdover?)

Another question comes to mind: Will this page be authenticated? Or will the first user accessing the application be able to edit the Kibana configuration file (maybe do more?) with 'SU' privileges?

[spalger]

I think we would need to manually bundle this application using our own specific webpack config, which is something we may want to avoid. But we will need @spalger opinion here.

I actually expected something like this to occur, so we do support building bundles that aren't plugins, which is how we build the core bundle. The Optimizer has "bundle type" support for "entry" bundles, which are intended to be loaded directly into HTML (several initialization scripts will need to be loaded too, like ui-shared-deps, etc, but there's a path in the optimizer for this type of stuff).

What we don't have is a way for plugins to expose entries like this. We might not need to make a generic mechanism for plugins to define these, and instead we could just "hard code" these bundles into packages/kbn-optimizer/src/optimizer/optimizer_config.ts similar to how we do for core. This script should be pretty limited in what it imports, and the core bundle is required to load referenced plugins, but we should be able to make this work if this is the path we want to take.

[arisonl]

@legrego @thomheymann love this, we probably need to analyse more the possible user flows (working with Thom on it) based on the use case (local vs online etc.) but I think that it is a great starting point. Quick question: Could this be used for the rest of the security initialisation, except TLS? For example for the 'quick start and try offline' use-case, you potentially don't even need TLS. But we still want to turn on authentication and authorisation. Can we leverage this to manage the elastic and kibana user credentials? Could the kibana users credentials get generated automatically since they are used by a service account and could we just display them here? On the other hand, could we let the admin user select the elastic credentials, through this screen?

[pgayvallet]

After a sync discussion with @legrego and @kobelb:

• This interactive setup mode should be also (or at least) provided as a CLI. @kobelb was saying that this is the direction that ES is taking too.
• If we want a UI, the most practical solution may be to have this as a totally separate process/server than our current Kibana server
  • we could adapt the /bin/kibana script to launch the interactive setup server if it detects that configuration is required, and then start Kibana once the process terminates

[joshdover]

If we want a UI, the most practical solution may be to have this as a totally separate process/server than our current Kibana server

Some benefits I see from this approach:

• More secure since we wouldn't need to allow the main Kibana process to be able to write config to disk
• We can still read config by leveraging the recently extracted @kbn/config package.
• Keeps Core simpler
• Eliminates risk of plugins using old ES config before interactive setup has been completed.

Cons:

• More work up front
• Now we have 3 different sets of HTTP routes that are served at different points
• We probably still need the /api/status endpoint serving on this server as well, though it probably would be acceptable to just always return a single status summary without all of the detailed plugin status information.

Things that are hard/annoying about both approaches:

• The Content-Security-Policy & i18n setup may need to be duplicated
• Can't use any frontend Core services without bootstrapping them yourself, which may require additional overhead or refactoring to be sure things are consistent.

An effort estimate for each path would be helpful, as well as any context on any other future additions we may make to this UI. If there are no plans for additional things, then maybe the notReadyServer path is more practical to get this UI in sooner. However, if we can imagine other things that users may be configuring with this UI, then maybe not. From a technical standpoint, there are very few configurations that I can imagine we'd need to require users to set up before starting Kibana, but I could be wrong.

[legrego]

If we want a UI, the most practical solution may be to have this as a totally separate process/server than our current Kibana server

Agreed -- another benefit we get from decoupling from the notReady server is that we can run on a different port altogether. If there is a reverse proxy already setup to forward traffic to 5601, then we would be exposing this setup UI outside of the loopback, thereby allowing a malicious (or accidental) user to configure Kibana to point to a different cluster altogether. It would also help reinforce the fact that this screen isn't Kibana proper, but rather an intermediate step to using Kibana.

Another question comes to mind: Will this page be authenticated? Or will the first user accessing the application be able to edit the Kibana configuration file (maybe do more?) with 'SU' privileges?

It won't be authenticated in the traditional sense. One idea I had was to have this setup process write a secret to its stdout, which we then ask the user to enter into the UI. This would be one added layer of protection if the setup screen was accidentally exposed to the Internet/intranet.

An effort estimate for each path would be helpful, as well as any context on any other future additions we may make to this UI. If there are no plans for additional things, then maybe the notReadyServer path is more practical to get this UI in sooner. However, if we can imagine other things that users may be configuring with this UI, then maybe not.

I'm not aware of any future additions, but this could very well open the door to more once it's available.

From a technical standpoint, there are very few configurations that I can imagine we'd need to require users to set up before starting Kibana, but I could be wrong.

No, I don't think there are many that are required, but there are quite a few that we could add in a followup to improve the getting started experience. I think we'd want to set the following initially (may be incomplete)

server.host
server.ssl.enabled
server.ssl.keystore.path
server.ssl.keystore.password
server.ssl.certificateAuthorities

elasticsearch.hosts
elasticsearch.username
elasticsearch.password
elasticsearch.ssl.certificateAuthorities
elasticsearch.ssl.verificationMode

xpack.encryptedSavedObjects.encryptionKey
xpack.security.encryptionKey
xpack.reporting.encryptionKey

[kobelb]

I share all of your concerns here. We'll need to take some time to discuss our options, and see what makes the most sense. For what it's worth, Elasticsearch will have a similar set of challenges, as it will also need to update its own configuration, while dealing with potential conflicts and permission constraints. I'd like to table this for now and instead focus on what (if anything) we can do to enable the experience. We'll collaborate with the ES team to see if there's an approach for config management that makes sense for both systems.

Where did we end up on this? Are we going to be having both Kibana and ES write the configuration to disk?

[legrego]

Where did we end up on this? Are we going to be having both Kibana and ES write the configuration to disk?

At this point, we are planning to have both Kibana and ES write the configuration to disk. I haven't verified if this would work, but we could potentially have Kibana change the file permissions after writing its config to disk, so that it's not able to alter the config once setup mode has completed.

Regarding multiple config files, it seems that support was added a long time ago (#6825 👋 Spencer) to support kibana.dev.yml. It's not clear to me if this is still beneficial or not. If it helps, we can attempt to drop support for this in 8.0.

[kobelb]

At this point, we are planning to have both Kibana and ES write the configuration to disk. I haven't verified if this would work, but we could potentially have Kibana change the file permissions after writing its config to disk, so that it's not able to alter the config once setup mode has completed.

At one point, when Kibana was installed using a RPM/DEB, the /etc/kibana/kibana.yml file was unwriteable by the kibana user that ran the service. We should confirm with @elastic/kibana-operations whether or not this would require changes to our packaging.

[jbudz]

Should be fine - the process is run as user/group kibana. File writes are available under that group.

~/dev/kibana(docs/build) » sudo -u kibana ls -lah /etc/kibana                                                                                                                                          
total 36K
drwxr-s---   2 root kibana 4.0K Apr 27 14:19 .
drwxr-xr-x 149 root root    12K Apr 27 14:19 ..
-rw-rw----   1 root kibana  130 Apr 21 15:10 kibana.keystore
-rw-r--r--   1 root kibana   62 Apr 21 15:10 .kibana.keystore.initial_md5sum
-rw-rw----   1 root kibana 5.0K Apr 20 15:34 kibana.yml
-rw-r--r--   1 root kibana  216 Apr 20 15:34 node.options
~/dev/kibana(docs/build) » echo "test" | sudo -u kibana tee -a /etc/kibana/kibana.yml                                                                                                                
test                                                                                                                                                                                                               
~/dev/kibana(docs/build) » sudo -u kibana cat /etc/kibana/kibana.yml | grep test                                                                                                                           
test

[kobelb]

Another question comes to mind: Will this page be authenticated? Or will the first user accessing the application be able to edit the Kibana configuration file (maybe do more?) with 'SU' privileges?

It won't be authenticated in the traditional sense. One idea I had was to have this setup process write a secret to its stdout, which we then ask the user to enter into the UI. This would be one added layer of protection if the setup screen was accidentally exposed to the Internet/intranet.

Do we have more details on how authentication and authorization will work for interactive setup mode?

[azasypkin]

Do we have more details on how authentication and authorization will work for interactive setup mode?

There are four "types" of authentication here:

• Kibana should be able to check that the user configuring connection to ES is the legitimate operator of this Kibana instance. We don't have exact details on how that would work yet, but most likely it will be a variation of the approach @legrego suggested (some sort of secret in stdout).

• Kibana should be able to check that ES cluster it connects to with the help of the enrollment token is the correct one. Kibana (or ES JS client to be precise) will use certificate fingerprint included in the enrollment token to perform this check.

• Kibana should be able to authenticate to ES enrollment API. Kibana will use a short-lived API key included in the enrollment token to do that.

• Kibana should be able to authenticate to ES after the enrollment process. During the enrollment process we expect ES to generate unique credentials for the Kibana system service account that we'll save to the kibana.yml/keystore (TBD), and won't expose to the user.

Apologies if it's all known to you already and you were looking for something else, let me know if it's the case.

[kobelb]

Thanks for the elaboration, @azasypkin; and apologies for the vague question, there is clearly a lot of different forms of authc/authz going on here. The following is what I was primarily concerned about since we can't use ES as the basis for this authc/authz because Kibana can't communicate with ES at the time the user is using the UI:

Kibana should be able to check that the user configuring connection to ES is the legitimate operator of this Kibana instance. We don't have exact details on how that would work yet, but most likely it will be a variation of the approach @legrego suggested (some sort of secret in stdout).

If we're outputting the secret to stdout, if users install Kibana as a Linux subsystem or a Windows services, they won't easily be able to see stdout, right? We'll also want to make sure that we don't log this secret to the standard log output, as it's common for users to ingest the logs into Elasticsearch or other external systems, and we shouldn't assume that just because you have access to the logs that you're an operator.

[azasypkin]

If we're outputting the secret to stdout, if users install Kibana as a Linux subsystem or a Windows services, they won't easily be able to see stdout, right?

That's correct, the same goes for the case when users use Kibana Docker image in a detached mode. Unfortunately, we don't have much details here yet, and are evaluating different options from not-requiring this secret during the first few minutes after Kibana starts, to guiding users where to get this secret from or relaxing certain requirements if Kibana uses private IP range to connect to ES.

We'll likely end up with a compromise solution, I'll keep this thread updated with our progress on that matter.

We'll also want to make sure that we don't log this secret to the standard log output, as it's common for users to ingest the logs into Elasticsearch or other external systems, and we shouldn't assume that just because you have access to the logs that you're an operator.

Yeah, we keep this in mind, thanks.

[azasypkin]

We'll likely end up with a compromise solution, I'll keep this thread updated with our progress on that matter.

Okay, after considering potential risks and UX implications, we agreed that the risk is fairly low, and requiring users to enter one more token/code/whatever will basically sabotage the idea of a simple and user-friendly setup. We settled on the following Security vs Usability compromise (at least for the first stage):

When Kibana starts in the interactive setup mode (= connection to ES isn't configured) it will allow anyone who can access it to configure the connection to Elasticsearch (either via enrollment process or manually), but only for a limited time (exact time is TBD).

Once this time limit has passed, Kibana will no longer allow configuration over UI and ask users to restart Kibana (and hence restart the timer) that would serve as reasonable proof they are the legitimate operators.

[kobelb]

When Kibana starts in the interactive setup mode (= connection to ES isn't configured) it will allow anyone who can access it to configure the connection to Elasticsearch (either via enrollment process or manually), but only for a limited time (exact time is TBD).

What kibana.yml settings will this allow a potential attacker to have full control over? Wouldn't this allow an attacker to get access to an unsecured instance of Elasticsearch that is firewalled off from the public internet using Kibana?

[jportner]

What kibana.yml settings will this allow a potential attacker to have full control over? Wouldn't this allow an attacker to get access to an unsecured instance of Elasticsearch that is firewalled off from the public internet using Kibana?

Interactive setup mode will only be available if Kibana is not connected to Elasticsearch.

The primary risk here is that, if Kibana is left in interactive setup mode, an attacker could set up a malicious ES cluster and "hijack" Kibana, configuring it to connect to the malicious cluster. The impact of this alone is low, but this would enable follow-on attacks:

• Steal data - conduct a phishing campaign to trick users into using Kibana to add data to the malicious cluster
• Compromise the Kibana machine - use some unknown ES vulnerability to cause the Kibana process to execute a malicious payload

Our mitigation for this is to limit the time that interactive setup mode is available, which seems to be an acceptable usability/security tradeoff. The likelihood of an attacker discovering a Kibana instance while it is in interactive setup mode (with a time limit on the order of hours) is extremely low.

I don't think the scenario you described is applicable here. Elasticsearch will have security enabled by default, and if I understand correctly, you cannot use an ES enrollment token to force Kibana to connect to an unsecured cluster. Enrollment tokens will only work for secured clusters.

[kobelb]

@jportner couldn't this be used as a "honey pot" for valid credentials as well? An attacker could configure the Kibana instance that's in interactive setup mode to connect to their own cluster, and then monitor the network traffic when users try to login.

I don't think the scenario you described is applicable here. Elasticsearch will have security enabled by default, and if I understand correctly, you cannot use an ES enrollment token to force Kibana to connect to an unsecured cluster. Enrollment tokens will only work for secured clusters.

Users will still be able to disable security in Elasticsearch. As a result, if there's an internal Elasticsearch cluster that has security disabled and it's only protected by firewall rules or network segmentation, an attacker could use the Kibana instance to circumvent this protection.

[kobelb]

Our mitigation for this is to limit the time that interactive setup mode is available, which seems to be an acceptable usability/security tradeoff. The likelihood of an attacker discovering a Kibana instance while it is in interactive setup mode (with a time limit on the order of hours) is extremely low.

How would the time limit be enforced? Would it be ephemeral and the timer would reset when Kibana restarts? If so, an attacker could simply crash an instance of Kibana using an exploit and when it comes back up, it'll be in interactive setup mode.

[jportner]

@jportner couldn't this be used as a "honey pot" for valid credentials as well? An attacker could configure the Kibana instance that's in interactive setup mode to connect to their own cluster, and then monitor the network traffic when users try to login.

Yes, I think that would fall under "Steal data" in #89287 (comment).

Users will still be able to disable security in Elasticsearch. As a result, if there's an internal Elasticsearch cluster that has security disabled and it's only protected by firewall rules or network segmentation, an attacker could use the Kibana instance to circumvent this protection.

Can you elaborate on this? At a minimum it sounds like it would necessitate an attacker finding a vulnerable Kibana instance (in interactive setup mode) and having prior knowledge of an unsecured ES cluster.

If you're suggesting that hijacking the Kibana instance could allow RCE with a follow-on attack to discover/exploiting unsecured ES clusters; in that case i'd say this falls under "Compromise the Kibana machine" in #89287 (comment).

How would the time limit be enforced? Would it be ephemeral and the timer would reset when Kibana restarts? If so, an attacker could simply crash an instance of Kibana using an exploit and when it comes back up, it'll be in interactive setup mode.

Sorry if it wasn't clear in Oleg's comment; the idea is that after the time limit, the preboot service actually writes something to disk (probably a setting to kibana.yml) that disables interactive setup mode. The only way to re-enable interactive setup mode after that is to remove that setting and restart Kibana.

[kobelb]

@jportner couldn't this be used as a "honey pot" for valid credentials as well? An attacker could configure the Kibana instance that's in interactive setup mode to connect to their own cluster, and then monitor the network traffic when users try to login.

Yes, I think that would fall under "Steal data" in #89287 (comment).

If you intended to use the term "data" to also include credentials, then I agree. However, the original phrasing mentioned tricking users to add data to an attacking cluster, not phish for valid credentials for other Elasticsearch clusters.

Can you elaborate on this? At a minimum it sounds like it would necessitate an attacker finding a vulnerable Kibana instance (in interactive setup mode) and having prior knowledge of an unsecured ES cluster.

This would require that the attacker have prior knowledge of an unsecured cluster. However, it would allow them to circumvent a user's intended network segmentation.

If you're suggesting that hijacking the Kibana instance could allow RCE with a follow-on attack to discover/exploiting unsecured ES clusters; in that case i'd say this falls under "Compromise the Kibana machine" in #89287 (comment).

This was not my intent.