Introduce PKI authentication provider.

[azasypkin]

This PR introduces PKI authentication provider that is supposed to authenticate Kibana users via PKI Elasticsearch realm.

Here is the list of blockers we need to resolve before we can merge this (PR includes all these changes as a reference of what we need, but we need a help of the @elastic/kibana-platform to merge them or get alternative proposals):

• Legacy schema shouldn't complain about server.ssl.requestCert (introduced in a ^half^ working state in Allow requestCert option to be set #38920) cleanup http config in legacy paltform for migrated params #42818
• KibanaRequest should allow access to the "detailed" (the one that includes issuerCertificate field) peer certificate. Or maybe there is a better way, not sure. add socket.getPeerCertificate to KibanaRequest #42929
• Replace requestCert with clientAuthentication proposed in Introduce PKI authentication provider. #42606 (comment) Add TLS client authentication support. #43090

---

Notes

• Figure out what to do with reporting (at the initial implementation, reporting will require server.ssl.clientAuthentication to be set to optional, support of required will be discussed in Reporting should work with server.ssl.clientAuthentication set to required #43309)
• Write unit tests (done) and integration tests, blocked by Add SSL support when starting Elasticsearch from yarn es #42527 and Add Elasticsearch SSL support for integration tests #41765 (done)
• Test in IE 11
• Test with non self-signed certificate (tested with Let's Encrypt, ping me if you want one as well)

---

How to test

1. Generate certificates for Elasticsearch, Kibana and End User or use my test certificate-bundle.zip. Steps below assume that you use certificates from the attached certificate bundle.

2. Run Elasticsearch with the following (or similar) config (currently works only with proxied-pk Elasticsearch feature branch):

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: certificate-bundle/es/es.key
xpack.security.http.ssl.certificate: certificate-bundle/es/es.crt
xpack.security.http.ssl.certificate_authorities: ["certificate-bundle/ca/ca.crt"]
xpack.security.http.ssl.client_authentication: required
xpack.security.http.ssl.verification_mode: certificate

xpack.security.enabled: true
xpack.security.authc.token.enabled: true

xpack.security.authc.realms.native.native1.order: 0

xpack.security.authc.realms.pki.pki1.order: 1
xpack.security.authc.realms.pki.pki1.delegation.enabled: true
xpack.security.authc.realms.pki.pki1.certificate_authorities: ["certificate-bundle/ca/ca.crt"]

3. Add role mappings for the End User (certificate-bundle/user/user.crt) and Kibana internal user (certificate-bundle/kibana/kibana.crt) authenticated with PKI realm, e.g.:

POST https://localhost:9200/_security/role_mapping/kibana_pki
Accept: application/json
Content-Type: application/json

{
  "roles" : [ "kibana_system" ],
  "rules" : { "field" : { "dn" : "CN=kibana" } },
  "enabled": true
}

------

POST https://localhost:9200/_security/role_mapping/user_pki
Accept: application/json
Content-Type: application/json

{
  "roles" : [ "superuser" ],
  "rules" : { "field" : { "dn" : "CN=user" } },
  "enabled": true
}

4. Run Kibana with the following config:

xpack.security.authc.providers: [pki, basic]

server.ssl.enabled: true
server.ssl.key: certificate-bundle/kibana/kibana.key
server.ssl.certificate: certificate-bundle/kibana/kibana.crt
server.ssl.certificateAuthorities: certificate-bundle/ca/ca.crt
server.ssl.requestCert: true

elasticsearch.hosts: https://localhost:9200
elasticsearch.ssl.certificate: certificate-bundle/kibana/kibana.crt
elasticsearch.ssl.key: certificate-bundle/kibana/kibana.key
elasticsearch.ssl.certificateAuthorities: certificate-bundle/ca/ca.crt
elasticsearch.ssl.verificationMode: certificate

4. Add certificate-bundle/user/user.pfx and certificate-bundle/kibana/kibana.pfx (to test 2 different users) to the list of your certificates in the browser.

---

Blocked by: elastic/elasticsearch#45906, #42527 and #41765 (to be able to run Elasticsearch with TLS in tests), #42069 (optional dependency), #42818, #42929, #43090

Documentation: #43572

Fixes: #7341

"Release Note: Adding support for PKI authentication in Kibana via Elasticsearch PKI realm."

/cc @albertzaharovits

[elasticmachine]

Pinging @elastic/kibana-security

[kobelb]

@tvernum I have a hazy memory of us discussing some validation that Kibana should be performing using the CN of the certificate, and Cloud potentially impacting this logic. Would you mind refreshing my memory? Apologies for not taking better notes.

[kobelb]

Just a note so I don't forget. I can't think of a way that PKI in Kibana will work if there's a HTTP layer reverse-proxy in front of Kibana, because we won't have a way to request the client's certificates. We'll want to ensure that this isn't going to be an issue for Kibana in ECE/ESS because if they're doing TLS termination before accessing Kibana, we won't have access to the client's certificates. As long as Cloud is using a TCP layer proxy we should be fine...

Also, I think we're going to a way to set rejectUnauthorized to false so that we can use PKI with the Basic auth provider. Ideally, we'd replicate what ES does and allow the user to specify client_authentication with either required, optional or none, with none being the default. We'd then translate required to requestCert: true and rejectUnauthorized: true; optional to requestCert: true and rejectUnauthorized: false; and none to requestCert: false.

If users are going to want to use Reporting, they're going to have to use client_authentication: optional, because we won't have the key to perform the TLS handshake. Normally, users wouldn't be able to get access to the access token, so perhaps we are alright with Reporting replaying the access token and skipping the cert fingerprint validation because this isn't a normal workflow? It's just a bummer that I can't think of a way to ever allow users to set client_authentication: required without drastic changes to the Reporting architecture.

[azasypkin]

Just a note so I don't forget. I can't think of a way that PKI in Kibana will work if there's a HTTP layer reverse-proxy in front of Kibana, because we won't have a way to request the client's certificates. We'll want to ensure that this isn't going to be an issue for Kibana in ECE/ESS because if they're doing TLS termination before accessing Kibana, we won't have access to the client's certificates. As long as Cloud is using a TCP layer proxy we should be fine...

Good point! I believe it's not even supported for Elasticsearch yet: https://github.com/elastic/cloud/issues/36871. We could theoretically try to workaround this by forcing proxy to forward certificate to us in a header (e.g. HAProxy's directive http-request set-header X-SSL-ClientCert-Chain c937a62d7218-%{+Q}[ssl_c_der,base64]) with some sort of random key configured in Kibana (e.g. xpack.security.authc.pki.proxy.{enabled, headerName, randomKey}). This approach has its own drawbacks and unknowns and it'd be much better to have a proper support in Cloud (and looking at the ticket above, it seems Cloud will need to add support because of some other reasons too).

Also, I think we're going to a way to set rejectUnauthorized to false so that we can use PKI with the Basic auth provider. Ideally, we'd replicate what ES does and allow the user to specify client_authentication with either required, optional or none, with none being the default. We'd then translate required to requestCert: true and rejectUnauthorized: true; optional to requestCert: true and rejectUnauthorized: false; and none to requestCert: false.

Yeah, I agree, that would be consistent with ES, for the core part of Kibana. But I'm not entirely sure if we want to allow PKI authentication provider in non-required configuration. Currently ES PKI realm works with any verification_mode, but I'm not sure if it's intended, let's check with ES team. Basic auth provider will still work, but we'll additionally require a trusted client certificate to establish connection.

If users are going to want to use Reporting, they're going to have to use client_authentication: optional, because we won't have the key to perform the TLS handshake. Normally, users wouldn't be able to get access to the access token, so perhaps we are alright with Reporting replaying the access token and skipping the cert fingerprint validation because this isn't a normal workflow? It's just a bummer that I can't think of a way to ever allow users to set client_authentication: required without drastic changes to the Reporting architecture.

Do you think it's not viable option to introduce xpack.reporting.kibanaServer.ssl.{key/cert} that will point to a reporting only key and certificate, which CA will be trusted via server.ssl.certificateAuthorities, but not by ES PKI realm (when access token expires and PKI auth provider will try to create a new token, ES will reject and that's it)?

[azasypkin]

Also, I think we're going to a way to set rejectUnauthorized to false so that we can use PKI with the Basic auth provider. Ideally, we'd replicate what ES does and allow the user to specify client_authentication with either required, optional or none, with none being the default. We'd then translate required to requestCert: true and rejectUnauthorized: true; optional to requestCert: true and rejectUnauthorized: false; and none to requestCert: false.

Just realized that even though requestCert was introduced in 7.3 already it's broken, so it's a good chance to drop it completely in favor of clientAuthentication without usual BWC hassle. Let's see if we can squeeze it into 7.4.

[tvernum]

I have a hazy memory of us discussing some validation that Kibana should be performing using the CN of the certificate

The conversation I recall was in the context of mutual-TLS between co-operating Kibana instances in large scale environments. I don't think we discussed it for PKI auth.

[kobelb]

The conversation I recall was in the context of mutual-TLS between co-operating Kibana instances in large scale environments. I don't think we discussed it for PKI auth.

That makes a lot more sense, thanks Tim!!

[kobelb]

Do you think it's not viable option to introduce xpack.reporting.kibanaServer.ssl.{key/cert} that will point to a reporting only key and certificate, which CA will be trusted via server.ssl.certificateAuthorities, but not by ES PKI realm (when access token expires and PKI auth provider will try to create a new token, ES will reject and that's it)?

That's an interesting idea. We'd have to generate the certificate for Reporting using the CA that we have configured, and then ensure it doesn't match any of the role mappings configured in ES... Do you happen to know if that's possible? I guess we could potentially require the user to do this themselves if they absolutely want to use client_authentication: required.

[tvernum]

We'd have to generate the certificate for Reporting using the CA that we have configured, and then ensure it doesn't match any of the role mappings configured in ES.

I'm not an expert on Node-TLS, but that seems like more than you need. You could simply generate a standalone cert (e.g. elasticsearch-certutil cert --name kibana-reporting-client) and implicitly update the trust anchors to include the client cert for reporting. It doesn't need to be the same CA that is used to sign PKI certs because you control the trust config.

If you do that, ES will never authenticate that cert, because it belongs to a chain that we don't know about.
However, for safety and potentially convenience, you might want to recognise the fingerprint automatically and switch to token auth without trying PKI.

[kobelb]

I'm not an expert on Node-TLS, but that seems like more than you need. You could simply generate a standalone cert (e.g. elasticsearch-certutil cert --name kibana-reporting-client) and implicitly update the trust anchors to include the client cert for reporting. It doesn't need to be the same CA that is used to sign PKI certs because you control the trust config.

If you do that, ES will never authenticate that cert, because it belongs to a chain that we don't know about.
However, for safety and potentially convenience, you might want to recognise the fingerprint automatically and switch to token auth without trying PKI.

Thanks Tim! Aleh suggested a similar solution when we were chatting earlier today.

For the initial implementation, we were considering requiring users to use clientAuthentication: 'optional' if they wish to use Reporting.

If this becomes intolerable, we were planning on implementing the approach that you outlined where we manually require users to generate an independent CA which Kibana's http server trusts (which isn't used by the PKI realm) and a cert/key signed by this independent CA to be used by Reporting.

If this is too much work, we talked about pre-providing the CA and cert/key to be used by Reporting. @tvernum I know that previously you mentioned your experience with some users requiring that all certs/keys be signed by a centralized "corporate CA". Do you think us implicitly trusting this pre-provided CA and cert/key would exacerbate the situation?

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[azasypkin]

@kobelb added tests you suggested, PR should be ready for the second pass whenever you have time. Thanks!

[rudolf]

👍 for platform changes

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[elasticmachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request

[azasypkin]

7.x/7.4.0: 8199253