Ingress controller does not support HTTP2 with mutual TLS

[PassKit]

When creating an Ingress controller for a service using the service.alpha.kubernetes.io/app-protocols: '{"grpc":"HTTP2"}' annotation, Ingress-GCE will automatically create a HTTP2 health check to /.

When mutual TLS is deployed on the containers, this check will never pass, since the health check does not have access to certificates.

The following readiness probes that use gRPC Health Probe are ignored by GCE.

readinessProbe:
  exec:
    command: ["/hc", "-addr=127.0.0.1:9999", "-tls", "-tls-ca-cert=/tls/ca-chain.pem", "-tls-client-cert=/tls/client.pem", "-tls-client-key=/tls/client-key.pem"]
  initialDelaySeconds: 
livenessProbe:
  exec:
    command: ["/hc", "-addr=127.0.0.1:9999", "-tls", "-tls-ca-cert=/tls/ca-chain.pem", "-tls-client-cert=/tls/client.pem", "-tls-client-key=/tls/client-key.pem"]
  initialDelaySeconds: 10

There appears to be no way to override the creation of the GCE default HTTP2 health check.

[ahmetb]

Right now the Google Cloud HTTP(S) Load Balancer does not provide a way to configure mTLS cert/key pair (or even a CA cert) to make TLS requests to a specific health check path.

I suspect there's not much GKE can do here (other than maybe allowing customization of the health check path == '/') today. It's recommended that your service has a URL that responds with HTTP 200 without any authentication to verify that the backend is up.

[PassKit]

Unfortunately, this isn’t practical. Unless you can point me to a resource that shows how a server can run both with and without TLS in Golang? You can’t have 2 listeners on the same port and the official Go gRPC implementation doesn’t allow toggling of TLS per route.

[rramkumar1]

@PassKit ahmetb is correct, from ingress-gce point of view we can't really do much until Google Cloud Load Balancer supports mTLS.

In the meantime however, we are working on supporting customization of health check via BackendConfig so please look out for that very soon.

Can we close this issue?

[PassKit]

The issue still remains, GKE mandates a health check that is impossible to pass for a gRPC (HTTP2) service with mTLS. The workarounds of launching a separate server without mTLS to serve a health check path, or to relax mTLS service-wide are not acceptable. Adding custom paths to the health check will also not resolve this issue.

Since it seems unlikely that mTLS will be available on the Load Balancers any time soon, the ability to replace or remove the health check is the only solution I can see working. @ahmetb, for a GRPC service, you are not going to get a HTTP 200 response. I feel that the team does not have sufficient understanding of the issue, so I would advocate it remains open.

[rramkumar1]

@PassKit I am in full agreement that the baseline issue here is that we do not support health check customization yet. However, we already have an open issue for that (#42) and I would like to avoid having duplicates.

[PassKit]

Ok - looks like #42 covers it, therefore this can be closed