Cannot Disable HTTP when using managed-certificates. HTTPS + HTTP are both provisioned #764

jakebolam · 2019-06-01T20:22:47Z

Cannot disable HTTP when using managed-certificates

When specifying the annotation kubernetes.io/ingress.allow-http: "false" the ingress still accepts HTTP traffic.

Versions

Kubernetes master/node-kube version: 1.12.7-gke.17 (Note: must be equal to or greater than this version, managed certificates had issues prior to this version see GoogleCloudPlatform/gke-managed-certs#2)

Additional Details:

I also tried (#738 (comment)):

Creating an ingress without kubernetes.io/ingress.allow-http: "false".
Verified the ingress was serving HTTPS
Then setting allow-http to "false"

Configuration

apiVersion: networking.gke.io/v1beta1
kind: ManagedCertificate
metadata:
  name: api-certifcate
spec:
  domains:
    - example.com
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.global-static-ip-name: "web-static-ip"
    networking.gke.io/managed-certificates: api-certifcate
spec:
  rules:
    - http:
        paths:
          - path: /my-service/*
            backend:
              serviceName: my-service
              servicePort: 8080

The text was updated successfully, but these errors were encountered:

jakebolam · 2019-06-01T20:24:27Z

This was previously reported, but the issue was masked by issues in managed-certificates up until recent versions. See #738

jakebolam · 2019-06-01T20:30:37Z

Debugging config from provisioned ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.gcp.kubernetes.io/pre-shared-cert: xxxxxx, xxxxx, xxxx
    ingress.kubernetes.io/backends: '{"xxxx":"HEALTHY","xxxx":"HEALTHY"}'
    ingress.kubernetes.io/forwarding-rule: xxxx
    ingress.kubernetes.io/https-forwarding-rule: xxxxx
    ingress.kubernetes.io/https-target-proxy: xxxxx
    ingress.kubernetes.io/ssl-cert: xxxxxxxx, xxxxxx, xxxx
    ingress.kubernetes.io/target-proxy: xxxxx
    ingress.kubernetes.io/url-map: xxxxxx
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.global-static-ip-name: web-static-ip
    networking.gke.io/managed-certificates: api-certifcate
  creationTimestamp: 2019-05-23T21:54:39Z
  generation: 2
  name: my-ingress
  namespace: default
  resourceVersion: "2986739"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/my-ingress
  uid:xxxxxx
spec:
  rules:
    - http:
        paths:
          - path: /my-service/*
            backend:
              serviceName: my-service
              servicePort: 8080
status:
  loadBalancer:
    ingress:
    - ip: xxx.xxx.xxx.xx

rramkumar1 · 2019-06-03T13:52:35Z

@jakebolam Ok, now I think you have discovered the bug. Turns out, we knew about this and were fixing it (#631) but the PR stalled a bit. I actually thought this PR was merged a while back and as a result I gave you the wrong advice in #738 (regarding adding the annotation after the frontend was configured) so apologies for that.

/assign

rramkumar1 · 2019-06-03T15:21:50Z

@jakebolam A potential workaround might be to delete the HTTP resources manually until we can ship the fix.

jakebolam · 2019-06-04T01:44:36Z

no worries, thanks for replying so fast. Looking forward to the PR getting merged.

Will do 😄

fejta-bot · 2019-09-02T03:03:13Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

AlienResidents · 2019-09-30T21:59:37Z

Hi,

I'm also looking forward to getting PR #631 merged. Any update on that stalled PR?

Chris-

bowei · 2019-09-30T23:47:50Z

BTW : We are looking at fixing the issue in #631 not on that specific PR.

bshibs · 2019-10-15T14:57:35Z

Just to confirm: manually deleting the LB Frontend port/protocol is a viable workaround until the fix lands?

fejta-bot · 2019-11-14T15:44:43Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

fejta-bot · 2019-12-14T16:27:22Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

k8s-ci-robot · 2019-12-14T16:27:30Z

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

svachmic · 2020-02-04T15:25:22Z

Could be related to this:
https://issuetracker.google.com/issues/35904733

Any signs that it will finally be resolved?

roobert · 2020-03-15T11:02:31Z

I'm also seeing this issue with GKE 1.14.10-gke.17.

Since it's not yet possible to redirect http->https (https://issuetracker.google.com/issues/35904733) and this annotation doesn't work when using GCP managed certificates, there doesn't appear to be a good way to easily disable the HTTP port left open by the GKE ingress controller.

Setheck · 2020-03-18T03:03:18Z

Unable to disable HTTP via annotation kubernetes.io/ingress.allow-http: "false" on 1.15.9-gke.12
Hoping for http->https rewrite 🤞
But should this issue be re-opened?

pgp · 2020-03-18T10:15:35Z

I think this should be solved as soon as possible, it's a real shame a security issue of this size is still open... Even without rewrite, disabling http completely should suffice at least for most restricted scenarios

svachmic · 2020-03-18T10:23:58Z

Quoting a Googler from the parent issue in Google's bug tracker I linked above:

We have an Alpha available for HTTP(S) Load Balancing URL Rewrites and Redirects. All Alpha terms apply. You need to be white-listed for the alpha. Please contact your Sales Representative or Account Team for more information.

We are proactively working on collecting feedback for the Alpha and will then open it up for public use without any white-listing.

Help is on its way, unfortunately I am not white-listed and cannot provide any feedback for how good/bad the solution is. If anybody is, please let us know anything that an alpha NDA allows you to.

jmound · 2020-04-22T21:27:02Z

/reopen

k8s-ci-robot · 2020-04-22T21:27:24Z

@jmound: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jmound · 2020-04-22T21:31:19Z

@jakebolam Feel free to reopen it, I don't think this issue should be closed.

svachmic · 2020-04-23T09:21:04Z

The 5year old underlying issue seems to have been resolved.

The question is - correctly asked as the next comment in the issue tracker - how can we make this work in GKE?

jmound · 2020-04-24T19:56:04Z

@rramkumar1 Are you able to reopen this issue? If not, should a new one be opened?

rramkumar1 · 2020-04-24T20:02:55Z

@jmound Let me refresh myself on what the exact issue was here and get back to you. Thanks.

rramkumar1 · 2020-04-29T03:07:39Z

@jmound So there are a couple things to unpack here:

The underlying issue that caused this issue to be filed has been fixed in clean up HTTP(S) resources when changing protocols #631. If you are not observing that the fix is working, then we can keep this open but otherwise this issue should be closed.
Support for SSL-redirects in GKE Ingress is being tracked in HTTP to HTTPS redirection #1075 and we will be providing updates on that issue.

Hope that helps.

skmatti · 2020-04-29T17:26:50Z

The fix(#894) is available in GKE version v1.16.4-gke.25 and above(or Ingress v1.8.2).

pgp · 2020-05-13T09:50:02Z

Hi, just tried this fix (allowHttp: false) with cluster version 1.17.5-gke.0
It actually works, in the sense that when I try to send a plain HTTP request, the response is a 404 from GCP backend. Anyway, with this strategy, any sensitive content in a plain HTTP request is actually sent over the internet (e.g. the body of a POST request); I rather expected the server to deny the TCP handshake - i.e. respond immediately with a Connection Reset, in so preventing sending data over the network at all. Do you believe such an improvement is possible?

rramkumar1 · 2020-05-13T13:26:58Z

@pgp I think the HTTPS redirects feature is what you are after. See #1075

pgp · 2020-05-13T14:31:15Z

No, in terms of actual security, http->https redirect is equivalent to the situation I described above, in that my plain HTTP request travels over the network, even if it shouldn't (the only difference is I receive a 302 response instead of a 404 one, but the concept is the same). Completely blocking the traffic on port 80 can be useful in case some REST API users do not comply with the given specifications (i.e. they deliberately or erroneously use http instead of https for an endpoint). Clearly, the steady-state situation is the same - a user receiving a 404 response will modify its client calls accordingly - but meanwhile some (possibly sensitive or private) data have been travelling in plaintext over the public internet.

rramkumar1 · 2020-05-13T14:37:22Z

@pgp I see, thanks for the clarification. I'll bring this up with our load balancing teams to see if there is a feature request here (or solution). Until this sort of feature is supported in the upstream load balancing API there is nothing we can do here.

rramkumar1 · 2020-05-13T14:41:57Z

@pgp Found https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security. Not sure if it helps in your case.

pgp · 2020-05-13T14:43:21Z

That always regards the response a server sends, nothing prevents the client from sending any kind of data in plain text before knowing this... Anyway, I'll look for possible updates on this issue, thanks for now.

Andrewangeta · 2020-05-31T00:05:25Z

I'm also facing this issue on the latest version and I'm losing my mind.

Andrewangeta · 2020-05-31T00:08:42Z

my hydrated yaml looks like this and the cert tries to create itself and every time it just ends up saying FAILED_NOT_VISIBLE after about 15 minutes of PROVISIONING

apiVersion: networking.gke.io/v1beta1
kind: ManagedCertificate
metadata:
  name: my-app-certificate
  namespace: my-app
spec:
  domains:
  - my.domain
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/force-ssl-redirect: "true"
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.global-static-ip-name: my-app-static-ip
    networking.gke.io/managed-certificates: my-app-certificate
  name: my-app-ingress
  namespace: my-app
spec:
  backend:
    serviceName: my-app
    servicePort: 80

jakebolam mentioned this issue Jun 1, 2019

Cannot Disable HTTP when using managed-certificates. Results in infinite Creating ingress and 'no frontend configuration' #738

Closed

k8s-ci-robot assigned rramkumar1 Jun 3, 2019

rramkumar1 added the kind/bug Categorizes issue or PR as related to a bug. label Jun 3, 2019

cblecker mentioned this issue Jun 20, 2019

Disable HTTP for Grafana kubernetes/test-infra#13099

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 2, 2019

skmatti mentioned this issue Oct 16, 2019

Bugfix: Delete unused GCE load-balancer resources on ingress spec change #894

Merged

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Nov 14, 2019

k8s-ci-robot closed this as completed Dec 14, 2019

jmound mentioned this issue Apr 24, 2020

Adding feature: http to https redirection GoogleCloudPlatform/pci-gke-blueprint#107

Merged

pgp mentioned this issue Nov 17, 2020

HTTP to HTTPS redirection #1075

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot Disable HTTP when using managed-certificates. HTTPS + HTTP are both provisioned #764

Cannot Disable HTTP when using managed-certificates. HTTPS + HTTP are both provisioned #764

jakebolam commented Jun 1, 2019 •

edited

Loading

jakebolam commented Jun 1, 2019

jakebolam commented Jun 1, 2019

rramkumar1 commented Jun 3, 2019

rramkumar1 commented Jun 3, 2019

jakebolam commented Jun 4, 2019 •

edited

Loading

fejta-bot commented Sep 2, 2019

AlienResidents commented Sep 30, 2019

bowei commented Sep 30, 2019

bshibs commented Oct 15, 2019

fejta-bot commented Nov 14, 2019

fejta-bot commented Dec 14, 2019

k8s-ci-robot commented Dec 14, 2019

svachmic commented Feb 4, 2020

roobert commented Mar 15, 2020

Setheck commented Mar 18, 2020

pgp commented Mar 18, 2020

svachmic commented Mar 18, 2020 •

edited

Loading

jmound commented Apr 22, 2020

k8s-ci-robot commented Apr 22, 2020

jmound commented Apr 22, 2020

svachmic commented Apr 23, 2020

jmound commented Apr 24, 2020

rramkumar1 commented Apr 24, 2020

rramkumar1 commented Apr 29, 2020

skmatti commented Apr 29, 2020

pgp commented May 13, 2020

rramkumar1 commented May 13, 2020

pgp commented May 13, 2020

rramkumar1 commented May 13, 2020 •

edited

Loading

rramkumar1 commented May 13, 2020

pgp commented May 13, 2020

Andrewangeta commented May 31, 2020

Andrewangeta commented May 31, 2020 •

edited

Loading

Cannot Disable HTTP when using managed-certificates. HTTPS + HTTP are both provisioned #764

Cannot Disable HTTP when using managed-certificates. HTTPS + HTTP are both provisioned #764

Comments

jakebolam commented Jun 1, 2019 • edited Loading

Versions

Additional Details:

Configuration

jakebolam commented Jun 1, 2019

jakebolam commented Jun 1, 2019

rramkumar1 commented Jun 3, 2019

rramkumar1 commented Jun 3, 2019

jakebolam commented Jun 4, 2019 • edited Loading

fejta-bot commented Sep 2, 2019

AlienResidents commented Sep 30, 2019

bowei commented Sep 30, 2019

bshibs commented Oct 15, 2019

fejta-bot commented Nov 14, 2019

fejta-bot commented Dec 14, 2019

k8s-ci-robot commented Dec 14, 2019

svachmic commented Feb 4, 2020

roobert commented Mar 15, 2020

Setheck commented Mar 18, 2020

pgp commented Mar 18, 2020

svachmic commented Mar 18, 2020 • edited Loading

jmound commented Apr 22, 2020

k8s-ci-robot commented Apr 22, 2020

jmound commented Apr 22, 2020

svachmic commented Apr 23, 2020

jmound commented Apr 24, 2020

rramkumar1 commented Apr 24, 2020

rramkumar1 commented Apr 29, 2020

skmatti commented Apr 29, 2020

pgp commented May 13, 2020

rramkumar1 commented May 13, 2020

pgp commented May 13, 2020

rramkumar1 commented May 13, 2020 • edited Loading

rramkumar1 commented May 13, 2020

pgp commented May 13, 2020

Andrewangeta commented May 31, 2020

Andrewangeta commented May 31, 2020 • edited Loading

jakebolam commented Jun 1, 2019 •

edited

Loading

jakebolam commented Jun 4, 2019 •

edited

Loading

svachmic commented Mar 18, 2020 •

edited

Loading

rramkumar1 commented May 13, 2020 •

edited

Loading

Andrewangeta commented May 31, 2020 •

edited

Loading