Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pkg/asset/tls: self-sign kube-ca #1179

Merged
merged 4 commits into from
Feb 19, 2019

Conversation

mrogers950
Copy link
Contributor

@mrogers950 mrogers950 commented Feb 1, 2019

This PR turns the kube-ca into a self-signed CA rather than intermediate from the root CA. Detaching it from the root CA trust chain gives us a proper separation of the trust domain while making it possible to accommodate non-golang clients without compromising the intended trust separation.

@openshift-ci-robot openshift-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Feb 1, 2019
@abhinavdahiya
Copy link
Contributor

@mrogers950

  • does this have a exception approval?
  • can you provide details in the PR or point to a doc that tells why and where the current setup is failing?
  • details on if in the future the users ask to bring their own CAs , how would that work?

@abhinavdahiya
Copy link
Contributor

/unassign @russellb
/assign @crawford @abhinavdahiya

@openshift-bot
Copy link
Contributor

/retest

@mrogers950
Copy link
Contributor Author

I've rebased this and changed to tackling self-signing one CA at a time, first the kube-ca.

  • da11378 is so the self-signed kube-ca does not get appended to the kube-apiserver cert (a server sending a self-signed CA causes client failures)
  • 4572400 and 014e9dc fix the kubeconfigs to use the kube-ca for the certificate authority data rather than root CA which is no longer part of the trust chain. Without these, the initial bootstrap fails.

The current problem that I am trying to debug is that now the network-operator can't verify the api server cert, even though the kubeconfig that it is mounting now contains the kube-ca (although it is passed in as a --url-only-kubeconfig so that suggests it may not be using the CA data from it).

CONTAINER ID        IMAGE                                                                                                                                           CREATED             STATE               NAME                       ATTEMPT             POD ID
89e2c6edb4431       4d81374ba15b441eb35e1238dc4008d8e25aa5374666fc725b1f4240f77ad5ca                                                                                3 minutes ago       Exited              network-operator           10                  f40e5d397c329
1bb51fbe17a06       registry.svc.ci.openshift.org/openshift/origin-release@sha256:55371073f7890d085ef45ef8ef43f7ca4ce77cb92f6efb85334e966933ebe8d9                  30 minutes ago      Running             cluster-version-operator   0                   97df4ce6304d1
5f1d6756fe9f4       registry.svc.ci.openshift.org/openshift/origin-v4.0-2019-02-08-175031@sha256:d3e128c248fa723e267d1a4e4eba9a8eb6e55e7e862ed8a5e0f49ea1d2a1cb13   32 minutes ago      Running             etcd-member                0                   8b641e5e6f030
0041ca4b4f90c       b02de22ff740f0bfa7e5dde5aa1a8169051375a5f0c69c28fafefc9408f72b06                                                                                32 minutes ago      Exited              certs                      1                   8b641e5e6f030
aa76294478cd5       registry.svc.ci.openshift.org/openshift/origin-v4.0-2019-02-08-175031@sha256:9a17718182f248141a411a6b7395333a665ac851d69e3deb91195af316ba177c   32 minutes ago      Exited              discovery                  0                   8b641e5e6f030
[core@ip-10-0-3-65 ~]$ sudo crictl logs 89e2c6edb4431
2019/02/08 20:09:53 Go Version: go1.10.3
2019/02/08 20:09:53 Go OS/Arch: linux/amd64
2019/02/08 20:09:53 operator-sdk Version: v0.1.0
2019/02/08 20:09:53 overriding kubernetes api to https://m9-api.devcluster.openshift.com:6443
time="2019-02-08T20:09:53Z" level=info msg="trying to become the leader"
2019/02/08 20:09:53 Get https://m9-api.devcluster.openshift.com:6443/api?timeout=32s: x509: certificate signed by unknown authority

@squeed where does the network-operator source its CA data from for the API connection?

@squeed
Copy link
Contributor

squeed commented Feb 11, 2019

@mrogers950 the network operator uses the default "in-cluster" config + CA. It only pulls the apiserver URL from the kubelet's kubeconfig.

@mrogers950
Copy link
Contributor Author

Retesting after openshift/cluster-kube-controller-manager-operator#152
/retest

@deads2k
Copy link
Contributor

deads2k commented Feb 11, 2019

@mrogers950 alright, you're getting installed now. Looks like something isn't happy though.

@mrogers950
Copy link
Contributor Author

Retesting for a debugging run
/retest

@mrogers950
Copy link
Contributor Author

The cluster comes up and no crashlooping pods. The test failures are all with the use of e2e.RunHostCmd(), unable to upgrade connection: Unauthorized so it seems there may be something about the change to using kube-ca in the kubeconfig that the test framework does not like.

@mrogers950
Copy link
Contributor Author

/retest

3 similar comments
@mrogers950
Copy link
Contributor Author

/retest

@mrogers950
Copy link
Contributor Author

/retest

@mrogers950
Copy link
Contributor Author

/retest

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Feb 15, 2019

@mrogers950: The following tests failed, say /retest to rerun them all:

Test name Commit Details Rerun command
ci/prow/e2e-openstack 51d6874 link /test e2e-openstack
ci/prow/e2e-libvirt 51d6874 link /test e2e-libvirt
ci/prow/launch-aws 51d6874 link /test launch-aws

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@mrogers950
Copy link
Contributor Author

/retest

@mrogers950
Copy link
Contributor Author

@abhinavdahiya tests are green, PTAL

@abhinavdahiya
Copy link
Contributor

This PR turns the aggregator, etcd, service-signer, kube and mco server CAs into self-signed CAs rather than intermediates off of the root CA. Detaching these from the root CA trust chain gives us a proper separation of the trust domains while making it possible to accommodate non-golang clients without compromising the intended trust separation.
More discussion at openshift/cluster-kube-controller-manager-operator#113

The only CA i see is the kubeCA getting self signed, what about the others ?

can you include more details to commits based on https://github.com/openshift/installer/blob/master/CONTRIBUTING.md#commit-message-format

/kind bug

/cc @crawford

@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Feb 18, 2019
@mrogers950 mrogers950 changed the title pkg/asset/tls: self-sign cluster CAs pkg/asset/tls: self-sign kube-ca Feb 18, 2019
Matt Rogers added 4 commits February 18, 2019 13:01
Detach kube-ca from the root-ca chain in order to make it a proper independent
chain of trust, and ensure compatibility with non-golang TLS clients that need
to trust kube-ca.

Part of https://jira.coreos.com/browse/CORS-999
A self-signed CA must not be included in a server certificate bundle.
Admin now requires the kube-ca CA data, not root CA.
The kubelet kubeconfig now requires kube-ca CA data, not root CA.
@mrogers950
Copy link
Contributor Author

@abhinavdahiya I had reduced the scope of this PR to take care of only kube-ca (the others will follow) and updated the text accordingly.

@abhinavdahiya
Copy link
Contributor

/approve

@abhinavdahiya I had reduced the scope of this PR to take care of only kube-ca (the others will follow) and updated the text accordingly.

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 18, 2019
@abhinavdahiya
Copy link
Contributor

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 19, 2019
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya, mrogers950

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot merged commit 820ff4c into openshift:master Feb 19, 2019
wking added a commit to wking/openshift-installer that referenced this pull request Feb 19, 2019
Through 820ff4c (Merge pull request openshift#1179 from mrogers950/ca_roots,
2019-02-19).
wking added a commit to wking/openshift-installer that referenced this pull request Feb 19, 2019
Through 820ff4c (Merge pull request openshift#1179 from mrogers950/ca_roots,
2019-02-19).
jstuever pushed a commit to jstuever/openshift-installer that referenced this pull request Feb 25, 2019
Through 820ff4c (Merge pull request openshift#1179 from mrogers950/ca_roots,
2019-02-19).
flaper87 pushed a commit to flaper87/installer that referenced this pull request Feb 28, 2019
Through 820ff4c (Merge pull request openshift#1179 from mrogers950/ca_roots,
2019-02-19).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. kind/bug Categorizes issue or PR as related to a bug. lgtm Indicates that a PR is ready to be merged. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants