Add readiness probe so controller does not report "Ready" prematurely #751

vdhanan · 2021-02-18T18:03:16Z

Is this a bug fix or adding new feature?
Fixes #548

What is this PR about? / Why do we need it?
This change reuses the liveness probe endpoint for a readiness probe, so that the csi controller's status can be reported accurately. Currently, if the ec2 instance metadata is unavailable, the csi controller deployment reports as ready, tries to fetch the metadata, panics when it cannot, and then stops reporting as ready. With this change, the controller deployment will not report ready from the get-go, and as a result will not report an inaccurate status if the instance metadata is unavailable.

What testing is done?
Manually tested this using both kubectl and helm. Applied a global network policy to block access to the ec2 instance metadata, then created an ebs csi driver deployment. Visually confirmed with kubectl get deployments -n kube-system that the behavior changed and the controller no longer reported as ready before entering the CrashLoopBackOff state.

k8s-ci-robot · 2021-02-18T18:03:20Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2021-02-18T18:03:23Z

Welcome @vdhanan!

It looks like this is your first PR to kubernetes-sigs/aws-ebs-csi-driver 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/aws-ebs-csi-driver has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2021-02-18T18:03:24Z

Hi @vdhanan. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

vdhanan · 2021-02-18T18:12:45Z

signed CLA

ayberk · 2021-02-18T18:16:55Z

@vdhanan looks like github can't associate your commits with your account. You can squash your commits into one and then use git commit --amend --author="Author Name <email@address.com>" to fix it. Email should be one of the emails you have on your github account.

coveralls · 2021-02-18T18:29:18Z

Pull Request Test Coverage Report for Build 1662

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 81.712%

Totals
Change from base Build 1657:	0.0%
Covered Lines:	1747
Relevant Lines:	2138

💛 - Coveralls

vdhanan · 2021-02-18T23:06:08Z

@ayberk thanks, i think i fixed it.

ayberk · 2021-02-19T16:42:37Z

/ok-to-test

AndyXiangLi · 2021-02-19T18:01:42Z

charts/aws-ebs-csi-driver/templates/controller.yaml

@@ -91,6 +91,14 @@ spec:
            timeoutSeconds: 3
            periodSeconds: 10
            failureThreshold: 5
+          readinessProbe:


Do we need to update node.yaml as node service is calling metadata service on initialization?
https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/pkg/driver/node.go#L86

I tried creating a ebs-csi-node DaemonSet with a GlobalNetworkPolicy blocking metadata access, and it indeed reported as Ready, when it probably shouldn't. However, even after adding a readinessProbe in node.yaml, the node service still reported as Ready. Not sure how to fix that issue -- if it even is an issue -- so I'll defer it for now.

vdhanan · 2021-02-23T21:23:37Z

@ayberk can you please review this CR when you get a chance?

ayberk · 2021-02-24T00:28:47Z

charts/aws-ebs-csi-driver/Chart.yaml

@@ -2,7 +2,7 @@ apiVersion: v1
 appVersion: "0.9.0"
 name: aws-ebs-csi-driver
 description: A Helm chart for AWS EBS CSI Driver
-version: 0.9.9
+version: 0.10.0


Sorry for the confusion. You need to bump the patch and keep minor version the same, ie, 0.9.x.

k8s-ci-robot · 2021-02-26T17:11:31Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

vdhanan · 2021-03-01T18:24:09Z

run tests

ayberk · 2021-03-01T18:30:47Z

/retest

vdhanan · 2021-03-01T18:34:04Z

rebuild

ayberk · 2021-03-01T18:36:03Z

/test continuous-integration/travis-ci/pr

k8s-ci-robot · 2021-03-01T18:36:06Z

@ayberk: The specified target(s) for /test were not found.
The following commands are available to trigger jobs:

/test pull-aws-ebs-csi-driver-verify
/test pull-aws-ebs-csi-driver-unit
/test pull-aws-ebs-csi-driver-e2e-single-az
/test pull-aws-ebs-csi-driver-e2e-multi-az
/test pull-aws-ebs-csi-driver-migration-test-latest
/test pull-aws-ebs-csi-driver-external-test-latest

Use /test all to run the following jobs:

pull-aws-ebs-csi-driver-verify
pull-aws-ebs-csi-driver-unit
pull-aws-ebs-csi-driver-e2e-single-az
pull-aws-ebs-csi-driver-e2e-multi-az
pull-aws-ebs-csi-driver-migration-test-latest

In response to this:

/test continuous-integration/travis-ci/pr

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ayberk · 2021-03-01T18:36:40Z

Ok makes sense bot is not responsible for this. We can merge this manually.

vdhanan · 2021-03-01T18:46:09Z

Travis build passed here: https://travis-ci.org/github/kubernetes-sigs/aws-ebs-csi-driver/jobs/760984692

k8s-ci-robot · 2021-03-01T21:58:22Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

vdhanan · 2021-03-02T17:17:07Z

/test pull-aws-ebs-csi-driver-migration-test-latest

bump patch number

k8s-ci-robot · 2021-03-02T18:25:10Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

ayberk · 2021-03-02T19:05:28Z

/lgtm
/approve

k8s-ci-robot · 2021-03-02T19:05:35Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ayberk, vdhanan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ayberk]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Feb 18, 2021

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Feb 18, 2021

k8s-ci-robot requested review from bertinatto and d-nishi February 18, 2021 18:03

k8s-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Feb 18, 2021

vdhanan force-pushed the master branch from bd809ce to 41a980b Compare February 18, 2021 23:05

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Feb 18, 2021

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Feb 19, 2021

AndyXiangLi reviewed Feb 19, 2021

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 20, 2021

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Feb 23, 2021

ayberk reviewed Feb 24, 2021

View reviewed changes

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Feb 26, 2021

vdhanan force-pushed the master branch from 98d554a to c736af4 Compare February 26, 2021 17:12

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Feb 26, 2021

k8s-ci-robot removed the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Mar 1, 2021

k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Mar 1, 2021

vdhanan force-pushed the master branch from ddf47e0 to aca876e Compare March 1, 2021 21:59

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Mar 1, 2021

add readiness probe and bump helm chart version

25c0e59

bump patch number

vdhanan force-pushed the master branch from aca876e to 25c0e59 Compare March 2, 2021 18:21

k8s-ci-robot removed the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Mar 2, 2021

k8s-ci-robot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Mar 2, 2021

Merge remote-tracking branch 'upstream/master'

1e644f8

vdhanan force-pushed the master branch from 190a1c5 to 1e644f8 Compare March 2, 2021 18:26

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Mar 2, 2021

k8s-ci-robot assigned ayberk Mar 2, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 2, 2021

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 2, 2021

k8s-ci-robot merged commit c07e654 into kubernetes-sigs:master Mar 2, 2021

vdhanan mentioned this pull request Apr 29, 2021

REQUEST: New membership for vdhanan kubernetes/org#2680

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add readiness probe so controller does not report "Ready" prematurely #751

Add readiness probe so controller does not report "Ready" prematurely #751

vdhanan commented Feb 18, 2021 •

edited

Loading

k8s-ci-robot commented Feb 18, 2021

k8s-ci-robot commented Feb 18, 2021

k8s-ci-robot commented Feb 18, 2021

vdhanan commented Feb 18, 2021

ayberk commented Feb 18, 2021

coveralls commented Feb 18, 2021 •

edited

Loading

vdhanan commented Feb 18, 2021

ayberk commented Feb 19, 2021

AndyXiangLi Feb 19, 2021 •

edited

Loading

vdhanan Feb 23, 2021

vdhanan commented Feb 23, 2021

ayberk Feb 24, 2021

vdhanan Feb 24, 2021

k8s-ci-robot commented Feb 26, 2021

vdhanan commented Mar 1, 2021

ayberk commented Mar 1, 2021

vdhanan commented Mar 1, 2021

ayberk commented Mar 1, 2021

k8s-ci-robot commented Mar 1, 2021

ayberk commented Mar 1, 2021

vdhanan commented Mar 1, 2021

k8s-ci-robot commented Mar 1, 2021

vdhanan commented Mar 2, 2021

k8s-ci-robot commented Mar 2, 2021

ayberk commented Mar 2, 2021

k8s-ci-robot commented Mar 2, 2021

Add readiness probe so controller does not report "Ready" prematurely #751

Add readiness probe so controller does not report "Ready" prematurely #751

Conversation

vdhanan commented Feb 18, 2021 • edited Loading

k8s-ci-robot commented Feb 18, 2021

k8s-ci-robot commented Feb 18, 2021

k8s-ci-robot commented Feb 18, 2021

vdhanan commented Feb 18, 2021

ayberk commented Feb 18, 2021

coveralls commented Feb 18, 2021 • edited Loading

Pull Request Test Coverage Report for Build 1662

💛 - Coveralls

vdhanan commented Feb 18, 2021

ayberk commented Feb 19, 2021

AndyXiangLi Feb 19, 2021 • edited Loading

Choose a reason for hiding this comment

vdhanan Feb 23, 2021

Choose a reason for hiding this comment

vdhanan commented Feb 23, 2021

ayberk Feb 24, 2021

Choose a reason for hiding this comment

vdhanan Feb 24, 2021

Choose a reason for hiding this comment

k8s-ci-robot commented Feb 26, 2021

vdhanan commented Mar 1, 2021

ayberk commented Mar 1, 2021

vdhanan commented Mar 1, 2021

ayberk commented Mar 1, 2021

k8s-ci-robot commented Mar 1, 2021

ayberk commented Mar 1, 2021

vdhanan commented Mar 1, 2021

k8s-ci-robot commented Mar 1, 2021

vdhanan commented Mar 2, 2021

k8s-ci-robot commented Mar 2, 2021

ayberk commented Mar 2, 2021

k8s-ci-robot commented Mar 2, 2021

vdhanan commented Feb 18, 2021 •

edited

Loading

coveralls commented Feb 18, 2021 •

edited

Loading

AndyXiangLi Feb 19, 2021 •

edited

Loading