Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[cinder-csi-plugin] Verify filesystem size after resize #1543

Closed
wants to merge 1 commit into from

Conversation

zuzzas
Copy link
Contributor

@zuzzas zuzzas commented May 26, 2021

What this PR does / why we need it:

Here is an issue that triggered a waterfall of "successful" Expands, which led to improper filesystem size: https://bugs.launchpad.net/openstack-ansible/+bug/1902914. Rescan does not work.

This means that in some cases, we can't trust a positive response from an API, and we should check an actual filesystem size after Expanding it.

Release note:

[cinder-csi-plugin] Verify filesystem size after resize

Signed-off-by: Andrey Klimentyev <andrey.klimentyev@flant.com>
@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels May 26, 2021
@k8s-ci-robot
Copy link
Contributor

Hi @zuzzas. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 26, 2021
@k8s-ci-robot k8s-ci-robot requested a review from ricolin May 26, 2021 06:17
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign zetaab after the PR has been reviewed.
You can assign the PR to them by writing /assign @zetaab in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot requested a review from zetaab May 26, 2021 06:17
@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label May 26, 2021
@theopenlab-ci
Copy link

theopenlab-ci bot commented May 26, 2021

Build succeeded.

@theopenlab-ci
Copy link

theopenlab-ci bot commented May 26, 2021

Build failed.

@theopenlab-ci
Copy link

theopenlab-ci bot commented May 26, 2021

Build succeeded.

@theopenlab-ci
Copy link

theopenlab-ci bot commented May 26, 2021

Build failed.

@zuzzas zuzzas marked this pull request as draft May 26, 2021 08:16
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 26, 2021
@zuzzas
Copy link
Contributor Author

zuzzas commented May 26, 2021

Nope, that won't work. Statfs() calculates available size without filesystem-specific.

There are some insights here. Working on figuring this out...

@theopenlab-ci
Copy link

theopenlab-ci bot commented May 26, 2021

Build failed.

@theopenlab-ci
Copy link

theopenlab-ci bot commented May 26, 2021

Build succeeded.

@theopenlab-ci
Copy link

theopenlab-ci bot commented May 26, 2021

Build succeeded.

@theopenlab-ci
Copy link

theopenlab-ci bot commented May 26, 2021

Build failed.

@@ -540,6 +541,17 @@ func (ns *nodeServer) NodeExpandVolume(ctx context.Context, req *csi.NodeExpandV
if _, err := r.Resize(devicePath, volumePath); err != nil {
return nil, status.Errorf(codes.Internal, "Could not resize volume %q: %v", volumeID, err)
}

newFsSize, err := filesystem.GetFilesystemSize(volumePath)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may want to use r.NeedResize() instead:

https://github.com/kubernetes/mount-utils/blob/master/resizefs_linux.go#L105

mount-utils is added in this PR: #1440, but it's interesting it doesn't exist in this file 🤔

It can also be used for #1539 as well :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought I've sent it as a comment, not a review! :D

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aye, but the problem is. Due to the bug that I linked, the block device won't be resized as well. I think I'll add a check for the block device size; let's see how it works out. And if I can persuade maintainers to merge this.

@jichenjc
Copy link
Contributor

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 27, 2021
@k8s-ci-robot
Copy link
Contributor

@zuzzas: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
pull-cloud-provider-openstack-test 60d54f3 link /test pull-cloud-provider-openstack-test

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@zuzzas
Copy link
Contributor Author

zuzzas commented May 27, 2021

[BlockStorage]
rescan-on-resize = true

This config triggers a code path on NodeVolumeExpand() that works around the aforementioned bug. This PR has no purpose.

@zuzzas zuzzas closed this May 27, 2021
@zuzzas zuzzas deleted the resize-check branch May 27, 2021 11:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants