You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
To Reproduce
I am not sure why the volume became unmounted on our system.
But to simulate the scenario, manually unmount the volume path.
ie. umount /var/lib/kubelet/pods/97a388a2-3dd4-4ca0-9694-72683feeea4d/volumes/kubernetes.io~csi/pvc-80a43fd6-9066-4b98-8a86-5de82fe2cf1b/mount
Then delete the pod without using force. (kubectl delete pod -n )
Expected behavior
Pod can be terminated successfully even if the volume is not mounted.
Describe the bug
Pod stuck in "Terminating" status forever due to the following error "volume not mounted" when unmounting volume.
2021-05-03T03:09:35.742478381Z stderr F time="2021-01-13T03:09:35Z" level=error msg="GRPC error: rpc error: code = NotFound desc = volume not mounted"
2021-05-03T03:09:36.325 compute-0 kubelet[24343]: info E0113 03:09:35.796487 24343 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi.trident.netapp.io^pvc-80a43fd6-9066-4b98-8a86-5de82fe2cf1b podName:97a388a2-3dd4-4ca0-9694-72683feeea4d nodeName:}" failed. No retries permitted until 2021-01-13 03:09:36.296442309 +0000 UTC m=+2077.724479529 (durationBeforeRetry 500ms). Error: "UnmountVolume.TearDown failed for volume "mon-elasticsearch-master" (UniqueName: "kubernetes.io/csi/csi.trident.netapp.io^pvc-80a43fd6-9066-4b98-8a86-5de82fe2cf1b") pod "97a388a2-3dd4-4ca0-9694-72683feeea4d" (UID: "97a388a2-3dd4-4ca0-9694-72683feeea4d") : kubernetes.io/csi: mounter.TearDownAt failed: rpc error: code = NotFound desc = volume not mounted"
Environment
Provide accurate information about the environment to help us reproduce the issue.
To Reproduce
I am not sure why the volume became unmounted on our system.
But to simulate the scenario, manually unmount the volume path.
ie. umount /var/lib/kubelet/pods/97a388a2-3dd4-4ca0-9694-72683feeea4d/volumes/kubernetes.io~csi/pvc-80a43fd6-9066-4b98-8a86-5de82fe2cf1b/mount
Then delete the pod without using force. (kubectl delete pod -n )
Expected behavior
Pod can be terminated successfully even if the volume is not mounted.
Additional context
Trident returns error if volume path is not mounted, see https://github.com/NetApp/trident/blob/master/frontend/csi/node_server.go#L171-L173
It causes volume teardown failed, see https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/csi/csi_mounter.go#L364-#L366
Trident should handle the case that the volume is not mounted and allow k8s to complete the teardown, do the same thing as for rbd (delete the volume path if it's not a mount point), see
https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/rbd/disk_manager.go#L112-#L115
The text was updated successfully, but these errors were encountered: