Unable to attach volumes to pod, Error failed to attach: node "xxxxxx-xxx4139" has no NodeID annotation

[san360]

Errors from log files
K8's version 1.13.12
Trident 19.10 and installed with csi flag

trident-csi-c98574d6f-2r8j8 csi-attacher I1114 23:23:29.480251       1 csi_handler.go:120] Attaching "csi-cf6fab5c7ed93f986c742e83359e3190c78263db1e212b2622ae3ae1ef230a08"
trident-csi-c98574d6f-2r8j8 csi-attacher I1114 23:23:56.295373       1 csi_handler.go:103] Error processing "csi-cf6fab5c7ed93f986c742e83359e3190c78263db1e212b2622ae3ae1ef230a08": failed to attach: node "xxxxxx-xxx4139" has no NodeID annotation
trident-csi-c98574d6f-2r8j8 csi-attacher I1114 23:25:18.215553       1 csi_handler.go:120] Attaching "csi-cf6fab5c7ed93f986c742e83359e3190c78263db1e212b2622ae3ae1ef230a08"
trident-csi-c98574d6f-2r8j8 csi-attacher I1114 23:25:18.240505       1 csi_handler.go:103] Error processing "csi-cf6fab5c7ed93f986c742e83359e3190c78263db1e212b2622ae3ae1ef230a08": failed to attach: node "xxxxxx-xxx4139" has no NodeID annotation

[gnarl]

Please contact NetApp Support to open a support case on this issue. The "has no NodeID annotation" may be a Kubernetes issue but more information is needed.

[scaleoutsean]

Haven't enabled required Feature Gates?

[san360]

All the required feature gates are enabled

On Sun, 17 Nov 2019 at 4:35 AM, scaleoutsean ***@***.***> wrote: Haven't enabled required Feature Gates? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#314?email_source=notifications&email_token=ABSOTUL32C7YIX57ORWQJFTQUC3XPA5CNFSM4JN37TP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEIA34I#issuecomment-554700273>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSOTUNEUH5MNZS5WVTE6QTQUC3XPANCNFSM4JN37TPQ> .

-- Thanks & Regards Sanjay Singh

[san360]

I noticed in our environment that below annotation is missing on nodes
csi.volume.kubernetes.io/nodeid='{"csi.trident.netapp.io":"xxxxx-xxxxx"}'
I added the annotation on one of the node and the error disappeared but another error popped up

Events:
  Type     Reason                  Age                  From                     Message
  ----     ------                  ----                 ----                     -------
  Normal   SuccessfulAttachVolume  4m15s                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-74456e52-0a31-11ea-93b1-00505608c43f"
  Warning  FailedMount             2m12s                kubelet, xxxxx-xxxxxx  Unable to mount volumes for pod "task-pv-pod_default(ac03886a-0a37-11ea-8d01-00505608c441)": timeout expired waiting for volumes to attach or mount for pod "default"/"task-pv-pod". list of unmounted volumes=[task-pv-storage]. list of unattached volumes=[task-pv-storage default-token-jfz8d]
  Warning  FailedMount             22s (x9 over 2m30s)  kubelet, xxxxx-xxxxxx  MountVolume.MountDevice failed for volume "pvc-74456e52-0a31-11ea-93b1-00505608c43f" : rpc error: code = Unimplemented desc = unknown service csi.v0.Node
`
``

[adkerr]

The original error (no nodeID) typically is because the trident node server was not able to start properly. Check the daemonset pod to make sure they all started without error

[san360]

All the pods under trident namespace are working.

NAME                          READY   STATUS    RESTARTS   AGE
trident-csi-58x27             2/2     Running   0          8h
trident-csi-6bnx8             2/2     Running   0          8h
trident-csi-7kqk8             2/2     Running   0          8h
trident-csi-9btdv             2/2     Running   0          8h
trident-csi-c98574d6f-gvpcd   5/5     Running   0          8h
trident-csi-fv7f5             2/2     Running   0          8h
trident-csi-g76z6             2/2     Running   0          8h
trident-csi-v4lt2             2/2     Running   0          8h
trident-csi-xmtx4             2/2     Running   0          8h

[trulleberg]

@san360, did you recieve any feedback via the ticket?

[san360]

So far nothing. Are you facing similar issue as well?

[korenaren]

@san360 - if you have a support case open with NetApp, please send me the case # to martyt@netapp.com. I'll see if I can figure out what's going on. Thanks.

[trulleberg]

@san360 Yes we had the same issue. We had to modify the mounts of the worker nodes (make them shared) and install trident with custom-yaml (adjusted the kubelet_path)

This sorted out our problmen and made trident work as it should.

[san360]

@san360 Yes we had the same issue. We had to modify the mounts of the worker nodes (make them shared) and install trident with custom-yaml (adjusted the kubelet_path)

This sorted out our problmen and made trident work as it should.

Thanks for the update.
In our case the kubelets are running as a docker container. I am not sure if you have the same setup. Could you please describe your steps in detail?

I tried to install trident with custom yaml but it failed to start the daemonset container due to missing triden-csi secret. Does it generate secret yaml for you?

[clintonk]

@san360 Yes, the Trident installer generates the trident-csi secret automatically.

[gnarl]

@san360 We were finally able to reproduce this issue. When using containerized kubelet or a container orchestrator that does not install kubelet in /var/lib/kubelet all references to /var/lib/kubelet in the daemonset YAML need to be updated. Also, the kubelet path needs to be the same on all nodes. Mixing containerized kubelet with non-containerized kubelet, for example, is not supported.

When using containerized kubelet, RancherOS, Rancher, PKS, or any other container orchestrator that changes the path of kubelet Trident needs to be installed with customized YAML. OpenShift is an exception where customized YAML is not needed.

• ./tridentctl install --generate-custom-yaml
• Edit setup/trident-daemonset.yaml
• Update all references in the file of '/var/lib/kubelet' to the correct path
• Install Trident with the '--use-custom-yaml' flag

As of Trident 19.10.0 these are all the references to /var/lib/kubelet that need to be updated.

grep -in "/var/lib/kubelet" *
trident-daemonset.yaml:53:          mountPath: /var/lib/kubelet/plugins
trident-daemonset.yaml:55:          mountPath: /var/lib/kubelet/pods
trident-daemonset.yaml:79:          value: "/var/lib/kubelet/plugins/csi.trident.netapp.io/csi.sock"
trident-daemonset.yaml:100:          path: /var/lib/kubelet/plugins/csi.trident.netapp.io/
trident-daemonset.yaml:104:          path: /var/lib/kubelet/plugins_registry/
trident-daemonset.yaml:108:          path: /var/lib/kubelet/plugins
trident-daemonset.yaml:112:          path: /var/lib/kubelet/pods

We are working on a change to make Trident easier to install when the kubelet path differs from the default. Thanks for your feedback and patience so far. Let us know if this fixes your issue.

[korenaren]

@san360 I believe this is resolved, so closing this issue. Please let me know if that is not the case, and we will reopen.