Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail get DataUploadResult for restore - multiple DataUpload result cms found #7057

Open
zanderson-aim opened this issue Nov 5, 2023 · 23 comments

Comments

@zanderson-aim
Copy link

What steps did you take and what happened:
When trying to restore a backup I get the following error:

    rc-aimit-io:  error preparing persistentvolumeclaims/rc-aimit-io/datadir-rocketchat-mongodb-0: rpc error: code = Unknown desc = fail get DataUploadResult for restore: velero-imdc-202311
05205455-20231105210928: multiple DataUpload result cms found with labels velero.io/pvc-namespace-name=rc-aimit-io.datadir-rocketchat-mongodb-0,velero.io/restore-uid=630b736f-e04a-4fb7-b937
-7654feb4062e,velero.io/resource-usage=DataUpload
                  error preparing persistentvolumeclaims/rc-aimit-io/datadir-rocketchat-mongodb-1: rpc error: code = Unknown desc = fail get DataUploadResult for restore: velero-imdc-202311
05205455-20231105210928: multiple DataUpload result cms found with labels velero.io/pvc-namespace-name=rc-aimit-io.datadir-rocketchat-mongodb-1,velero.io/restore-uid=630b736f-e04a-4fb7-b937
-7654feb4062e,velero.io/resource-usage=DataUpload

The restore command
velero restore create --from-backup velero-imdc-20231105205455 --include-namespaces rc-aimit-io --include-resources persistentvolumeclaims,persistentvolumes,dataupload

What did you expect to happen:
The PV/PVCs would be restored.

The following information will help us better understand what's going on:
bundle-2023-11-05-23-09-42.tar.gz

Anything else you would like to add:
I was able to confirm the data is there by manually mounting the bucket using Kopia tools and restoring the files from there.

Environment:

  • Velero version (use velero version):
Client:
        Version: v1.12.1
        Git commit: 5c4fdfe147357ec7b908339f4516cd96d6b97c61
Server:
        Version: v1.12.1
  • Velero features (use velero client config get features):
    features: <NOT SET>
  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.17+rke2r1", GitCommit:"a7736eaf34d823d7652415337ac0ad06db9167fc", GitTreeState:"clean", BuildDate:"2022-12-08T21:42:35Z", GoVersion:"go1.16.15b7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.17+rke2r1", GitCommit:"a7736eaf34d823d7652415337ac0ad06db9167fc", GitTreeState:"clean", BuildDate:"2022-12-08T21:42:35Z", GoVersion:"go1.16.15b7", Compiler:"gc", Platform:"linux/amd64"}
  • Kubernetes installer & version:
  • RKE2
  • v1.22.17+rke2r1
@Lyndon-Li
Copy link
Contributor

@zanderson-aim
For a quick check, could you run below command and share the result?
kubectl get cm -n velero

@blackpiglet
Copy link
Contributor

{"level":"error","logSource":"pkg/controller/restore_controller.go:560","msg":"Namespace rc-aimit-io, resource restore error: error preparing persistentvolumeclaims/rc-aimit-io/datadir-rocketchat-mongodb-0: rpc error: code = Unknown desc = fail get DataUploadResult for restore: velero-imdc-20231105205455-20231105210928: multiple DataUpload result cms found with labels velero.io/pvc-namespace-name=rc-aimit-io.datadir-rocketchat-mongodb-0,velero.io/restore-uid=630b736f-e04a-4fb7-b937-7654feb4062e,velero.io/resource-usage=DataUpload","restore":"velero/velero-imdc-20231105205455-20231105210928","time":"2023-11-05T21:20:28Z"}
{"level":"error","logSource":"pkg/controller/restore_controller.go:560","msg":"Namespace rc-aimit-io, resource restore error: error preparing persistentvolumeclaims/rc-aimit-io/datadir-rocketchat-mongodb-1: rpc error: code = Unknown desc = fail get DataUploadResult for restore: velero-imdc-20231105205455-20231105210928: multiple DataUpload result cms found with labels velero.io/pvc-namespace-name=rc-aimit-io.datadir-rocketchat-mongodb-1,velero.io/restore-uid=630b736f-e04a-4fb7-b937-7654feb4062e,velero.io/resource-usage=DataUpload","restore":"velero/velero-imdc-20231105205455-20231105210928","time":"2023-11-05T21:20:28Z"}

Please also check the PVCs rc-aimit-io/datadir-rocketchat-mongodb-0 and rc-aimit-io/datadir-rocketchat-mongodb-1 are mounted multiple times.

@zanderson-aim
Copy link
Author

@zanderson-aim For a quick check, could you run below command and share the result? kubectl get cm -n velero

This results in about 9000 lines which look like this

velero-imdc-20231031000006-55gfn-j4jk6   1      5d5h
velero-imdc-20231031000006-55gfn-ldm8d   1      5d5h
velero-imdc-20231031000006-55gfn-qf8hn   1      4d4h
velero-imdc-20231031000006-55gfn-qzkb4   1      4d6h
velero-imdc-20231031000006-7n9b9-c5ph9   1      4d4h
velero-imdc-20231031000006-7n9b9-d8w6b   1      5d5h
velero-imdc-20231031000006-7n9b9-j4gpj   1      4d6h
velero-imdc-20231031000006-7n9b9-ns4xs   1      5d5h
velero-imdc-20231031000006-7wg25-bmgxg   1      4d6h
velero-imdc-20231031000006-7wg25-rml9s   1      4d4h
velero-imdc-20231031000006-7wg25-sd85w   1      5d5h
velero-imdc-20231031000006-7wg25-zm8zq   1      5d5h
velero-imdc-20231031000006-92sqt-5dfmz   1      4d4h
velero-imdc-20231031000006-92sqt-7lhk9   1      4d6h
velero-imdc-20231031000006-92sqt-bdr7h   1      5d5h
velero-imdc-20231031000006-92sqt-kmwf9   1      5d5h
velero-imdc-20231031000006-cdqft-5smd7   1      4d4h
velero-imdc-20231031000006-cdqft-hc99l   1      5d5h
velero-imdc-20231031000006-cdqft-m2lbp   1      5d5h
velero-imdc-20231031000006-cdqft-vwmc7   1      4d6h
velero-imdc-20231031000006-cvmt7-8qx8p   1      5d5h
velero-imdc-20231031000006-cvmt7-gpr97   1      4d6h
velero-imdc-20231031000006-cvmt7-ql5zw   1      4d4h
velero-imdc-20231031000006-cvmt7-rlswx   1      5d5h
velero-imdc-20231031000006-d8zbx-5vfnz   1      4d6h
velero-imdc-20231031000006-d8zbx-cl68t   1      4d4h
velero-imdc-20231031000006-d8zbx-jwgwp   1      5d5h
velero-imdc-20231031000006-d8zbx-t9thk   1      5d5h
velero-imdc-20231031000006-dkzqm-4brdn   1      5d5h
velero-imdc-20231031000006-dkzqm-dpls4   1      4d6h
velero-imdc-20231031000006-dkzqm-vsjfd   1      4d4h
velero-imdc-20231031000006-dkzqm-wf2fr   1      5d5h
velero-imdc-20231031000006-dqqrj-pd4bx   1      5d5h
velero-imdc-20231031000006-dqqrj-pf58j   1      4d4h
velero-imdc-20231031000006-dqqrj-spw8q   1      4d6h

@zanderson-aim
Copy link
Author

{"level":"error","logSource":"pkg/controller/restore_controller.go:560","msg":"Namespace rc-aimit-io, resource restore error: error preparing persistentvolumeclaims/rc-aimit-io/datadir-rocketchat-mongodb-0: rpc error: code = Unknown desc = fail get DataUploadResult for restore: velero-imdc-20231105205455-20231105210928: multiple DataUpload result cms found with labels velero.io/pvc-namespace-name=rc-aimit-io.datadir-rocketchat-mongodb-0,velero.io/restore-uid=630b736f-e04a-4fb7-b937-7654feb4062e,velero.io/resource-usage=DataUpload","restore":"velero/velero-imdc-20231105205455-20231105210928","time":"2023-11-05T21:20:28Z"}
{"level":"error","logSource":"pkg/controller/restore_controller.go:560","msg":"Namespace rc-aimit-io, resource restore error: error preparing persistentvolumeclaims/rc-aimit-io/datadir-rocketchat-mongodb-1: rpc error: code = Unknown desc = fail get DataUploadResult for restore: velero-imdc-20231105205455-20231105210928: multiple DataUpload result cms found with labels velero.io/pvc-namespace-name=rc-aimit-io.datadir-rocketchat-mongodb-1,velero.io/restore-uid=630b736f-e04a-4fb7-b937-7654feb4062e,velero.io/resource-usage=DataUpload","restore":"velero/velero-imdc-20231105205455-20231105210928","time":"2023-11-05T21:20:28Z"}

Please also check the PVCs rc-aimit-io/datadir-rocketchat-mongodb-0 and rc-aimit-io/datadir-rocketchat-mongodb-1 are mounted multiple times.

I deleted them from the live system to test the restore, so they don't currently exist any more.

@Lyndon-Li
Copy link
Contributor

@zanderson-aim Are there running data mover restores at the time you run kubectl get cm -n velero?

@zanderson-aim
Copy link
Author

@zanderson-aim Are there running data mover restores at the time you run kubectl get cm -n velero?

No currently nothing running

kubectl -n velero get pods
NAME                      READY   STATUS    RESTARTS      AGE
node-agent-24ct4          1/1     Running   0             11d
node-agent-6ckx4          1/1     Running   0             11d
node-agent-8f559          1/1     Running   0             11d
node-agent-b2kpg          1/1     Running   0             11d
node-agent-pzbnq          1/1     Running   0             11d
node-agent-qzlsh          1/1     Running   0             11d
velero-6798d896db-5hpnk   1/1     Running   1 (40h ago)   11d
# kubectl get cm -n velero | wc -l
9756

@Lyndon-Li
Copy link
Contributor

Lyndon-Li commented Nov 6, 2023

@zanderson-aim
Could you share the content of one of these configMaps? E.g., kubectl get cm -n velero -o yaml velero-imdc-20231031000006-dqqrj-spw8q
Please also let us know the timestamp of the latest one of these configMaps? You can also run kubectl get cm -n velero -o yaml for the latest configMap

@Lyndon-Li
Copy link
Contributor

@zanderson-aim
We want to do several checks on the list of configMaps as you shared, different info are required. So how about this:

  1. Run kubectl get cm -n velero and share us the entire output
  2. Run kubectl get cm -n velero -o yaml and share us the entire output

2 may be very large, not sure if you can make it, but let's see it.

@zanderson-aim
Copy link
Author

zanderson-aim commented Nov 6, 2023

@zanderson-aim We want to do several checks on the list of configMaps as you shared, different info are required. So how about this:

  1. Run kubectl get cm -n velero and share us the entire output
  2. Run kubectl get cm -n velero -o yaml and share us the entire output

2 may be very large, not sure if you can make it, but let's see it.

kubectl get cm -n velero > /tmp/velero-cm-list.txt
velero-cm-list.txt

kubectl get cm -n velero -o yaml > /tmp/velero-cm.yaml
velero-cm.zip

@Lyndon-Li
Copy link
Contributor

Lyndon-Li commented Nov 6, 2023

@zanderson-aim
Looks like you have included velero namespace in the backup as what we can see in the backup describe:

velero.io/v1/Backup:
    - velero/velero-imdc-20231007180046
    - velero/velero-imdc-20231008180047
    - velero/velero-imdc-20231013060021
    - velero/velero-imdc-20231013180021
    - velero/velero-imdc-20231014120022
    - velero/velero-imdc-20231014180023
    - velero/velero-imdc-20231015000059
    - velero/velero-imdc-20231015060059
    - velero/velero-imdc-20231015120059
    - velero/velero-imdc-20231015180000
    - velero/velero-imdc-20231016000000
    - velero/velero-imdc-20231016060000
    - velero/velero-imdc-20231016120001
    - velero/velero-imdc-20231016180001
    - velero/velero-imdc-20231017000001
    - velero/velero-imdc-20231017120047
    - velero/velero-imdc-20231017180047
    - velero/velero-imdc-20231018000047
    - velero/velero-imdc-20231018060048
    - velero/velero-imdc-20231019000049
    - velero/velero-imdc-20231019060049
    - velero/velero-imdc-20231019120049
    - velero/velero-imdc-20231020000032
    - velero/velero-imdc-20231020060038
    - velero/velero-imdc-20231020120038
    - velero/velero-imdc-20231020180039
    - velero/velero-imdc-20231021000039
    - velero/velero-imdc-20231021060013
    - velero/velero-imdc-20231021120013
    - velero/velero-imdc-20231021180011
    - velero/velero-imdc-20231022000012
    - velero/velero-imdc-20231022060012
    - velero/velero-imdc-20231022120053
    - velero/velero-imdc-20231022180054
    - velero/velero-imdc-20231023000054
    - velero/velero-imdc-20231023060054
    - velero/velero-imdc-20231023120057
    - velero/velero-imdc-20231023180057
    - velero/velero-imdc-20231024000057
    - velero/velero-imdc-20231024060058
    - velero/velero-imdc-20231024120058
    - velero/velero-imdc-20231024180058
    - velero/velero-imdc-20231025000059
    - velero/velero-imdc-20231025060059
    - velero/velero-imdc-20231025120049
    - velero/velero-imdc-20231025180049
    - velero/velero-imdc-20231026000059
    - velero/velero-imdc-20231026060059
    - velero/velero-imdc-20231026120000
    - velero/velero-imdc-20231026180000
    - velero/velero-imdc-20231027000001
    - velero/velero-imdc-20231027060001
    - velero/velero-imdc-20231027120001
    - velero/velero-imdc-20231027180001
    - velero/velero-imdc-20231028000002
    - velero/velero-imdc-20231028060002
    - velero/velero-imdc-20231028120002
    - velero/velero-imdc-20231028180003
    - velero/velero-imdc-20231029000003
    - velero/velero-imdc-20231029060003
    - velero/velero-imdc-20231029120004
    - velero/velero-imdc-20231029180004
    - velero/velero-imdc-20231030000004
    - velero/velero-imdc-20231030060005
    - velero/velero-imdc-20231030120005
    - velero/velero-imdc-20231030180005
    - velero/velero-imdc-20231030183306
    - velero/velero-imdc-20231031000006
    - velero/velero-imdc-20231031060006
    - velero/velero-imdc-20231031120006
    - velero/velero-imdc-20231031180007
    - velero/velero-imdc-20231101000007
    - velero/velero-imdc-20231101060008
    - velero/velero-imdc-20231101120008
    - velero/velero-imdc-20231101180008
    - velero/velero-imdc-20231101201654
    - velero/velero-imdc-20231102000009
    - velero/velero-imdc-20231102060009
    - velero/velero-imdc-20231102120009
    - velero/velero-imdc-20231102180010
    - velero/velero-imdc-20231103000010
    - velero/velero-imdc-20231103060011
    - velero/velero-imdc-20231103120011
    - velero/velero-imdc-20231103180011
    - velero/velero-imdc-20231104000012
    - velero/velero-imdc-20231104060012
    - velero/velero-imdc-20231104120045
    - velero/velero-imdc-20231104180045
    - velero/velero-imdc-20231105000046
    - velero/velero-imdc-20231105060046
    - velero/velero-imdc-20231105120046
    - velero/velero-imdc-20231105180046
    - velero/velero-imdc-20231105205455
  velero.io/v1/BackupRepository:
    - velero/4luvofthegame-com-backblaze-imdc-velero-kopia-lgzfh
    - velero/4x5program-org-backblaze-imdc-velero-kopia-xgzg5
    - velero/alumcoprecision-com-backblaze-imdc-velero-kopia-8s4wv
    - velero/argocd-backblaze-imdc-velero-kopia-lmbz8
    - velero/blingtour-com-backblaze-imdc-velero-kopia-n9s8j
    - velero/budibase-backblaze-imdc-velero-kopia-wnccx
    - velero/camplulu-com-backblaze-imdc-velero-kopia-zt6b5
    - velero/ceph-csi-backblaze-imdc-velero-kopia-675cp
    - velero/drillrdd-com-backblaze-imdc-velero-kopia-5j7kz
    - velero/fleet-aimit-io-backblaze-imdc-velero-kopia-kht45
    - velero/grafana-agent-backblaze-imdc-velero-kopia-kvmx2
    - velero/hs-aimit-io-backblaze-imdc-velero-kopia-576hs
    - velero/ingress-contour-backblaze-imdc-velero-kopia-v85kn
    - velero/inventory-aimit-io-backblaze-imdc-velero-kopia-2tl79
    - velero/kube-system-backblaze-imdc-velero-kopia-xc97l
    - velero/pm-aimit-io-backblaze-imdc-velero-kopia-dk9q8
    - velero/raisingthebar-org-backblaze-imdc-velero-kopia-4kfpb
    - velero/rc-aimit-io-backblaze-imdc-velero-kopia-gpvx9
    - velero/test-velero-backblaze-imdc-velero-kopia-8p58x
    - velero/velero-backblaze-imdc-velero-kopia-gsnhb
    - velero/velero-test-backblaze-imdc-velero-kopia-d5z69
  velero.io/v1/BackupStorageLocation:
    - velero/backblaze-imdc-velero
  velero.io/v1/Restore:
    - velero/velero-imdc-20231030183306-20231030204827
    - velero/velero-imdc-20231030183306-20231031223223
    - velero/velero-imdc-20231030183306-20231031225141
    - velero/velero-imdc-20231030183306-20231031225242
    - velero/velero-imdc-20231030183306-20231031231546
    - velero/velero-imdc-20231030183306-20231031231633
    - velero/velero-imdc-20231030183306-20231031231704
    - velero/velero-imdc-20231030183306-20231101025845
    - velero/velero-imdc-20231030183306-20231101032154
    - velero/velero-imdc-20231031120006-20231031221148
    - velero/velero-imdc-20231031180007-20231031214800
    - velero/velero-imdc-20231101201654-20231101205439
    - velero/velero-imdc-20231101201654-20231101225032
  velero.io/v1/Schedule:
    - velero/velero-imdc
  velero.io/v2alpha1/DataDownload:
    - velero/velero-imdc-20231030183306-20231030204827-hcrbx
    - velero/velero-imdc-20231031180007-20231031214800-cssf6
    - velero/velero-imdc-20231101201654-20231101205439-hvhjs
    - velero/velero-imdc-20231101201654-20231101205439-m7fpx
    - velero/velero-imdc-20231101201654-20231101205439-tvrnb
  velero.io/v2alpha1/DataUpload:
    - velero/velero-imdc-20231014120022-22nb7
    - velero/velero-imdc-20231014120022-26cqh
    - velero/velero-imdc-20231014120022-2jj5l
    - velero/velero-imdc-20231014120022-2wtwh
    - velero/velero-imdc-20231014120022-2zrlt
    - velero/velero-imdc-20231014120022-428xg
    - velero/velero-imdc-20231014120022-5wztf
    - velero/velero-imdc-20231014120022-62sb9
    - velero/velero-imdc-20231014120022-77k7p
    - velero/velero-imdc-20231014120022-7fzjj
    - velero/velero-imdc-20231014120022-8mqzs
    - velero/velero-imdc-20231014120022-9jdtp
    - velero/velero-imdc-20231014120022-b8gm4
    - velero/velero-imdc-20231014120022-bcsvm
    - velero/velero-imdc-20231014120022-d275t
    - velero/velero-imdc-20231014120022-d5dv7
    - velero/velero-imdc-20231014120022-d8xc7
    - velero/velero-imdc-20231014120022-df8h8
    - velero/velero-imdc-20231014120022-drxv4
    - velero/velero-imdc-20231014120022-dtj4z
    - velero/velero-imdc-20231014120022-dwhv9
    - velero/velero-imdc-20231014120022-hfmlh
    - velero/velero-imdc-20231014120022-hj9b8
    - velero/velero-imdc-20231014120022-jgdj5
    - velero/velero-imdc-20231014120022-k57c9
    - velero/velero-imdc-20231014120022-kg59k
    - velero/velero-imdc-20231014120022-lgcxg
    - velero/velero-imdc-20231014120022-ls8s5
    - velero/velero-imdc-20231014120022-m4wmc
    - velero/velero-imdc-20231014120022-mrksz
    - velero/velero-imdc-20231014120022-n6nbs
    - velero/velero-imdc-20231014120022-p6hmd
    - velero/velero-imdc-20231014120022-pcbpj
    - velero/velero-imdc-20231014120022-qkxpf

Most resources in the velero namespace cannot be backed up, if you want to backup the entire cluster, please exclude velero namespace.

@zanderson-aim
Copy link
Author

Makes sense, I will update the job and test again. I guess these makes all the "old" backups that include velero already unrestorable?

@Lyndon-Li
Copy link
Contributor

I guess these makes all the "old" backups that include velero already unrestorable?

yes

@shawn-hurley
Copy link
Contributor

If this is true, can we change the default to permanently exclude the velero namespace, or even better, maybe, always exclude velero resources by default?

@zanderson-aim
Copy link
Author

I can confirm it worked after I excluded the namespace.

@zanderson-aim
Copy link
Author

I did tried some older backups before 1.12 and they worked fine with the namespace included. Must be something with changes in 1.12.

@Lyndon-Li Lyndon-Li self-assigned this Nov 9, 2023
@Lyndon-Li
Copy link
Contributor

I did tried some older backups before 1.12 and they worked fine with the namespace included. Must be something with changes in 1.12.

OK, I will keep this issue open and see if we can make some improvement around DataUpload CRs.
However, we will not take this as high priority since we don't see too much value to include velero namespace into the backup, and even it is required, the DataUpload CRs could be filtered out as a workaround.

@shawn-hurley
Copy link
Contributor

@shubham-pampattiwar @sseago We probably want to make sure that this is documented in our downstream docs

@oboudry-mvp
Copy link

For info I had exactly the same issue with v.12.1, restore of snapshots (Kopia to Minio) were failing when velero was included in the backup. I excluded all kube- namespaces but left velero in.

New backups without velero namespace restore fine.

Copy link

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days. If a Velero team member has requested log or more information, please provide the output of the shared commands.

Copy link

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days. If a Velero team member has requested log or more information, please provide the output of the shared commands.

@filipe-silva-magalhaes-alb

I have the same issue.

I solved this by cleaning the s3 bucket, installing the Velero doing the backup again, and restoring.

Kopia might get confused when the bucket has multiple backups.

Copy link

github-actions bot commented Aug 4, 2024

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days. If a Velero team member has requested log or more information, please provide the output of the shared commands.

@github-actions github-actions bot added the staled label Aug 4, 2024
@Lyndon-Li Lyndon-Li removed the staled label Aug 5, 2024
@dharanui
Copy link

dharanui commented Oct 8, 2024

Facing same issue when trying to restore with velero version 1.14

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

7 participants