Back up VolumeSnapshotContents into backup tarball

[nrb]

The current CSI snapshotter plugin will create a VolumeSnapshot for a given PVC, however the VolumeSnapshot and associated VolumeSnapshotContents (and possibly the VolumeSnapshotClass) will need to be backed up into the tarball in order to actually restore the volume.

VolumeSnapshotContents are a cluster-scoped resource, so the logic here should be similar to walking from PVCs to PVs, and may be a separate plugin.

[skriss]

Is this only necessary if you're restoring into a fresh cluster (either all your objs got deleted or it's a different cluster entirely)? Or is this needed for the "i deleted my namespace, now i want to restore it" use case too?

If it's only for the fresh cluster use case, we could opt to just not support that for the prototype. We should think about how important it is.

[nrb]

The VolumeSnapshotContents appear to get deleted with a VolumeSnapshot, so deleting the namespace does indeed destroy them.

I’ll dig for the code on this, but that was observed behavior that led to this issue.

[skriss]

Ah, lame. Seems like they weren't really designing with a DR scenario in mind? In any case, if we can come up with a couple additional plugins that handle this, that's 💯for prototype.

[nrb]

See the following session - VolumeSnapshotContents do indeed get deleted with a namespace going away. It seems to be because of the snapshot.storage.kubernetes.io/volumesnapshotcontent-protection finalizer on the VSC and VS. The VSC also has a deletionPolicy: Delete, which may be something we can change, but I'll have to dig around in the code.

I'll do some experimentation, as we'll need to capture the VSC after the VolumeSnapshot is done. I may just do a wait for it in the snapshotter plugin for the prototype for now.

x1c in /home/nrb/go/src/github.com/heptio/velero-csi-plug
% k get volumesnapshots -n demo
No resources found.

x1c in /home/nrb/go/src/github.com/heptio/velero-csi-plug
% k get volumesnapshotcontents
NAME                                               AGE
snapcontent-2cd2fb15-87dc-11e9-bb60-02c4fcac63ba   6d20h
snapcontent-a2f19630-87d7-11e9-bb60-02c4fcac63ba   6d20h

x1c in /home/nrb/go/src/github.com/heptio/velero-csi-plugin (git) master U
% v backup create demo --include-namespaces demo
Backup request "demo" submitted successfully.
Run `velero backup describe demo` or `velero backup logs demo` for more details.

x1c in /home/nrb/go/src/github.com/heptio/velero-csi-plugin (git) master U
% k get volumesnapshots -n demo
NAME                     AGE
velero-ebs-claim-clgl8   3s

x1c in /home/nrb/go/src/github.com/heptio/velero-csi-plugin (git) master U
% k get volumesnapshotcontents
NAME                                               AGE
snapcontent-1d68a1a3-8d3c-11e9-bb60-02c4fcac63ba   10s
snapcontent-2cd2fb15-87dc-11e9-bb60-02c4fcac63ba   6d20h
snapcontent-a2f19630-87d7-11e9-bb60-02c4fcac63ba   6d20h

x1c in /home/nrb/go/src/github.com/heptio/velero-csi-plugin (git) master U
% k delete ns/demo
namespace "demo" deleted

x1c in /home/nrb/go/src/github.com/heptio/velero-csi-plugin (git) master U
% k get volumesnapshotcontents
NAME                                               AGE
snapcontent-2cd2fb15-87dc-11e9-bb60-02c4fcac63ba   6d20h
snapcontent-a2f19630-87d7-11e9-bb60-02c4fcac63ba   6d20h

[nrb]

Doing some more research, you can set a deletionPolicy:Retain on your VolumeSnapshotClass, which will keep the VolumeSnapshotContents around should a namespace be deleted.

In addition, it appears the the underlying, provider snapshots are retained when deletionPolicy:Retain is set.

This is not the default, so I'll make sure our instructions include it.

[dns2utf8]

Dear all

Are there any news on this issue?
Also, is it related to #2066 ?

Cheers

[skriss]

@dns2utf8 yes, this is related to #2066. We'll be starting to tackle CSI snapshot integration holistically in v1.3, i.e. over the next 2-3 months!