backupPVC readOnly configuration doesn't work for SELinux

[Lyndon-Li]

See discussions in #8243, for data mover backup, if backupPVC readOnly is configured (design #7982, implementation #8109), VGDP always fails because of permission denied error in SELinux env.

[Lyndon-Li]

@sseago @shubham-pampattiwar
We discussed this issue locally, we suggest to do more tests:
We need to test with ceph storage to see whether the snapshot clone is compromised once we remove readOnly flag from volumeSource.

If snapshot full clone is not happening, it means backupPVC readOnly configuration is still working at least for ceph. Then we can consider to remove the readOnly flag from volumeSource.
Meanwhile, we need to document that supporting readOnly mount + selinux is a limitation of Kubernetes, there may be problems to use the backupPVC readOnly configuration feature in some envs.

cc @reasonerjt

[sseago]

@Lyndon-Li another option is to make this configurable -- something like this:

--datamover-selinux:
valid values:
- "none": default option, only appropriate for non-selinux environments
- "no-relabeling": preferred configuration for selinux, but won't work with Restricted pods
- "no-readonly": for selinux environments where spc_t privileges are not allowed, but may break shallow copy

This would allow anyone using selinux without Restricted pods to use the spc_t solution, and those with Restricted pods would remove readonly.

[sseago]

@Lyndon-Li here's the proposed configurable fix: #8255

This has a few advantages over just doing the "always remove readonly" one:

1. in environments where Velero is not running in Restricted pods, skipping the relabel has an actual performance gain for volumes with large numbers of files. I believe @msfrucht has seen performance impact from this in the past.
2. Even if we can verify that removing ReadOnly doesn't break ceph shallow copy (we intend to test this but have not done so yet), it's possible that there are other volume types where removing ReadOnly similarly hurts performance. This gives users two different options to deal with selinux.
3. For non-SELinux envionments, no changes are made to the pod configuration.

[msfrucht]

@sseago Performance improvements to overall backup time of the affected namespaces, not to data transfer.

SELinux rebaleling is a non-transparent action in Kubernetes during mount before the container runs.

The application is not typically aware of SELinux relbaling in progress. Most only notice when the Kubernetes scheduler starts throwing up Events about deadlines exceeded.

Kubernetes is in the process of converting to the far more efficient volume relabeling in 1.31-1.32 timeframe and went to beta at 1.27.

[kaovilai]

Kubernetes is in the process of converting to the far more efficient volume relabeling in 1.31-1.32 timeframe and went to beta at 1.27.

kubernetes/enhancements#1710