Auto osd cache drop

[bengland2]

Description

Depends on benchmark-wrapper PR 269.

This adds support for ceph OSD cache dropping, without each workload having to specify a pod IP.
At present, it cannot automatically start the Ceph toolbox or the cache dropping pod because of a need for authorization
from openshift-storage/rook-ceph namespaces, but the user can start these pods and leave them running and the rest of it works.

for openshift, ceph toolbox can be started with

$ oc patch OCSInitialization ocsinit -n openshift-storage --type json --patch  '[{ "op": "replace", "path": "/spec/enableCephTools", "value": true }]'

for the cache drop pod, you can start it by filling in the vars in roles/ceph_osd_cache_drop/rook_ceph_drop_cache_pod.yaml
and running the pod with kubectl/oc.

Fixes

lack of ceph cache dropping makes benchmark-operator incomplete for OCS.

[bengland2]

for the reviewers, @chaitanyaenr and @dry923 , easiest way to review this is by looking at the "files" tab instead of each commit.
I'll squash the commits at merge time. the problem I'm having with this right now is the startup of two pods:

• Ceph toolbox pod, which is an OCS pod that runs the /usr/bin/ceph CLI - I attempt to start with ansible k8s equivalent of

oc patch OCSInitialization ocsinit -n openshift-storage --type json --patch  '[{ "op": "replace", "path": "/spec/enableCephTools", "
value": true }]'

• Ceph cache dropper pod, which is a cherrypy pod that executes "ceph tell osd.* cache drop" whenever a client does a HTTP GET on /DropOSDCache URL.

The problem with both of these is that I get HTTP 403 error code from the K8S API when I attempt to start them in roles/ceph_osd_cache_drop/tasks/main.yml . I know it's just privs because the same pod startups work with $KUBECONFIG (admin) as my authorization. And when I start the pods by hand using "oc" then the code all works.

I've tried to start the cache dropper pod in my-ripsaw but then it can't get access to the secrets that are only available within the openshift-storage namespace. But I don't have privs to start a pod within openshift-storage namespace.

[jtaleric]

/rerun all

[jtaleric]

We good to remove this?

[bengland2]

@jtaleric I want to be able to remove the rook_ceph_namespace bit, but I don't know how to get authentication and authorization to get the secrets yet, still working that out. Same with the oc patch OCSInitialization task in the same file.

[comet-perf-ci]

Results for ec2_jjb

Test	Result	Retries	Duration (HH:MM:SS)
test_backpack	Pass	0	00:03:52
test_byowl	Pass	0	00:00:12
test_fiod	Pass	1	00:34:29
test_flent	Pass	0	00:06:54
test_fs_drift	Pass	0	00:04:33
test_hammerdb	Pass	0	00:03:57
test_iperf3	Pass	0	00:05:46
test_kubeburner	Pass	0	00:22:50
test_log_generator	Pass	0	00:02:53
test_pgbench	Pass	0	00:02:20
test_smallfile	Pass	0	00:04:17
test_stressng	Pass	0	00:01:31
test_sysbench	Pass	0	00:01:04
test_uperf	Pass	1	00:41:29
test_vegeta	Pass	0	00:04:08
test_ycsb	Pass	0	00:02:53
test_scale_openshift	Pass	0	00:08:09

[bengland2]

I'd merge it now but this depends on bohica quay.io/cloud-bulldozer/ceph-cache-dropper being updated and that hasn't rebuilt. How to trigger rebuild? Also I'd like to figure out how to be able to put the ceph_cache_dropper pod in my-ripsaw namespace instead of openshift-storage/rook-ceph namespace.

[stale]

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[bengland2]

I think I'm going to have to change this so that the user is responsible for actually starting the cache dropper pod and the ceph toolbox pod, but the rest of it can stay the same. This is because the benchmark-operator doesn't have authorization to start pods in the openshift-storage namespace. Will try to get to this next week and finish up.

[jtaleric]

@bengland2 ack - so are we good with your current implementation? LGTM..

[bengland2]

@jtaleric current implementation will not work for reasons discussed above, but if I make the user create the ceph toolbox pod and the cache dropper pod using a provided YAML file, then the authorization problem goes away and the rest of it should work fine. I still think there is a way to get some sort of token and do it but for now that's beyond what I know how to do. At least the CR will not need to change to contain the IP addr of the cache dropper pod - benchmark-operator will discover the pod and make use of it automatically if the user specifies that Ceph OSD cache dropping is desired. This change is small and I should be able to test it this week and merge it. Sorry for the delays, the Multus Alias allocation required me to postpone work on this. Thanks for the other merge.

[comet-perf-ci]

Results for ec2_jjb

Test	Result	Retries	Duration (HH:MM:SS)
test_backpack	Pass	0	00:05:06
test_byowl	Pass	0	00:00:11
test_fiod	Pass	1	00:35:17
test_flent	Pass	0	00:08:59
test_fs_drift	Pass	0	00:04:41
test_hammerdb	Pass	0	00:03:54
test_iperf3	Pass	0	00:06:08
test_kubeburner	Pass	0	00:23:29
test_log_generator	Pass	0	00:04:16
test_pgbench	Pass	0	00:02:42
test_smallfile	Pass	0	00:04:48
test_stressng	Pass	0	00:01:48
test_sysbench	Pass	0	00:01:06
test_uperf	Pass	0	00:26:19
test_vegeta	Pass	0	00:04:24
test_ycsb	Pass	0	00:02:48
test_scale_openshift	Pass	0	00:09:00

[comet-perf-ci]

Results for ec2_jjb

Test	Result	Retries	Duration (HH:MM:SS)
test_backpack	Pass	0	00:03:53
test_byowl	Pass	0	00:00:16
test_fiod	Pass	1	00:36:34
test_flent	Pass	0	00:06:53
test_fs_drift	Pass	0	00:04:13
test_hammerdb	Pass	0	00:03:37
test_iperf3	Pass	0	00:06:02
test_kubeburner	Pass	0	00:23:42
test_log_generator	Pass	0	00:04:00
test_pgbench	Pass	0	00:02:56
test_smallfile	Pass	0	00:04:52
test_stressng	Pass	0	00:02:05
test_sysbench	Pass	0	00:00:59
test_uperf	Pass	0	00:24:46
test_vegeta	Pass	0	00:03:45
test_ycsb	Pass	0	00:01:37
test_scale_openshift	Pass	0	00:03:54

[bengland2]

I'm merging because the test_fiod.sh script passed when run against an AWS cluster, and the test result in terms of passes and fails is identical to what it was before I included OSD cache dropping tests in the PR. Specifically there was an fio failure back 28 days ago when none of the test_crs used OSD cache dropping. So none of the OSD cache dropping code actually executed, and fio test passed 2nd time. This makes me think there was a timeout due to slow image load instead of actual problem with the code. I would like to see the CI save the logs for each test run so that it would be possible to further diagnose things instead of guessing.