No es index smf fsd #99

bengland2 · 2019-11-22T15:48:42Z

there is no need to use es_index environment variable in wrapper, since the part of the index name after the "ripsaw" prefix is determined by the wrapper. Also, documentation has been updated to be consistent with ripsaw use of elasticsearch.server and elasticsearch.port in CRs.

bengland2 · 2019-11-22T15:50:20Z

@acalhounRH does this look ok?

clarify what run_snafu.py wrapper developer has to do to post an ES doc remove prefix hyphen from index names in yield statements do not associate uuid, test_user and clustername with elasticsearch in CR fix wrappers that use run_snafu.py to work this way

bengland2 · 2019-11-22T19:30:57Z

this commit has undergone considerable change, let me know if you like it now,tested standalone with fs-drift and smallfile, now testing with ripsaw. Goal is to pass CI tests.

run_snafu.py

README.md

acalhounRH · 2019-11-25T17:48:36Z

Looks good. Okay to Merge.

bengland2 · 2019-12-03T20:45:01Z

Needs "OK to test" label once PR97 merges (i.e. snafu CI is in place).

dry923 · 2019-12-04T20:08:11Z

/rerun all

rht-perf-ci · 2019-12-04T21:11:25Z

Results for SNAFU CI Test

Test	Result	Runtime
cluster_loader	FAIL	00:00:00
fio_wrapper	PASS	00:06:16
fs_drift_wrapper	FAIL	00:06:43
hammerdb	FAIL	00:06:43
iperf	PASS	00:05:31
pgbench-wrapper	PASS	00:06:36
smallfile_wrapper	FAIL	00:05:59
sysbench	PASS	00:05:13
uperf-wrapper	PASS	00:06:21
ycsb-wrapper	PASS	00:05:12

bengland2 · 2019-12-04T21:18:56Z

@dry923 @aakarsh do we have a situation where snafu PR 99 and ripsaw PR 249 depend on each other and can't work unless both are committed? I think so. If so, I'd suggest committing ripsaw PR 249, which only affects smallfile and fs-drift test CRs. Then retest snafu PR 99. OK?

aakarshg · 2019-12-04T21:21:00Z

@bengland2 oh okay so oddly 249 ripsaw PR is failing fs_drift and smallfile o.O have to look into why.

bengland2 · 2019-12-04T21:24:31Z

oops, ripsaw PR 249 affects more than the smallfile+fs-drift test CRs, the thanksgiving break destroyed my memory ;-) So when I tested these two benchmarks successfully in ripsaw, I used both PRs together, not one at a time. @acalhounRH FYI

aakarshg · 2019-12-09T19:07:54Z

/rerun all

rht-perf-ci · 2019-12-09T20:09:49Z

Results for SNAFU CI Test

Test	Result	Runtime
cluster_loader	FAIL	00:00:00
fio_wrapper	PASS	00:06:17
fs_drift_wrapper	PASS	00:07:13
hammerdb	FAIL	00:06:37
iperf	PASS	00:05:43
pgbench-wrapper	PASS	00:06:45
smallfile_wrapper	FAIL	00:06:00
sysbench	PASS	00:05:13
uperf-wrapper	PASS	00:06:41
ycsb-wrapper	PASS	00:05:38

aakarshg · 2019-12-09T20:20:39Z

@bengland2 so the CI is correctly failing on smallfile_wrapper as there seems to be no documents indexed into ripsaw-smallfile-rsptimes, have to check what's up.

bengland2 · 2019-12-12T13:27:04Z

ok, I'll look at it, perhaps last change to ripsaw PR 249 somehow interfered with it.

bengland2 · 2019-12-15T18:36:53Z

@aakarsh once ripsaw PR 261 merges (lengthen smallfile CI test), then this problem will go away and we can finally be done with this, I tested that today.

bengland2 · 2019-12-16T15:52:50Z

ripsaw PR 261 (lengthen smallfile CI test) fixes smallfile problem here.

aakarshg · 2019-12-17T10:55:08Z

merged ripsaw pr 261, will recheck this

aakarshg · 2019-12-17T10:55:15Z

/rerun all

aakarshg · 2019-12-17T10:56:13Z

/rerun all

rht-perf-ci · 2019-12-17T12:02:59Z

Results for SNAFU CI Test

Test	Result	Runtime
cluster_loader	FAIL	00:00:00
fio_wrapper	PASS	00:06:11
fs_drift_wrapper	PASS	00:06:46
hammerdb	FAIL	00:06:35
iperf	PASS	00:05:27
pgbench-wrapper	PASS	00:06:41
smallfile_wrapper	PASS	00:07:55
sysbench	PASS	00:05:10
uperf-wrapper	PASS	00:06:23
ycsb-wrapper	PASS	00:05:15

aakarshg

this is missing fio-analyzed result so this https://github.com/cloud-bulldozer/snafu/blob/master/fio_wrapper/fio_analyzer.py#L135 will also need to change, once done I'll merge it. What's odd is that this should've failed CI with fio, as it'd have gone to a ripsaw-fio--analyzed_result but the CI script doesn't look there.

bengland2 · 2020-01-07T14:50:04Z

this needs to be re-based, and README.md needs to be tweaked. Trying to do that now.

bengland2 · 2020-01-07T15:01:21Z

/rerun minikube_jjb

bengland2 · 2020-01-07T15:30:03Z

@aakarshg I did the code change you requested, you were right, I missed a spot.

rht-perf-ci · 2020-01-07T16:39:08Z

Results for SNAFU CI Test

Test	Result	Runtime
cluster_loader	FAIL	00:00:00
fio_wrapper	PASS	00:07:14
fs_drift_wrapper	PASS	00:07:37
hammerdb	FAIL	00:13:25
iperf	PASS	00:05:19
pgbench-wrapper	PASS	00:06:42
smallfile_wrapper	PASS	00:05:59
sysbench	PASS	00:05:18
uperf-wrapper	PASS	00:06:31
ycsb-wrapper	PASS	00:05:18

aakarshg · 2020-01-07T16:43:25Z

/rerun all

rht-perf-ci · 2020-01-07T17:56:41Z

Results for SNAFU CI Test

Test	Result	Runtime
cluster_loader	FAIL	00:00:00
fio_wrapper	PASS	00:06:27
fs_drift_wrapper	PASS	00:07:09
hammerdb	FAIL	00:13:46
iperf	PASS	00:05:30
pgbench-wrapper	PASS	00:08:21
smallfile_wrapper	PASS	00:06:31
sysbench	PASS	00:05:37
uperf-wrapper	PASS	00:07:02
ycsb-wrapper	PASS	00:06:42

bengland2 · 2020-01-07T18:16:46Z

hammerdb FAIL seems to be caused by it pushing benchmark image to cloud-bulldozer instead of rht_perf_ci. whereas the operator image is pushed to rht_perf_ci. Not a problem with this PR. Same kind of problem we're seeing elsewhere, failures to push image to repo are not detected. Other failure is cluster loader.

bengland2 · 2020-01-07T23:02:56Z

cluster loader failed because there was no cluster_loader/ci_test.sh for the CI to run -- issue 112. pgbench failed because of a random non-reproducible error in uploading pgbench image.

Error: Error copying image to the remote destination: Error trying to reuse blob sha256:b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95 at destination: unexpected http code: 500 (Internal Server Error), URL: https://quay.io/v2/auth?account=bengland2&scope=repository%3Abengland2%2Fpgbench%3Apull%2Cpush&service=quay.io

So I consider this to be a pass. CI reliability issue 111 is where intermittent errors like this should be addressed.

aakarshg

overall looks good (ignoring all the unrelated CI errors ), but is missing the updates to clusterloader specifically https://github.com/cloud-bulldozer/snafu/blob/master/cluster_loader/trigger_cluster_loader.py#L86 which where the index needs to be an empty string as 'snafu-cl' will be added through this pr change. Will merge as soon as its fixed, sorry to have to kept this pr waiting for a while.

bengland2 · 2020-01-09T14:52:04Z

/rerun minikube_jjb

bengland2 · 2020-01-09T15:15:58Z

I made the change you requested and rebased, @aakarshg but it's not running the CI for some reason, can you clear your requested change because I can't.

aakarshg · 2020-01-09T18:28:32Z

/rerun all

aakarshg · 2020-01-09T18:41:51Z

I made the change you requested and rebased, @aakarshg but it's not running the CI for some reason, can you clear your requested change because I can't.

looks like this build has been in queue given that the other prs were triggered before this thats why its taking long.

rht-perf-ci · 2020-01-09T20:21:23Z

Results for SNAFU CI Test

Test	Result	Runtime
fio_wrapper	PASS	00:06:48
fs_drift_wrapper	PASS	00:07:14
hammerdb	FAIL	00:13:41
iperf	PASS	00:05:44
pgbench-wrapper	PASS	00:07:08
smallfile_wrapper	PASS	00:06:24
sysbench	PASS	00:05:30
uperf-wrapper	PASS	00:07:03
ycsb-wrapper	PASS	00:05:44

bengland2 · 2020-01-09T21:47:42Z

hammerdb failed because of:

++ grep 'SEQUENCE COMPLETE'
Error from server (BadRequest): container "hammerdb" in pod "hammerdb-workload-ba4af13d-st9gt" is waiting to start: trying and failing to pull image
++ echo 'Hammerdb test: Success'
Hammerdb test: Success

and then it did not see any update to its ES index. But that doesn't seem to have anything to do with this PR, since hammerdb does not use run_snafu.py. I think the hammerdb image is big enough that it is timing out on the download?

aakarshg

LGTM nice work @bengland2

bengland2 force-pushed the no-es-index-smf-fsd branch from 3a507cb to a1b5fc3 Compare November 22, 2019 19:28

acalhounRH reviewed Nov 25, 2019

View reviewed changes

run_snafu.py Outdated Show resolved Hide resolved

acalhounRH reviewed Nov 25, 2019

View reviewed changes

run_snafu.py Show resolved Hide resolved

jtaleric reviewed Nov 25, 2019

View reviewed changes

README.md Outdated Show resolved Hide resolved

default to snafu as index name prefix

bba24d4

Merge https://github.com/cloud-bulldozer/snafu into no-es-index-smf-fsd

34399a8

dry923 added the ok to test Kick off our CI framework label Dec 4, 2019

Merge https://github.com/cloud-bulldozer/snafu into no-es-index-smf-fsd

dbd12bd

bengland2 mentioned this pull request Dec 15, 2019

lengthen test to generate response time data to ES cloud-bulldozer/benchmark-operator#261

Merged

aakarshg suggested changes Dec 18, 2019

View reviewed changes

bengland2 added 2 commits January 7, 2020 09:53

Merge https://github.com/cloud-bulldozer/snafu into no-es-index-smf-fsd

42e4f39

match default index name in run_snafu.py

24c05d6

leading dash not needed anymore

d315f7c

aakarshg suggested changes Jan 8, 2020

View reviewed changes

bengland2 added 2 commits January 9, 2020 09:48

only specify wrapper-specific part of index name

e0e955b

Merge https://github.com/cloud-bulldozer/snafu into no-es-index-smf-fsd

9431e69

aakarshg approved these changes Jan 10, 2020

View reviewed changes

aakarshg merged commit d25226b into cloud-bulldozer:master Jan 10, 2020

No es index smf fsd #99

No es index smf fsd #99

Conversation

bengland2 commented Nov 22, 2019

bengland2 commented Nov 22, 2019

bengland2 commented Nov 22, 2019

acalhounRH commented Nov 25, 2019

bengland2 commented Dec 3, 2019

dry923 commented Dec 4, 2019

rht-perf-ci commented Dec 4, 2019

bengland2 commented Dec 4, 2019

aakarshg commented Dec 4, 2019

bengland2 commented Dec 4, 2019

aakarshg commented Dec 9, 2019

rht-perf-ci commented Dec 9, 2019

aakarshg commented Dec 9, 2019

bengland2 commented Dec 12, 2019

bengland2 commented Dec 15, 2019

bengland2 commented Dec 16, 2019

aakarshg commented Dec 17, 2019

aakarshg commented Dec 17, 2019

aakarshg commented Dec 17, 2019

rht-perf-ci commented Dec 17, 2019

aakarshg left a comment

Choose a reason for hiding this comment

bengland2 commented Jan 7, 2020

bengland2 commented Jan 7, 2020

bengland2 commented Jan 7, 2020

rht-perf-ci commented Jan 7, 2020

aakarshg commented Jan 7, 2020

rht-perf-ci commented Jan 7, 2020

bengland2 commented Jan 7, 2020

bengland2 commented Jan 7, 2020

aakarshg left a comment

Choose a reason for hiding this comment

bengland2 commented Jan 9, 2020

bengland2 commented Jan 9, 2020

aakarshg commented Jan 9, 2020

aakarshg commented Jan 9, 2020

rht-perf-ci commented Jan 9, 2020

bengland2 commented Jan 9, 2020

aakarshg left a comment

Choose a reason for hiding this comment