Adding log generator script and container file #254

dry923 · 2021-02-18T15:59:30Z

Depends-On: 545

Description

Adding a log generator workload that will generate X messages of Y size (in bytes) per second/minute over a duration of Z minutes. If provided it will also check the backed log aggregator to confirm it received the expected number of messages. Supported backends are elasticsearch and AWS cloudwatch.

NOTE: This requires a backend storage to be deployed if we want to create a valid ci test for this.

benchmark-operator PR: cloud-bulldozer/benchmark-operator#545

comet-perf-ci · 2021-02-18T16:00:07Z

Can one of the admins verify this patch?

jtaleric · 2021-02-18T16:36:08Z

I like the idea of having this part of snafu/benchmark-wrapper, here are some more ideas for this work -

Add method to "check", the expected number of logs have landed in the expected back-end. Today, it is Elastic, tomorrow CloudWatch?
Index results from the execution of the workload -- How rate, message size, back-end, and messages received
This can easily be scaled with benchmark-operator. We might want to consider the 1:N log-generator server and log-generator workers.

dry923 · 2021-03-18T13:14:25Z

@chaitanyaenr @jtaleric @rsevilla87 out of draft and good to review please.

rsevilla87 · 2021-03-23T17:38:22Z

snafu/log_generator_wrapper/trigger_log_generator.py

+            self._wait_for_pods(my_redis,channel)
+
+        timestamp = time.strftime("%Y-%m-%dT%H:%M:%S")
+        start_time = time.time()


Maybe it's better to use datetime.datetime.now() ?, I'm suggesting this because this way you can subtract dates natively (w/o converting them to epoch)

start_time = datetime.datetime.now() message_count = self._run_log_test() end_time = datetime.datetime.now() elapsed_time = (end_time - start_time).total_seconds()

@rsevilla87 fair point. It functionally works out the same though in terms of the end result. I'll play around with it after the demo today since I don't want to break it prior :-D

snafu/log_generator_wrapper/trigger_log_generator.py

rsevilla87 · 2021-03-24T15:06:31Z

/rerun all

rsevilla87 · 2021-03-24T15:11:51Z

@dry923 I wonder if we should execute all pod synchronization from here rather than benchmark-operator. Looking at other more complex cases like uperf I think we should standardize the place where we perform this kind of operations.

comet-perf-ci · 2021-03-24T16:44:19Z

Results for SNAFU CI Test

Test	Result	Runtime
snafu/hammerdb	FAIL	00:12:21
snafu/smallfile_wrapper	FAIL	00:05:21
snafu/flent_wrapper	FAIL	00:06:49
snafu/scale_openshift_wrapper	FAIL	00:03:29
snafu/sysbench	PASS	00:01:52
snafu/ycsb_wrapper	FAIL	00:10:13
snafu/stressng_wrapper	FAIL	00:11:17
snafu/fio_wrapper	FAIL	00:05:28
snafu/upgrade_openshift_wrapper	PASS	00:00:00
snafu/uperf_wrapper	FAIL	00:05:27
snafu/iperf	FAIL	00:07:15
snafu/vegeta_wrapper	FAIL	00:11:39
snafu/fs_drift_wrapper	FAIL	00:05:05
snafu/pgbench_wrapper	FAIL	00:04:40

dry923 · 2021-03-26T13:48:31Z

@dry923 I wonder if we should execute all pod synchronization from here rather than benchmark-operator. Looking at other more complex cases like uperf I think we should standardize the place where we perform this kind of operations.

@rsevilla87 updated the operator PR to include managing the syncing of the pods (see: https://github.com/dry923/ripsaw/blob/da1b75beafd66f0b9265bdaca97118923fc31c8c/roles/log_generator/templates/log_generator.yml#L75). Do you think its worth leaving the code in here as well in case it is used outside the operator?

openshift and kubernetes version

dry923 · 2021-03-26T14:27:05Z

@dry923 I wonder if we should execute all pod synchronization from here rather than benchmark-operator. Looking at other more complex cases like uperf I think we should standardize the place where we perform this kind of operations.

@rsevilla87 updated the operator PR to include managing the syncing of the pods (see: https://github.com/dry923/ripsaw/blob/da1b75beafd66f0b9265bdaca97118923fc31c8c/roles/log_generator/templates/log_generator.yml#L75). Do you think its worth leaving the code in here as well in case it is used outside the operator?

@rsevilla87 removed code from the wrapper

comet-perf-ci · 2021-03-26T15:56:44Z

Results for SNAFU CI Test

Test	Result	Runtime
snafu/hammerdb	FAIL	00:12:15
snafu/smallfile_wrapper	FAIL	00:05:08
snafu/flent_wrapper	FAIL	00:05:50
snafu/scale_openshift_wrapper	FAIL	00:03:21
snafu/sysbench	PASS	00:01:51
snafu/ycsb_wrapper	FAIL	00:09:47
snafu/stressng_wrapper	FAIL	00:11:15
snafu/fio_wrapper	FAIL	00:05:41
snafu/upgrade_openshift_wrapper	PASS	00:00:00
snafu/uperf_wrapper	FAIL	00:05:33
snafu/iperf	PASS	00:07:01
snafu/vegeta_wrapper	FAIL	00:11:36
snafu/fs_drift_wrapper	FAIL	00:05:10
snafu/pgbench_wrapper	FAIL	00:04:26

dry923 · 2021-03-29T18:21:41Z

/rerun all

comet-perf-ci · 2021-03-29T20:16:24Z

Results for SNAFU CI Test

Test	Result	Runtime
snafu/hammerdb	PASS	00:06:37
snafu/smallfile_wrapper	PASS	00:06:51
snafu/flent_wrapper	PASS	00:11:15
snafu/scale_openshift_wrapper	FAIL	00:03:30
snafu/sysbench	PASS	00:01:55
snafu/ycsb_wrapper	PASS	00:06:15
snafu/stressng_wrapper	PASS	00:03:56
snafu/fio_wrapper	PASS	00:20:18
snafu/log_generator_wrapper	PASS	00:04:11
snafu/upgrade_openshift_wrapper	PASS	00:00:00
snafu/uperf_wrapper	FAIL	00:19:59
snafu/iperf	PASS	00:07:51
snafu/vegeta_wrapper	PASS	00:06:15
snafu/fs_drift_wrapper	PASS	00:06:25
snafu/pgbench_wrapper	PASS	00:04:49

jtaleric · 2021-03-29T22:53:21Z

/rerun all

comet-perf-ci · 2021-03-30T00:53:02Z

Results for SNAFU CI Test

Test	Result	Runtime
snafu/hammerdb	PASS	00:07:40
snafu/smallfile_wrapper	PASS	00:06:50
snafu/flent_wrapper	PASS	00:10:40
snafu/scale_openshift_wrapper	FAIL	00:03:24
snafu/sysbench	PASS	00:01:52
snafu/ycsb_wrapper	PASS	00:05:54
snafu/stressng_wrapper	PASS	00:03:52
snafu/fio_wrapper	PASS	00:20:50
snafu/log_generator_wrapper	PASS	00:04:13
snafu/upgrade_openshift_wrapper	PASS	00:00:00
snafu/uperf_wrapper	PASS	00:27:44
snafu/iperf	PASS	00:06:57
snafu/vegeta_wrapper	PASS	00:06:04
snafu/fs_drift_wrapper	PASS	00:06:30
snafu/pgbench_wrapper	PASS	00:04:54

comet-perf-ci · 2021-03-30T12:08:50Z

Results for SNAFU CI Test

Test	Result	Runtime
snafu/hammerdb	PASS	00:06:45
snafu/smallfile_wrapper	PASS	00:07:00
snafu/flent_wrapper	PASS	00:12:11
snafu/scale_openshift_wrapper	PASS	00:10:11
snafu/sysbench	PASS	00:01:53
snafu/ycsb_wrapper	PASS	00:05:54
snafu/stressng_wrapper	PASS	00:03:53
snafu/fio_wrapper	FAIL	00:17:02
snafu/log_generator_wrapper	PASS	00:04:30
snafu/upgrade_openshift_wrapper	PASS	00:00:00
snafu/uperf_wrapper	FAIL	00:20:02
snafu/iperf	PASS	00:07:11
snafu/vegeta_wrapper	FAIL	00:11:44
snafu/fs_drift_wrapper	PASS	00:06:15
snafu/pgbench_wrapper	PASS	00:04:38

dry923 · 2021-03-30T12:27:59Z

/rerun all

comet-perf-ci · 2021-03-30T15:10:15Z

Results for SNAFU CI Test

Test	Result	Runtime
snafu/hammerdb	PASS	00:07:10
snafu/smallfile_wrapper	PASS	00:06:51
snafu/flent_wrapper	PASS	00:10:39
snafu/scale_openshift_wrapper	PASS	00:07:42
snafu/sysbench	PASS	00:01:49
snafu/ycsb_wrapper	PASS	00:05:39
snafu/stressng_wrapper	PASS	00:03:44
snafu/fio_wrapper	PASS	00:20:50
snafu/log_generator_wrapper	PASS	00:04:13
snafu/upgrade_openshift_wrapper	PASS	00:00:00
snafu/uperf_wrapper	PASS	00:27:56
snafu/iperf	PASS	00:07:02
snafu/vegeta_wrapper	PASS	00:06:05
snafu/fs_drift_wrapper	PASS	00:06:22
snafu/pgbench_wrapper	PASS	00:04:25

jtaleric

LGTM!

rsevilla87

LGTM!

dry923 requested review from jtaleric and chaitanyaenr February 18, 2021 15:59

dry923 changed the title ~~Adding log generator script and container file~~ [DRAFT] Adding log generator script and container file Feb 18, 2021

dry923 force-pushed the log_generator branch 5 times, most recently from 7742a7c to 5cd571c Compare March 8, 2021 17:11

dry923 force-pushed the log_generator branch 5 times, most recently from 6d7fb4f to 89b943d Compare March 18, 2021 13:13

dry923 requested a review from rsevilla87 March 18, 2021 13:14

dry923 force-pushed the log_generator branch from 89b943d to 5fbe626 Compare March 18, 2021 13:32

dry923 changed the title ~~[DRAFT] Adding log generator script and container file~~ Adding log generator script and container file Mar 18, 2021

dry923 mentioned this pull request Mar 18, 2021

Add log generator workload cloud-bulldozer/benchmark-operator#545

Merged

dry923 added the enhancement New feature or request label Mar 18, 2021

rsevilla87 reviewed Mar 23, 2021

View reviewed changes

snafu/log_generator_wrapper/trigger_log_generator.py Outdated Show resolved Hide resolved

dry923 force-pushed the log_generator branch from 5fbe626 to b5601bf Compare March 23, 2021 17:46

dry923 requested a review from mohit-sheth March 24, 2021 10:37

dry923 force-pushed the log_generator branch from b5601bf to e826f9f Compare March 24, 2021 13:22

rsevilla87 added the ok to test Kick off our CI framework label Mar 24, 2021

Adding log generator script and container file. Updated setup.cfg to set

680f77b

openshift and kubernetes version

dry923 force-pushed the log_generator branch from e826f9f to 680f77b Compare March 26, 2021 13:50

dry923 requested a review from rsevilla87 March 26, 2021 15:46

dry923 force-pushed the log_generator branch 2 times, most recently from 5c3d7c6 to 04de932 Compare March 29, 2021 18:20

Remove redis coordination

4d5c7da

dry923 force-pushed the log_generator branch from 04de932 to 4d5c7da Compare March 30, 2021 09:58

jtaleric approved these changes Mar 30, 2021

View reviewed changes

rsevilla87 approved these changes Mar 30, 2021

View reviewed changes

rsevilla87 merged commit 2f5fee3 into cloud-bulldozer:master Mar 30, 2021

sarahbx mentioned this pull request Apr 2, 2021

Add __init__.py to log_generator_wrapper #261

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding log generator script and container file #254

Adding log generator script and container file #254

dry923 commented Feb 18, 2021 •

edited

Loading

comet-perf-ci commented Feb 18, 2021

jtaleric commented Feb 18, 2021

dry923 commented Mar 18, 2021

rsevilla87 Mar 23, 2021

dry923 Mar 24, 2021

rsevilla87 commented Mar 24, 2021

rsevilla87 commented Mar 24, 2021 •

edited

Loading

comet-perf-ci commented Mar 24, 2021

dry923 commented Mar 26, 2021

dry923 commented Mar 26, 2021

comet-perf-ci commented Mar 26, 2021

dry923 commented Mar 29, 2021

comet-perf-ci commented Mar 29, 2021

jtaleric commented Mar 29, 2021

comet-perf-ci commented Mar 30, 2021

comet-perf-ci commented Mar 30, 2021

dry923 commented Mar 30, 2021

comet-perf-ci commented Mar 30, 2021

jtaleric left a comment

rsevilla87 left a comment

Adding log generator script and container file #254

Adding log generator script and container file #254

Conversation

dry923 commented Feb 18, 2021 • edited Loading

Description

comet-perf-ci commented Feb 18, 2021

jtaleric commented Feb 18, 2021

dry923 commented Mar 18, 2021

rsevilla87 Mar 23, 2021

Choose a reason for hiding this comment

dry923 Mar 24, 2021

Choose a reason for hiding this comment

rsevilla87 commented Mar 24, 2021

rsevilla87 commented Mar 24, 2021 • edited Loading

comet-perf-ci commented Mar 24, 2021

dry923 commented Mar 26, 2021

dry923 commented Mar 26, 2021

comet-perf-ci commented Mar 26, 2021

dry923 commented Mar 29, 2021

comet-perf-ci commented Mar 29, 2021

jtaleric commented Mar 29, 2021

comet-perf-ci commented Mar 30, 2021

comet-perf-ci commented Mar 30, 2021

dry923 commented Mar 30, 2021

comet-perf-ci commented Mar 30, 2021

jtaleric left a comment

Choose a reason for hiding this comment

rsevilla87 left a comment

Choose a reason for hiding this comment

dry923 commented Feb 18, 2021 •

edited

Loading

rsevilla87 commented Mar 24, 2021 •

edited

Loading