Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added functionality for creating and processing archive file #279

Conversation

acalhounRH
Copy link
Contributor

Description

added functionality to create and re-index a archive file for benchmark-wrapper
when creating an archive we would just include --create-archive on any run and it will drop a user_clustername_uuid.archive file in the cwd

for example python3.7 ~/git/benchmark-wrapper/snafu/run_snafu.py --tool sysbench -f /home/acalhoun/sysbench_results/example_cpu_test.conf --create-archive

when indexing from an archive we would do, run_snafu --tool archive --archive-file <file>. run_snafu will go straight to processing and indexing the listed ES friendly documents in the archive.

For example python3.7 ~/git/benchmark-wrapper/snafu/run_snafu.py --tool archive --archive-file myuser_mycluster_1b88311d-1230-409a-9ffa-94e6f9aa4754.archive -v

@jtaleric @dry923

Fixes

@dry923
Copy link
Member

dry923 commented May 26, 2021

@acalhounRH can you make it so the archive file location and possibly name can be set?

@acalhounRH acalhounRH force-pushed the add_create_run_snafu_archive_output_file branch from e5fa327 to c0de90f Compare May 26, 2021 19:18
@acalhounRH
Copy link
Contributor Author

acalhounRH commented May 26, 2021

Sure thing I used the --archive-file=<path to file/filename> and the user can provide a full path to file, it will default to cwd and user_cluster_uuid.archive. @dry923

python3.7 ~/git/benchmark-wrapper/snafu/run_snafu.py --tool sysbench -f /home/acalhoun/sysbench_results/example_cpu_test.conf --create-archive --archive-file ./alc-test.archive

@acalhounRH acalhounRH force-pushed the add_create_run_snafu_archive_output_file branch from 2269dc3 to cddbad3 Compare May 26, 2021 19:22
@acalhounRH acalhounRH force-pushed the add_create_run_snafu_archive_output_file branch 2 times, most recently from 220fb5f to d576e60 Compare May 27, 2021 15:35
@jtaleric jtaleric added the ok to test Kick off our CI framework label Jun 1, 2021
@jtaleric
Copy link
Member

jtaleric commented Jun 1, 2021

/rerun all

Copy link
Member

@jtaleric jtaleric left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor nit, but can we add a test to archive then check if the archive is present?

@acalhounRH
Copy link
Contributor Author

Minor nit, but can we add a test to archive then check if the archive is present?

No testing within benchmark-wrapper, would this request be a follow on activity for benchmark-operator?

@dry923
Copy link
Member

dry923 commented Jun 1, 2021

@acalhounRH can you update the Readme for this as well?

@jtaleric
Copy link
Member

jtaleric commented Jun 1, 2021

Minor nit, but can we add a test to archive then check if the archive is present?

No testing within benchmark-wrapper, would this request be a follow on activity for benchmark-operator?

bah-humbug...

@comet-perf-ci
Copy link
Collaborator

Results for SNAFU CI Test

Test Result Runtime
snafu/hammerdb PASS 00:06:04
snafu/smallfile_wrapper PASS 00:05:56
snafu/flent_wrapper PASS 00:10:00
snafu/scale_openshift_wrapper PASS 00:09:58
snafu/sysbench PASS 00:01:32
snafu/image_pull_wrapper PASS 00:04:12
snafu/ycsb_wrapper PASS 00:05:33
snafu/stressng_wrapper PASS 00:03:24
snafu/fio_wrapper PASS 00:18:56
snafu/log_generator_wrapper PASS 00:03:54
snafu/upgrade_openshift_wrapper PASS 00:00:00
snafu/uperf_wrapper PASS 00:26:20
snafu/iperf PASS 00:05:57
snafu/vegeta_wrapper PASS 00:05:21
snafu/fs_drift_wrapper PASS 00:05:37
snafu/pgbench_wrapper PASS 00:04:22

@acalhounRH acalhounRH force-pushed the add_create_run_snafu_archive_output_file branch from 1352fe4 to 228e43d Compare June 1, 2021 19:42
README.md Outdated Show resolved Hide resolved
@acalhounRH acalhounRH force-pushed the add_create_run_snafu_archive_output_file branch 2 times, most recently from 7322051 to d7d70d3 Compare June 1, 2021 20:04
README.md Outdated Show resolved Hide resolved
@acalhounRH acalhounRH force-pushed the add_create_run_snafu_archive_output_file branch from d28a13a to 1ef1f69 Compare June 1, 2021 20:58
@comet-perf-ci
Copy link
Collaborator

Results for SNAFU CI Test

Test Result Runtime
snafu/hammerdb PASS 00:06:10
snafu/smallfile_wrapper PASS 00:06:08
snafu/flent_wrapper PASS 00:10:02
snafu/scale_openshift_wrapper PASS 00:10:46
snafu/sysbench PASS 00:01:31
snafu/image_pull_wrapper PASS 00:04:21
snafu/ycsb_wrapper PASS 00:05:18
snafu/stressng_wrapper PASS 00:03:25
snafu/fio_wrapper PASS 00:19:05
snafu/log_generator_wrapper PASS 00:07:04
snafu/upgrade_openshift_wrapper PASS 00:00:00
snafu/uperf_wrapper PASS 00:26:19
snafu/iperf PASS 00:05:54
snafu/vegeta_wrapper PASS 00:05:24
snafu/fs_drift_wrapper PASS 00:05:32
snafu/pgbench_wrapper PASS 00:04:31

@acalhounRH acalhounRH force-pushed the add_create_run_snafu_archive_output_file branch from baddeb6 to 5ea73e9 Compare June 2, 2021 12:34
Copy link
Member

@dry923 dry923 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM pending the last CI passing.

@comet-perf-ci
Copy link
Collaborator

Results for SNAFU CI Test

Test Result Runtime
snafu/hammerdb PASS 00:06:21
snafu/smallfile_wrapper PASS 00:06:05
snafu/flent_wrapper PASS 00:09:47
snafu/scale_openshift_wrapper PASS 00:10:49
snafu/sysbench PASS 00:01:29
snafu/image_pull_wrapper PASS 00:03:53
snafu/ycsb_wrapper PASS 00:06:29
snafu/stressng_wrapper PASS 00:03:29
snafu/fio_wrapper PASS 00:19:12
snafu/log_generator_wrapper PASS 00:03:54
snafu/upgrade_openshift_wrapper PASS 00:00:00
snafu/uperf_wrapper PASS 00:26:05
snafu/iperf PASS 00:05:56
snafu/vegeta_wrapper PASS 00:05:27
snafu/fs_drift_wrapper PASS 00:05:42
snafu/pgbench_wrapper PASS 00:04:39

README.md Outdated
Note: SNAFU does not depend upon Kubernetes, so you can use run_snafu.py on a bare-metal or VM cluster without relying
on Kubernetes to start/stop pods. So if you need your benchmark to collect data for both Kubernetes and non-Kubernetes
environments, develop in SNAFU and then write benchmark-operator benchmark to integrate with Kubernetes.
** Why Should I use Benchmark-wrapper? **
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you expect some formatting like h2 with this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was expecting Bold

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, i don't think it worked.

@acalhounRH acalhounRH force-pushed the add_create_run_snafu_archive_output_file branch from 2095b8e to ff296a9 Compare June 2, 2021 17:41
Copy link
Member

@jtaleric jtaleric left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jtaleric jtaleric merged commit 2c3a05a into cloud-bulldozer:master Jun 2, 2021
@acalhounRH acalhounRH deleted the add_create_run_snafu_archive_output_file branch June 2, 2021 18:01
@comet-perf-ci
Copy link
Collaborator

!!! Couldn't read commit file !!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ok to test Kick off our CI framework
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants