CI code improvements #123

rsevilla87 · 2020-01-16T13:02:31Z

This PR aims to improve ci code with things such as simplify some operatives and use more reliable methods to look for available ci scripts.
Helps on #111

rht-perf-ci · 2020-01-16T13:05:06Z

Can one of the admins verify this patch?

bengland2 · 2020-01-28T11:37:26Z

ci/run_ci.sh

  echo "Running full test"
-  test_list=`ls -d */ | grep -Ev "(utils|ci|ripsaw|image_resources)/"`
+  test_list=`find * -name ci_test.sh -type f -exec dirname {} \;`


would -maxdepth 1 be appropriate here because the ci_test.sh has to appear in the top wrapper directory? just being paranoid, but that matches what ls -d */ does more closely. But overall this is an improvement, gets rid of grep -v.

Adding -maxdepth flag... Thanks for the suggestion!

bengland2 · 2020-01-28T11:42:54Z

ci/run_ci.sh

 else
  echo "Running specific tests"
  echo $diff_list
-  test_list=`echo $diff_list | awk -F "/" '{print $1}' | uniq`
+  test_list=`echo "${diff_list}" | xargs dirname | uniq`


call me paranoid, but if someone creates a wrapper with subdirectories, would this work? I think the awk command would. I don't think any existing wrappers have subdirectories, but that could change, there is nothing requiring a wrapper not to use subdirectories. I don't think this is an essential change, do you?

You're right, this change is not essential. It is just an attempt to get the rid of awk here, which sometimes can be confusing, as this command is not focused on path parsing but in strings processing. You're also right in the fact that we're not currently using sub-directories, dirname might be problematic here.

bengland2 · 2020-01-28T11:52:10Z

ci/run_ci.sh

-      test_rc=1
-    fi
-    wait_clean
+for dir in ${test_list}; do


so if I understand this correctly, we got rid of the -f $my_dir/ci_test.sh because the test_list was constructed to have only directories containing a ci_test.sh. The only other change I see here is ${dir::-1} to ${dir}, which makes sense because if "dir" is an empty string or undefined then how did it get into test_list? it's hard to see if there are other changes here because of the indentation change, but I think that's all, right?

Yes, this is exactly what I wanted to do.

bengland2

overall good cleanups that make it more maintainable, just a couple of minor changes requested. Because it's core code for the CI, am being very picky. HTH -ben

bengland2 · 2020-01-28T13:54:30Z

LGTM now -ben

dry923 · 2020-01-28T14:03:49Z

/rerun all

rht-perf-ci · 2020-01-28T16:19:19Z

Results for SNAFU CI Test

Test	Result	Runtime
fio_wrapper	FAIL	00:22:13
fs_drift_wrapper	PASS	00:09:03
hammerdb	PASS	00:17:29
iperf	FAIL	00:03:01
pgbench-wrapper	PASS	00:22:08
smallfile_wrapper	FAIL	00:09:35
sysbench	FAIL	00:05:47
uperf-wrapper	PASS	00:09:41
ycsb-wrapper	PASS	00:28:56

dry923 · 2020-01-28T18:51:29Z

@rsevilla87 FYI theres a conflict in ci/common.sh

bengland2 · 2020-01-29T22:15:53Z

I don't understand output of snafu_linters CI job. It seems not to have run at all, did it?

> git rev-parse origin-pull/pull/130/merge^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch configuration for this job.

@aakarshg

dry923 · 2020-02-13T15:44:18Z

I don't understand output of snafu_linters CI job. It seems not to have run at all, did it?
> git rev-parse origin-pull/pull/130/merge^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch configuration for this job.
@aakarshg

@aakarshg hey any idea on ^

aakarshg · 2020-02-13T15:53:35Z

I don't understand output of snafu_linters CI job. It seems not to have run at all, did it?
> git rev-parse origin-pull/pull/130/merge^{commit} # timeout=10
ERROR: Couldn't find any revision to build. Verify the repository and branch configuration for this job.
@aakarshg
@aakarshg hey any idea on ^

this was when we're testing the job. It'll run fine now if we hit the rerun

aakarshg · 2020-03-03T17:02:07Z

/rerun all

rht-perf-ci · 2020-03-03T17:58:24Z

Results for SNAFU CI Test

Test	Result	Runtime
fio_wrapper	FAIL	00:05:10
fs_drift_wrapper	FAIL	00:04:30
hammerdb	FAIL	00:05:19
iperf	FAIL	00:02:53
pgbench-wrapper	FAIL	00:04:40
smallfile_wrapper	FAIL	00:04:30
sysbench	FAIL	00:03:01
uperf-wrapper	FAIL	00:04:26
ycsb-wrapper	FAIL	00:10:31

rht-perf-ci · 2020-03-03T20:08:13Z

Results for SNAFU CI Test

Test	Result	Runtime
fio_wrapper	PASS	00:13:04
fs_drift_wrapper	PASS	00:10:59
hammerdb	PASS	00:10:03
iperf	PASS	00:06:34
pgbench-wrapper	PASS	00:07:48
smallfile_wrapper	PASS	00:12:09
sysbench	PASS	00:05:37
uperf-wrapper	FAIL	00:10:24
ycsb-wrapper	FAIL	00:05:45

rsevilla87 · 2020-03-05T09:31:30Z

/rerun all

rht-perf-ci · 2020-03-05T11:07:12Z

Results for SNAFU CI Test

Test	Result	Runtime
fio_wrapper	PASS	00:13:50
fs_drift_wrapper	PASS	00:11:04
hammerdb	PASS	00:10:10
iperf	PASS	00:06:24
pgbench-wrapper	PASS	00:07:38
smallfile_wrapper	PASS	00:12:17
sysbench	PASS	00:05:36
uperf-wrapper	PASS	00:10:33
ycsb-wrapper	PASS	00:12:14

dry923

LGTM!

aakarshg

LGTM, will merge once @bengland2 also signs off, as he'd a few comments on some parts of the PR

bengland2

sorry, thought I already approved this. Anyway, it is good cleanup work that will make it more maintainable. I already commented on it to Raul and in here.

rsevilla87 requested a review from dry923 January 16, 2020 16:12

rsevilla87 requested review from aakarshg and bengland2 and removed request for dry923 January 28, 2020 08:40

bengland2 reviewed Jan 28, 2020

View reviewed changes

bengland2 suggested changes Jan 28, 2020

View reviewed changes

rsevilla87 force-pushed the ci-cleanup branch 2 times, most recently from b9a3d49 to 7c8021b Compare January 28, 2020 12:18

dry923 added the ok to test Kick off our CI framework label Jan 28, 2020

rsevilla87 force-pushed the ci-cleanup branch from 7c8021b to 5354275 Compare January 28, 2020 17:03

rsevilla87 force-pushed the ci-cleanup branch from 5354275 to 5fdb69e Compare January 28, 2020 22:27

aakarshg requested a review from dry923 February 20, 2020 15:45

rsevilla87 force-pushed the ci-cleanup branch 3 times, most recently from 8113c92 to f37d420 Compare March 3, 2020 18:39

CI code improvements

890d070

rsevilla87 force-pushed the ci-cleanup branch from f37d420 to 890d070 Compare March 4, 2020 08:53

dry923 approved these changes Mar 5, 2020

View reviewed changes

aakarshg requested a review from bengland2 March 5, 2020 19:24

aakarshg approved these changes Mar 5, 2020

View reviewed changes

bengland2 approved these changes Mar 5, 2020

View reviewed changes

aakarshg merged commit 65c73ff into cloud-bulldozer:master Mar 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI code improvements #123

CI code improvements #123

rsevilla87 commented Jan 16, 2020 •

edited

Loading

rht-perf-ci commented Jan 16, 2020

bengland2 Jan 28, 2020

rsevilla87 Jan 28, 2020

bengland2 Jan 28, 2020

rsevilla87 Jan 28, 2020

bengland2 Jan 28, 2020

rsevilla87 Jan 28, 2020

bengland2 left a comment

bengland2 commented Jan 28, 2020

dry923 commented Jan 28, 2020

rht-perf-ci commented Jan 28, 2020

dry923 commented Jan 28, 2020

bengland2 commented Jan 29, 2020

dry923 commented Feb 13, 2020

aakarshg commented Feb 13, 2020

aakarshg commented Mar 3, 2020

rht-perf-ci commented Mar 3, 2020

rht-perf-ci commented Mar 3, 2020

rsevilla87 commented Mar 5, 2020

rht-perf-ci commented Mar 5, 2020

dry923 left a comment

aakarshg left a comment

bengland2 left a comment

CI code improvements #123

CI code improvements #123

Conversation

rsevilla87 commented Jan 16, 2020 • edited Loading

rht-perf-ci commented Jan 16, 2020

bengland2 Jan 28, 2020

Choose a reason for hiding this comment

rsevilla87 Jan 28, 2020

Choose a reason for hiding this comment

bengland2 Jan 28, 2020

Choose a reason for hiding this comment

rsevilla87 Jan 28, 2020

Choose a reason for hiding this comment

bengland2 Jan 28, 2020

Choose a reason for hiding this comment

rsevilla87 Jan 28, 2020

Choose a reason for hiding this comment

bengland2 left a comment

Choose a reason for hiding this comment

bengland2 commented Jan 28, 2020

dry923 commented Jan 28, 2020

rht-perf-ci commented Jan 28, 2020

dry923 commented Jan 28, 2020

bengland2 commented Jan 29, 2020

dry923 commented Feb 13, 2020

aakarshg commented Feb 13, 2020

aakarshg commented Mar 3, 2020

rht-perf-ci commented Mar 3, 2020

rht-perf-ci commented Mar 3, 2020

rsevilla87 commented Mar 5, 2020

rht-perf-ci commented Mar 5, 2020

dry923 left a comment

Choose a reason for hiding this comment

aakarshg left a comment

Choose a reason for hiding this comment

bengland2 left a comment

Choose a reason for hiding this comment

rsevilla87 commented Jan 16, 2020 •

edited

Loading