split ci files #4027
split ci files #4027
Conversation
|
@joshdholtz tagged you for early thoughts since you had had a stab at this in the past |
.circleci/config.yml
Outdated
|
|
||
|
|
||
|
|
There was a problem hiding this comment.
lol guess I had a ton of line breaks at the end of the other files.
But the idea is for this one to get generated from the others
.circleci/configuration/config.yml
Outdated
| @@ -0,0 +1,1194 @@ | |||
| orbs: | |||
There was a problem hiding this comment.
this is an exact copy of the other config file minus the workflows part
| @@ -0,0 +1,252 @@ | |||
| workflows: | |||
There was a problem hiding this comment.
and here is just the workflows part
scripts/pre-commit.sh
Outdated
| @@ -82,3 +82,34 @@ if [ $SHOULD_FAIL_PRECOMMIT -ne 0 ]; then | |||
| else | |||
| verify_no_included_apikeys | |||
| fi | |||
|
|
|||
|
|
|||
| # Function to concatenate CircleCI configuration files | |||
There was a problem hiding this comment.
I'm generally allergic to bash, I might study porting all the logic into ruby at some point
| @@ -0,0 +1,56 @@ | |||
| aliases: | |||
There was a problem hiding this comment.
this is just cut/paste from what used to be only in config.yml
|
|
||
| # CircleCI doesn't let us split a config file into multiple, so we have a pre-commit hook that does it for us | ||
| # so that the file doesn't become unmanageable. | ||
| concatenate_circleci_configs() { |
There was a problem hiding this comment.
this is the only file with real changes aside from the original config.yml changes, so those are the only two ones worth really reviewing
|
the diff makes this look big but there really are only 2 small files worth reviewing in here, the rest are mostly ignorable in they're cut / paste from the original config.yml file |
| @@ -1,11 +1,20 @@ | |||
| # This file is generated automatically by concatenating the files in the .circleci/configuration/ directory. | |||
| # DO NOT EDIT this file directly. Instead, edit the files in the .circleci/configuration/ directory and rely on the pre-commit hook to update this file. | |||
There was a problem hiding this comment.
I was wondering if we should add a test job to make sure this file matches the output from the individual configuration files, to make sure it's not outdated/it's changed accidentally.
|
|
||
| # CircleCI doesn't let us split a config file into multiple, so we have a pre-commit hook that does it for us | ||
| # so that the file doesn't become unmanageable. | ||
| concatenate_circleci_configs() { |
There was a problem hiding this comment.
Might be worth extracting this to a separate script and just call it from the pre-commit.sh? Specially if we follow my other suggestion about adding a test job to make sure the combined file matches the individual files.
|
|
||
| # CircleCI doesn't let us split a config file into multiple, so we have a pre-commit hook that does it for us | ||
| # so that the file doesn't become unmanageable. | ||
| concatenate_circleci_configs() { |
There was a problem hiding this comment.
Could we use a CircleCI setup job to generate the final CircleCI config file instead? This would avoid the need for a pre-commit hook, and keep the CircleCI-related logic self contained (and running on CI instead of locally, the concatenation that is).
Here's an example for exactly our use case.
In this example, a script is used to generate a YAML configuration file. The continuation orb is used to continue the pipeline using the generated configuration.
CircleCI doesn't have direct support for splitting a config file into multiple, so this adds support for it a bit more manually:
It adds logic to our pre-commit hook to concatenate files in a configuration folder.
This only runs if files in that folder have changed.
This is the basic file structure:
So all of the files in that folder, in order, comprise the config.yml file that looks pretty much identical to the one we had before.
This approach does take two assumptions: