Auto-generate next version manifest(s) #461

dblock · 2021-09-13T16:50:41Z

Signed-off-by: dblock dblock@dblock.org

Description

Automatically generate next version's manifest.

Locate the latest manifest.
Attempt to get gradle properties from each component.
Generate a manifest for any new version.
Make a pull request, e.g. [AUTO] Updated input manifests. dblock/opensearch-build#2

Now generates OpenSearch 1.0.1, 1.2.0 and 2.0.0 manifests.

I'm still writing more tests, but otherwise this is ready.

Issues Resolved

Closes #388.

Check List

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov-commenter · 2021-09-13T21:08:52Z

Codecov Report

Merging #461 (b26d20e) into main (378f171) will increase coverage by 3.13%.
The diff coverage is 91.11%.

@@            Coverage Diff             @@
##             main     #461      +/-   ##
==========================================
+ Coverage   65.18%   68.32%   +3.13%     
==========================================
  Files          51       59       +8     
  Lines        1307     1487     +180     
==========================================
+ Hits          852     1016     +164     
- Misses        455      471      +16

Impacted Files	Coverage Δ
bundle-workflow/src/manifests.py	`0.00% <0.00%> (ø)`
...workflow/src/manifests_workflow/input_manifests.py	`95.65% <95.65%> (ø)`
bundle-workflow/src/git/git_repository.py	`91.11% <100.00%> (+0.41%)`	⬆️
bundle-workflow/src/manifests/input_manifests.py	`100.00% <100.00%> (ø)`
bundle-workflow/src/manifests/manifests.py	`100.00% <100.00%> (ø)`
...undle-workflow/src/manifests_workflow/component.py	`100.00% <100.00%> (ø)`
...low/src/manifests_workflow/component_opensearch.py	`100.00% <100.00%> (ø)`
...src/manifests_workflow/component_opensearch_min.py	`100.00% <100.00%> (ø)`
...-workflow/src/manifests_workflow/manifests_args.py	`100.00% <100.00%> (ø)`
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 378f171...b26d20e. Read the comment docs.

peternied · 2021-09-14T16:08:18Z

bundle-workflow/src/manifests_workflow/component_opensearch_min.py

+    def get_root_branches(self):
+        branches = ["main"]
+        remote_branches = (
+            subprocess.check_output(


Move to GitRepository? Or some kind of git actions class

Makes sense, will revisit in another PR.

peternied · 2021-09-14T17:00:12Z

.github/workflows/versions.yml

+        run: |
+          ./bundle-workflow/manifests.sh update
+      - name: Create Pull Request
+        uses: peter-evans/create-pull-request@v3


Could this pile up duplicate pull requests? Do we have any control over managing duplication?

The plugin assumes a branch name (which here I didn't specify so it will just be called "patch") and if it sees a PR from a branch called "patch", it will update it instead of creating a new one. It's pretty smart like that.

Awesome! Could we add a branch name that is based off of the version number? This way if there is a patch release and a minor release updates at the same time separate pull requests are created.

Not so easy to know what's being added, would require returning something meaningful from the build that can be parsed by the workflow. I did update the branch name to manifests/auto so that the auto-pr doesn't walk over other PRs.

Setting branch: didn't work on my fork with a permissions error that doesn't happen with defaults, so I undid that.

Signed-off-by: dblock <dblock@dblock.org>

mch2 · 2021-09-15T18:41:08Z

bundle-workflow/src/manifests_workflow/component_opensearch_min.py

+        super().__init__("OpenSearch", repo, snapshot)
+
+    @classmethod
+    def get_root_branches(self):


We should document what it means to be a 'root branch' so that it is clear feature branches with the same versions of a 'root branch' are excluded.

mch2 · 2021-09-15T18:55:17Z

bundle-workflow/src/manifests_workflow/component_opensearch.py

+from system.properties_file import PropertiesFile
+
+
+class ComponentOpenSearch(Component):


Nit - OpenSearchComponent & OpenSearchMinComponent?

I like that file names sort :)

mch2 · 2021-09-15T19:05:51Z

bundle-workflow/README.md

+
+### Auto-Generate Manifests
+
+The [manifests workflow](src/manifests) reacts to version increments in OpenSearch and its components by extracting Gradle properties from project branches. When a new version is identified, it creates a new input manifest in [manifests](../manifests) and [opens a pull request](../.github/workflows/manifests.yml).


Can we specify which branches of core & components are scanned?

If I'm reading this correctly, we are only scanning branches of core (with the specific 'root' pattern) and the 'main' branch of component repos. Is that correct?

Correct. We’re looking at anything that seems like a version number. It’s really not that important though, we could be examining all branches.

We should probably not actually be looking at branches like 1.0 to save time - in our branching strategy we always increment on main or 1.x first.

dblock mentioned this pull request Sep 13, 2021

Refactor manifest writing. #465

Merged

1 task

dblock force-pushed the generate-manifests branch 2 times, most recently from 324a8e6 to d88a549 Compare September 13, 2021 21:07

dblock force-pushed the generate-manifests branch from cba6d14 to d7db869 Compare September 14, 2021 16:10

dblock changed the title ~~WIP: Auto-generate next version manifest.~~ Auto-generate next version manifest(s) Sep 14, 2021

dblock requested review from peternied and mch2 September 14, 2021 16:18

dblock marked this pull request as ready for review September 14, 2021 16:18

peternied previously approved these changes Sep 14, 2021

View reviewed changes

dblock dismissed peternied’s stale review via d7db869 September 14, 2021 17:50

peternied self-requested a review September 14, 2021 18:42

dblock added 7 commits September 14, 2021 22:03

Auto-generate next version manifest.

223751a

Signed-off-by: dblock <dblock@dblock.org>

Added GitHub Actions workflow.

8d01b54

Signed-off-by: dblock <dblock@dblock.org>

Fixed pipenv run mypy .

ce026f5

Signed-off-by: dblock <dblock@dblock.org>

Refactored manifest writing.

00945c0

Signed-off-by: dblock <dblock@dblock.org>

Moved file.

631d30b

Signed-off-by: dblock <dblock@dblock.org>

Extracted manifests and components.

1629e4b

Signed-off-by: dblock <dblock@dblock.org>

Added unit tests for manifests_workflow.

ddf08b0

Signed-off-by: dblock <dblock@dblock.org>

dblock force-pushed the generate-manifests branch from d7db869 to ddf08b0 Compare September 14, 2021 22:03

dblock added 2 commits September 15, 2021 12:50

Added --keep.

bf3dd79

Signed-off-by: dblock <dblock@dblock.org>

Documented manifests workflow.

298fe3e

Signed-off-by: dblock <dblock@dblock.org>

peternied previously approved these changes Sep 15, 2021

View reviewed changes

Name the pull request branch.

ecfc07a

Signed-off-by: dblock <dblock@dblock.org>

dblock dismissed peternied’s stale review via ecfc07a September 15, 2021 15:39

dblock added 2 commits September 15, 2021 16:10

Cache maven local dependencies.

fc3af5a

Signed-off-by: dblock <dblock@dblock.org>

Naming branch doesn't work.

b26d20e

Signed-off-by: dblock <dblock@dblock.org>

peternied approved these changes Sep 15, 2021

View reviewed changes

This was referenced Sep 15, 2021

Automatically add new manifests to the Jenkins pipeline #487

Closed

Working backwards from the 1.1.0 release #427

Closed

mch2 approved these changes Sep 15, 2021

View reviewed changes

dblock merged commit 9a5843f into opensearch-project:main Sep 15, 2021

dblock deleted the generate-manifests branch September 15, 2021 20:43

dblock mentioned this pull request Sep 15, 2021

Followup from auto-generation of branches #494

Merged

1 task

peternied mentioned this pull request Dec 16, 2021

[Meta] Automate the OpenSearch release process #1361

Closed

34 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Auto-generate next version manifest(s) #461

Auto-generate next version manifest(s) #461

dblock commented Sep 13, 2021 •

edited

Loading

codecov-commenter commented Sep 13, 2021 •

edited

Loading

peternied Sep 14, 2021

dblock Sep 15, 2021

peternied Sep 14, 2021

dblock Sep 14, 2021

peternied Sep 15, 2021

dblock Sep 15, 2021

dblock Sep 15, 2021

mch2 Sep 15, 2021

dblock Sep 15, 2021

mch2 Sep 15, 2021

dblock Sep 15, 2021

mch2 Sep 15, 2021

dblock Sep 15, 2021

		from system.properties_file import PropertiesFile


		class ComponentOpenSearch(Component):


		### Auto-Generate Manifests

		The [manifests workflow](src/manifests) reacts to version increments in OpenSearch and its components by extracting Gradle properties from project branches. When a new version is identified, it creates a new input manifest in [manifests](../manifests) and [opens a pull request](../.github/workflows/manifests.yml).

Auto-generate next version manifest(s) #461

Auto-generate next version manifest(s) #461

Conversation

dblock commented Sep 13, 2021 • edited Loading

Description

Issues Resolved

Check List

codecov-commenter commented Sep 13, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dblock commented Sep 13, 2021 •

edited

Loading

codecov-commenter commented Sep 13, 2021 •

edited

Loading