Release v3.2.1 #729
Merged
Release v3.2.1 #729
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* add autoescape true * Update initial_commit.py with space * Update initial_commit.py Add autoescape=True to avoid XSS
Co-authored-by: Christoffer Pozeus <christoffer.pozeus.2@consultant.volvo.com> Co-authored-by: Stewart Wallace <stewwall@amazon.com>
**Why?** To allow the deletion of the Default VPC if one were to manage the management account via ADF's Account Management (adf-accounts). **What?** Added the missing permissions to delete and describe the default VPCs in the management account.
…wslabs#700) **Why?** When ADF needs to tag or untag the cross-account access role, it fails to do so. **What?** * Granting the required permissions to tag and untag the roles. * Removal of the invalid `s3:listObjects` permission, the permissions required to list objects in a bucket are managed via the `s3:listBucket` permission. Which is already present.
**Why?** With the CloudFormation deployment provider: It would fail to create the pipeline if one were to set the `change_set_approval` property to `True`, while the pipeline lives in another region than where it is asked to deploy to. **What?** * The approval action should be performed in the region that hosts the pipeline. Not the target region.
* New install process **Why?** With the upcoming release of ADF, CDK v2 support is added. The dependencies that go with it, unfortunately increased the deployment size beyond the limit that is supported by the Serverless Application Repository. Hence a new installation mechanism had to be developed. **What?** This change proposes to use make as the deployment mechanism. I chose make, as this eases support queries on deployments of old version and future versions. For example, by default, the `make` command will install the latest `aws-sam-cli` and `yq` dependencies. However, if these appear to be incompatible, we can instruct to use a specific version of the make install script: ```bash make update_makefile make UPDATE_VERSION=make/2.0 update_makefile ``` Similarly, we can instruct using the AWS SAM CLI used at the time the ADF version was developed by running: ```bash make clean deps docker version_number git_ignore sam_build post_build ``` In case someone runs into an issue while building or deploying ADF, they can generate a report of their environment with: ```bash make version_report ``` This generates a report of all the versions of the build environment. _Long story short:_ The proposed Makefile enables various paths to resolve and investigate issues. **Changes** * Updated the Makefile to act as the new deployment mechanism. * Moved tox related tasks to test and lint the code to Makefile.tox. As these need to run in the virtual environment managed by Tox, not the one managed by the new Makefile. * Updated the docs to support ADF administrators updating and installing using the new Makefile. * Tested with v3.2.0 deployments and the upcoming release of ADF. * Added support to generate version numbers on customized ADF installations. * Updated the contribution docs to clarify how to deploy and test. * Added support to update the Makefile in the future, such that a deployment issue in a specific ADF version can be fixed by updating the Makefile. * Updated GitHub workflows accordingly, plus added new tests to ensure the dependencies installed by ADF are compatible with each other. * Added a new step in the GitHub workflow to test a full build of ADF. * Updated the Bug Issue template to include the version report and to allow the reporter to indicate that they are working on a fix already. * Add macOS support to generate a version report * Update contribution guide to include upstream tags fetch step * Fix contributing guide typo
…wslabs#702) **Why?** When running the make / install process of ADF, we need to ensure it will not continue if some of the tools are not installed yet. For example, in the past, if you did not install jq, it would still continue with the build process. Resulting in a broken template at deployment time. As reported in awslabs#696, the Python version in the environment was set to v3.8, which resulted in a broken installation process too. **What?** * Updated the documentation to indicate that `jq` needs to be available too. * Added checks in the build process to exit and warn the user about missing dependencies. * Updated the version number of the Makefile.
**Why?** When you attempt to install ADF the first time, it will bootstrap the deployment account via the account bootstrap state machine. This, however, happens just before executing the bootstrap pipeline the first time. Resulting in missing parameters in the deployment account at the time the `regional` stack is deployed. This stack requires the `extensions/terraform/enabled` parameter. If that is not there yet, which is the case in the situation above, it will fail to bootstrap the account. Hereby the bootstrap state machine and pipeline both fail initially. **What?** When the bootstrap state machine determines the event details, it should try to retrieve the parameter in the management account to see if Terraform is enabled or not. If this is not configured yet, it should default to False.
…ootstrap **Why?** Issues: awslabs#659 and awslabs#594. When installing ADF the first time, the global IAM bootstrap stack that gets deployed is sourced from the `adf-bootstrap/global-iam.yml`. The reason for this behaviour is the absence of the `global-iam.yml` file in the deployment OU bootstrap folder (`adf-bootstrap/deployment/global-iam.yml`). It iterates to the parent directory until it finds a `global-iam.yml` to deploy. Hence, when the `adf-bootstrap/global-iam.yml` gets deployed in the deployment account, it was looking for the `deployment_account_id` SSM parameter. That did not get deployed in the deployment account. **What?** * Add the creation of the `deployment_account_id` in the deployment account, so if the global IAM bootstrap stack failed to deploy before, it will work in the next release. This would be the case if the previous deployment failed but the same `aws-deployment-framework-bootstrap` repository is used in the upgrade. * When installing the first time, it creates the bootstrap repository. At the time of creation, it will copy the `adf-bootstrap/deployment/example-global-iam.yml` to `adf-bootstrap/deployment/global-iam.yml`. The same logic as how ADF creates the initial `adf-bootstrap/global-iam.yml`.
**Why?** At the moment, pytest failures were ignored due to a change in the Makefile used to execute tests. The ADF CI GitHub Workflow would result in a success, even when a test case failed. **What?** Fixed by exiting on the first failure using Makefile foreach instead.
awslabs#721) * Bump jinja2 from 3.1.3 to 3.1.4 in /src/lambda_codebase/initial_commit Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.3 to 3.1.4. - [Release notes](https://github.com/pallets/jinja/releases) - [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst) - [Commits](pallets/jinja@3.1.3...3.1.4) --- updated-dependencies: - dependency-name: jinja2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Move PR 720 into 721 **Why?** Both requirements need to be changed in synchrony. Otherwise the pipeline will fail. Hence updating this PR to include the changes from awslabs#720. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Simon Kok <sbkok@users.noreply.github.com>
**Why?**
When you clone the ADF Github repository, it will not fetch the git tags.
The installation docs describe that you should run `git fetch --tags` to
retrieve those. However, if someone forgets to run that command, you run
into a weird error that does not describe what is going wrong in a resolvable
manner:
> Error: Failed to create changeset for the stack: aws-deployment-framework,
> Parameter validation failed:
> Invalid length for parameter Tags[0].Value, value: 0, valid min length: 1
Additionally, if you installed ADF with the updated installer before. Then
forget to fetch the latest git tags when you are about to deploy a new version
of ADF, it will return a version number that is based on the old version
instead of the new one. Making it hard to determine what version of ADF was
installed. Long story short, the version in the `src/template.yml` should never
be newer than the version determined via git tags.
**What?**
Added check to verify that the version number can be determined correctly.
* If it did not return the version number, but an empty string instead:
* it will test whether there is a .git folder.
* If not, the user should clone a repository instead.
* If the .git directory exists, it will instruct the user to fetch the
git tags.
* If a version number is returned, it will check whether the version number
matches with the tag.
* If it does, this means that it is building a released version of ADF.
* If it does not, the user might not have switched to the right version tag
yet. Hence, we warn the user to proceed with caution.
Additionally, this change will add a warning when ADF gets deployed that the
specific version and changes should be reviewed and tested in a non-production
environment.
Lastly, a check is added to ensure that the ADF version of the
`src/template.yml` file should not be newer than the version resolved via git
tags. If this is the case, it should instruct the user to fetch the latest tags
from the ADF repository.
javydekoning
previously approved these changes
May 17, 2024
**Why?** In case one were to checkout a tagged release, then make changes, ADF will receive the tagged release number. Instead of tracking the change that was made in the release number as well. By preventing these uncommitted changes, it makes it easier to detect what code is running in a specific version of ADF. So in case someone were to report an issue we can assume the number indicates which version of the code failed. **What?** * Added check for uncommitted changes, these include files that are tracked by the git repository. * Added check for newly created files, these would not be tracked by git yet.
**Why?** In the next v4 release, the parameters automatically add the required prefixes. With v3.2, this was not the case yet. Since we are porting changes back into a new release of v3.2 we need to make sure they are compatible. **What?** Fixing the lookup and put path references as required to install and bootstrap ADF.
sbkok
force-pushed
the
fix/v3.2-first-install
branch
from
ec43408
to
86a383f
Compare
sbkok
added a commit
to sbkok/aws-deployment-framework
that referenced
this pull request
May 23, 2024
The upcoming v4 release will introduce breaking changes. As always, it is recommended to thoroughly review and test the upgrade procedure in a non-production environment. Unfortunately, a few issues popped up in ADF v3.2.0 that block a clean installation in a new environment. Making it harder to test the upgrade process. This release resolves those issues and includes an updated installer for ADF to simplify installation. We hope this shortens the time required to prepare for the v4 upgrade. --- **Fixes 🐞** * Fix management account config alias through ADF account management (awslabs#596) by @sbkok. * Fix CodeBuild stage naming bug (awslabs#628) by @pozeus. * Fix Jinja2 template rendering with autoescape enabled (awslabs#690) by @sujay0412. * Fix missing deployment_account_id and initial deployment global IAM bootstrap (awslabs#686) by @sbkok, resolves awslabs#594 and awslabs#659. * Fix permissions to enable delete default VPC in management account (awslabs#699) by @sbkok. * Fix tagging of Cross Account Access role in the management account (awslabs#700) by @sbkok. * Fix CloudFormation cross-region changeset approval (awslabs#701) by @sbkok. * Fix clean bootstrap of the deployment account (awslabs#703) by @sbkok, resolves awslabs#696. * Bump Jinja2 from 3.1.3 to 3.1.4 (awslabs#720 and awslabs#721) by @dependabot. * Fix account management lambdas in v3.2 (awslabs#729) by @sbkok. --- **Installation enhancements 🚀** This release is the first release with the new installation process baked in. Please read the [Installation Guide](https://github.com/awslabs/aws-deployment-framework/blob/make/latest/docs/installation-guide.md) how to install ADF. In case you are upgrading, please follow [the admin guide on updating ADF](https://github.com/awslabs/aws-deployment-framework/blob/make/latest/docs/admin-guide.md#updating-between-versions) instead. Changes baked into this release to support the new installation process: * New install process (awslabs#677) by @sbkok. * Install: Add checks to ensure installer dependencies are available (awslabs#702) by @sbkok. * Install: Add version checks and pre-deploy warnings (awslabs#726) by @sbkok. * Ensure tox fails at first pytest failure (awslabs#686) by @sbkok.
javydekoning
previously approved these changes
May 23, 2024
sbkok
added a commit
to sbkok/aws-deployment-framework
that referenced
this pull request
May 24, 2024
The upcoming v4 release will introduce breaking changes. As always, it is recommended to thoroughly review and test the upgrade procedure in a non-production environment. Unfortunately, a few issues popped up in ADF v3.2.0 that block a clean installation in a new environment. Making it harder to test the upgrade process. This release resolves those issues and includes an updated installer for ADF to simplify installation. We hope this shortens the time required to prepare for the v4 upgrade. --- **Fixes 🐞** * Fix management account config alias through ADF account management (awslabs#596) by @sbkok. * Fix CodeBuild stage naming bug (awslabs#628) by @pozeus. * Fix Jinja2 template rendering with autoescape enabled (awslabs#690) by @sujay0412. * Fix missing deployment_account_id and initial deployment global IAM bootstrap (awslabs#686) by @sbkok, resolves awslabs#594 and awslabs#659. * Fix permissions to enable delete default VPC in management account (awslabs#699) by @sbkok. * Fix tagging of Cross Account Access role in the management account (awslabs#700) by @sbkok. * Fix CloudFormation cross-region changeset approval (awslabs#701) by @sbkok. * Fix clean bootstrap of the deployment account (awslabs#703) by @sbkok, resolves awslabs#696. * Bump Jinja2 from 3.1.3 to 3.1.4 (awslabs#720 and awslabs#721) by @dependabot. * Fix account management lambdas in v3.2 (awslabs#729) by @sbkok. * Fix management account missing required IAM Tag Role permission in v3.2 (awslabs#729) by @sbkok. --- **Installation enhancements 🚀** This release is the first release with the new installation process baked in. Please read the [Installation Guide](https://github.com/awslabs/aws-deployment-framework/blob/make/latest/docs/installation-guide.md) how to install ADF. In case you are upgrading, please follow [the admin guide on updating ADF](https://github.com/awslabs/aws-deployment-framework/blob/make/latest/docs/admin-guide.md#updating-between-versions) instead. Changes baked into this release to support the new installation process: * New install process (awslabs#677) by @sbkok. * Ensure tox fails at first pytest failure (awslabs#686) by @sbkok. * Install: Add checks to ensure installer dependencies are available (awslabs#702) by @sbkok. * Install: Add version checks and pre-deploy warnings (awslabs#726) by @sbkok. * Install: Add uncommitted changes check (awslabs#733)
sbkok
force-pushed
the
fix/v3.2-first-install
branch
from
86a383f
to
ac77301
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
**Why?** Issue: awslabs#728 With the latest Python runtime update provided by AWS Lambda, the boto3 dependencies were upgraded. However, with the account management Lambda functions, the `aws-xray-sdk` requires botocore v1.29.54. This version is incompatible with the latest boto3 package that was added to the Lambda function runtime. Hence, it returned an error as described in 728. **What?** To ensure compatible versions are installed, ADF's lambda functions should set the specific version of boto3 in its requirements.txt file if they rely on that.
**Why?** When deploying ADF as a fresh installation, it requires `iam:TagRole` permission to create the cross-account management role. This was not required in the past, so existing IAM roles were not impacted. **What?** Added the required permission and limited it to the two roles it needs to create. Additionally, removal of `iam:PassRole` as that was not used anywhere in the management account.
The upcoming v4 release will introduce breaking changes. As always, it is recommended to thoroughly review and test the upgrade procedure in a non-production environment. Unfortunately, a few issues popped up in ADF v3.2.0 that block a clean installation in a new environment. Making it harder to test the upgrade process. This release resolves those issues and includes an updated installer for ADF to simplify installation. We hope this shortens the time required to prepare for the v4 upgrade. --- **Fixes 🐞** * Fix management account config alias through ADF account management (awslabs#596) by @sbkok. * Fix CodeBuild stage naming bug (awslabs#628) by @pozeus. * Fix Jinja2 template rendering with autoescape enabled (awslabs#690) by @sujay0412. * Fix missing deployment_account_id and initial deployment global IAM bootstrap (awslabs#686) by @sbkok, resolves awslabs#594 and awslabs#659. * Fix permissions to enable delete default VPC in management account (awslabs#699) by @sbkok. * Fix tagging of Cross Account Access role in the management account (awslabs#700) by @sbkok. * Fix CloudFormation cross-region changeset approval (awslabs#701) by @sbkok. * Fix clean bootstrap of the deployment account (awslabs#703) by @sbkok, resolves awslabs#696. * Bump Jinja2 from 3.1.3 to 3.1.4 (awslabs#720 and awslabs#721) by @dependabot. * Fix account management lambdas in v3.2 (awslabs#729) by @sbkok. * Fix management account missing required IAM Tag Role permission in v3.2 (awslabs#729) by @sbkok. --- **Installation enhancements 🚀** This release is the first release with the new installation process baked in. Please read the [Installation Guide](https://github.com/awslabs/aws-deployment-framework/blob/make/latest/docs/installation-guide.md) how to install ADF. In case you are upgrading, please follow [the admin guide on updating ADF](https://github.com/awslabs/aws-deployment-framework/blob/make/latest/docs/admin-guide.md#updating-between-versions) instead. Changes baked into this release to support the new installation process: * New install process (awslabs#677) by @sbkok. * Ensure tox fails at first pytest failure (awslabs#686) by @sbkok. * Install: Add checks to ensure installer dependencies are available (awslabs#702) by @sbkok. * Install: Add version checks and pre-deploy warnings (awslabs#726) by @sbkok. * Install: Add uncommitted changes check (awslabs#733)
sbkok
force-pushed
the
fix/v3.2-first-install
branch
from
ac77301
to
a70bb12
Compare
StewartW
approved these changes
May 24, 2024
javydekoning
approved these changes
May 24, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
v3.2.1
The upcoming v4 release will introduce breaking changes. As always, it is
recommended to thoroughly review and test the upgrade procedure in a
non-production environment.
Unfortunately, a few issues popped up in ADF v3.2.0 that block a clean
installation in a new environment. Making it harder to test the upgrade
process. This release resolves those issues and includes an updated installer
for ADF to simplify installation.
We hope this shortens the time required to prepare for the v4 upgrade.
Fixes 🐞
(Fix missing deployment_account_id and initial deployment global IAM bootstrap #686) by @sbkok, resolves [Bug]: Fresh installation of v3.2.0 fails to bootstrap the deployment account (upgrades work) #594 and [Bug]: Parameter "deployment_account_id" into deployment account's parameter store #659.
Installation enhancements 🚀
This release is the first release with the new installation process baked in.
Please read the Installation Guide
how to install ADF. In case you are upgrading, please follow the admin guide
on updating ADF
instead.
Changes baked into this release to support the new installation process:
@sbkok.
By submitting this pull request, I confirm that you can use, modify, copy, and
redistribute this contribution, under the terms of your choice.