This repository contains the source code for the session I presented at the Festive Tech Calendar 2023. The session covered various aspects of how to deploy Self-Hosted Azure DevOps Agent with Container App Jobs within Azure Container App Environment.
Some Pre-requisites that needs to be in tact before running the bicep deployment.
- Make sure that you have an Azure Subscription.
- Make sure that you have an Azure DevOps Organization that is connected to your Azure Subscription.
- Make sure you have the appropiate permissions to create resources in your subscription,
Owner
or relevant RBAC permissions are needed due to Role Assignments, ascontributor
role cannot assign roles to other resources. - Make sure that the latest version of Azure PowerShell is installed and user is signed in with permission to create container app environments etc
- Make sure that the latest bicep version is installed as well.
Use the following commands to install the latest version of Azure PowerShell and bicep.
Install-Module -Name Az -AllowClobber -Scope CurrentUser -Force
It's also good to register the following providers in your subscription, as they are used in the deployment.
az provider register --namespace Microsoft.ContainerRegistry
az provider register --namespace Microsoft.KeyVault
To set up the project, follow these steps:
- Clone the repository.
-
Azure DevOps - Create A Personal Access Token with
Agent Pools (read, manage)
scope and choose how long you want the token to be valid for and remember to save the token for later use, as it will not be shown again.IMPORTANT: The PAT token is needed for the KEDA azure-pipline scaler as it doesn't support Managed Identity yet, the PAT token will be added to Key Vault during the deployment and referenced in the KEDA Scaler configuration.
-
In Azure DevOps - Create a Self-hosted Agent pool at Organization Level and copy the pool name for later use.
- Modify the
.bicepparams
files with your own values for the parameters. - It's important that you add the correct pool name and PAT token in theiac/bicep/agent-infrastrucutre/main.bicepparam
file, otherwise the deployment will fail.
To run the project, follow these steps:
-
Run the 'bicep-deploy.ps1' with the following command to create the pre-requisite resources first.
$preReqDeployment = @{ adoResourceGroupName = "cnappenv-ado-rg" adoUserAssignedIdentityName = "cnappenv-ado-mi" tfResourceGroupName = "tf-infra-env-rg" tfUserAssignedIdentityName = "tf-infra-env-mi" location = "westeurope" subscriptionId = "<REPLACE WITH YOUR SUBSCRIPTION ID>" # Replace with your subscription ID tenantId = "<REPLACE WITH YOUR TENANT ID>" # Replace with your tenant ID runOnlyPreReq = $true } .\.powershell\bicep-deploy.ps1 @preReqDeployment
-
Add the
adoUserAssignedIdentity
that you created with the script to[YourAzureDevOpsOrganization]\Project Collection Service Accounts
group in Azure DevOps. This is needed so the deployment scripts and the container app can add the agent to the Azure DevOps Agent Pool, if you don't do this, the deploymentscripts for the placeholderagent and the agentjobs will fail.Note: The identity can be added to the
Project Collection Servcie Accounts
DevOps Group after the bicep deployment, but just remember to run theplaceholderagent
deployment script again so it will create the placeholder agent in Azure DevOps. -
Run the 'bicep-deploy.ps1' to create the container app environment and the terraform test environment.
$environmentParameters = @{ adoResourceGroupName = "cnappenv-ado-rg" adoUserAssignedIdentityName = "cnappenv-ado-mi" tfResourceGroupName = "tf-infra-env-rg" tfUserAssignedIdentityName = "tf-infra-env-mi" location = "westeurope" subscriptionId = "<REPLACE WITH YOUR SUBSCRIPTION ID>" # Replace with your subscription ID tenantId = "<REPLACE WITH YOUR TENANT ID>" # Replace with your tenant ID } .\.powershell\bicep-deploy.ps1 @environmentParameters
To test the environment, follow these steps:
-
Go to your Azure DevOps Organization and create a project.
-
Navigate to the project and clone this repository into your new project.
-
Create a Service Connection to your Azure Subscription, I recommend using the New Workload Identity federation feature, as it's more secure than using a Service Principal.
-
Under Pipelines - Environment - Create a new environment called
terraform
. -
Create two pipelines from existing files within the repository and give it appropriate names.
- One pipeline for scale testing:
.ado\azure-pipelines\azure-pipelines-scale-test.yml
rename it to for examplescale-test
. - One pipeline for the terraform test:
.ado\azure-pipelines\azure-pipelines-tf-test.yml
rename it to for exampleterraform-test
.
- One pipeline for scale testing:
-
Modify the pipelines to use the correct agent pool and service connection.
-
Under Pipelines - Library - Create a Variable Group called
terraform
with the following values:
Name | Value | Description |
---|---|---|
container_name | terraform-state |
The name of the container where the terraform state file will be stored. |
key_name | terraform.tfstate |
The name of the terraform state file. |
resource_group_name | Name of the resouce group that was created during the bicep deployment |
The name of the resource group where the terraform state file will be stored. |
storage_account_name | Name of the storage account that was created during the bicep deployment |
The name of the storage account where the terraform state file will be stored. |
tfversion | latest |
The version of terraform that will be used in the pipeline. |
userAzureADAuth | true |
Entra ID authentication will be used to authenticate to the Storage Account. |
- Go to your Azure Subscription and make sure to give your Azure DevOps Service Connection
Storage Blob Data Owner
role on the storage account that was created by the bicep deployment.
- Run the
scale-test
pipeline and make sure that it succeeds, you can run it multiple times to see that the agent pool scales up, just remember you need to have more then one license for parralel jobs in your Azure DevOps organization, for the agent pool to scale up, otherwise it will only be able to run one job at a time. - You might get a prompt for allowing access to use the agent pool, environment etc, click permit and continue.
- Run the
terraform-test
pipeline. - You might get a prompt for allowing access to use the agent pool, environment etc, click permit and continue.
- The pipeline will run required terraform pre-requisites such as
terraform init
,terraform plan
etc. - After running the first stage it will wait for manual appoval, then run the
terraform apply
wich will create a resource group defined in the terraform code.
To clean up the project, follow these steps:
-
Delete the Resource Group created by the Terraform test pipeline.
-
Delete the Azure DevOps Project.
-
Delete the Azure DevOps Agent Pool.
-
Remove the managed identity from the
Project Collection Service Accounts
group in Azure DevOps. -
Run the
bicep-deploy.ps1
with the following command to delete the resources that were created by pre-requisite deployment.$cleanUpParameters = @{ adoResourceGroupName = "ftechcal2023-ado-rg" tfResourceGroupName = "ftechcal2023-tf-rg" location = "westeurope" subscriptionId = "<REPLACE WITH YOUR SUBSCRIPTION ID>" # Replace with your subscription ID tenantId = "<REPLACE WITH YOUR TENANT ID>" # Replace with your tenant ID cleanup = $true } .\.powershell\bicep-deploy.ps1 @cleanUpParameters
Originally created by Haflidi Fridthjofsson