Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stress development maturity updates #3481

Merged
merged 14 commits into from
Jun 24, 2022
Merged

Conversation

benbp
Copy link
Member

@benbp benbp commented Jun 21, 2022

  • Adds a flag -LocalAddonsPath to the stress deploy script to use a local stress-test-addons instead of the remote one. Currently this has to be updated manually in Chart.yaml and is easy to forget to do.
  • Adds a -WhatIf flag to the provision script to do bicep and helm template validation.
  • Adds a pipeline to deploy the stress test examples and validate buildout actions on merge to main and nightly (it can't be run on PR since it uses deployment secrets).
  • Adds a pipeline to build out/provision a stress cluster environment.
  • Fixes an issue where the deployId is the same for all local users, so anyone deploying stress tests locally may see other users deployed tests listed in the output.
  • Fixes a minor bug with the network chaos example.
  • Fixes an unrelated issue in the powershell test pipeline that has the images mixed up.

Fixes #3135 #2635 #2764 #2635

@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@benbp benbp marked this pull request as ready for review June 22, 2022 17:52
@benbp benbp requested review from ckairen, sima-zhu, weshaggard and a team as code owners June 22, 2022 17:52
@benbp benbp self-assigned this Jun 22, 2022
@benbp benbp added Central-EngSys This issue is owned by the Engineering System team. Stress This issue is related to stress testing, part of our reliability pillar. labels Jun 22, 2022
@benbp
Copy link
Member Author

benbp commented Jun 22, 2022

The pipeline is currently failing due to permissions on the service principal that fail validation. I am following up on this.

@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@benbp
Copy link
Member Author

benbp commented Jun 22, 2022

FYI @richardpark-msft

@benbp
Copy link
Member Author

benbp commented Jun 22, 2022

Pipelines will be failing until I can rebase with @ckairen's changes to deploy to the new playground cluster which has been migrated to the latest versions of our cluster dependencies. Everything is looking as I expect it to otherwise.

@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

default: test
values:
- prod
- test
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it be pg?

Copy link
Member Author

@benbp benbp Jun 22, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I was planning on rebasing/updating after your changes went in.

$WhatIfPreference = $false
. (Join-Path $PSScriptRoot "../../../eng/common/scripts/Helpers" PSModule-Helpers.ps1)
Install-ModuleIfNotInstalled "powershell-yaml" "0.4.1" | Import-Module
$WhatIfPreference = $LastWhatIfPreference
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't quite understand why we are changing WhatIfPreference to false and then to original value again

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's somewhat annoying. There are sub-components called within Install-ModuleIfNotInstalled that don't run when -WhatIf is set that we absolutely need to run, but because the top level cmdlet doesn't have the SupportsShouldProcess binding, I don't think I can just pass -WhatIf:$false. We need this dependency installed whether we are in whatif mode or not.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for Import-Module. It doesn't support -WhatIf but if Install-ModuleIfNotInstalled doesn't return a module, then we throw.

Copy link
Member

@weshaggard weshaggard Jun 23, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explicitly pass -WhatIf:$false to those cmdlets instead of mucking around with the global variable?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated the underlying function to add SupportsShouldProcess, so it should work now.

@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

- prod
- test
- name: WhatIf
type: boolean
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be an interesting pattern. If you find that it works out OK can you please write up some guideline docs about this pattern.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep can do. Plumbing WhatIf support through all our scripts will be pretty useful for general testing and development, I think. Similarly we could run a lot more validation pipelines/scripts on PR changes if we had WhatIf enabled to catch surface level issues.

Copy link
Member

@weshaggard weshaggard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Took a high-level look at add a couple comments but otherwise seems reasonable.

@azure-sdk
Copy link
Collaborator

The following pipelines have been queued for testing:
java - template
java - template - tests
js - template
net - template
net - template - tests
python - template
python - template - tests
You can sign off on the approval gate to test the release stage of each pipeline.
See eng/common workflow

@ghost
Copy link

ghost commented Jun 24, 2022

Hello @azure-sdk!

Because this pull request has the auto-merge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

@benbp benbp merged commit 552b521 into Azure:main Jun 24, 2022
@benbp benbp deleted the benbp/stress-addon-dev branch June 24, 2022 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Central-EngSys This issue is owned by the Engineering System team. Stress This issue is related to stress testing, part of our reliability pillar.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[stress testing] Use local stress-test-addons reference in dev environment
4 participants