-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ACR] Add performance tests for common scenarios #21229
Merged
Merged
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
589ee65
initial addition of perf tests
annelo-msft fa51ab9
further updates to perf tests
annelo-msft 235ca25
use separate scenarios
annelo-msft 0d9953a
doc bug
annelo-msft ab31fe3
pr fb
annelo-msft 4a787ad
Merge remote-tracking branch 'upstream/master' into acr-perf-tests
annelo-msft 6b09b03
pr fb
annelo-msft File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
15 changes: 15 additions & 0 deletions
15
...rf/Azure.Containers.ContainerRegistry.Perf/Azure.Containers.ContainerRegistry.Perf.csproj
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
<Project Sdk="Microsoft.NET.Sdk"> | ||
<PropertyGroup> | ||
<OutputType>Exe</OutputType> | ||
</PropertyGroup> | ||
<ItemGroup> | ||
<PackageReference Include="Microsoft.Azure.Management.ContainerRegistry" VersionOverride="4.0.0" /> | ||
<PackageReference Include="Microsoft.Azure.Management.ContainerRegistry.Fluent"/> | ||
</ItemGroup> | ||
|
||
<ItemGroup> | ||
<ProjectReference Include="$(MSBuildThisFileDirectory)..\..\src\Azure.Containers.ContainerRegistry.csproj" /> | ||
<ProjectReference Include="$(MSBuildThisFileDirectory)..\..\..\..\..\common\Perf\Azure.Test.Perf\Azure.Test.Perf.csproj" /> | ||
<ProjectReference Include="$(MSBuildThisFileDirectory)..\..\..\..\core\Azure.Core.TestFramework\src\Azure.Core.TestFramework.csproj" /> | ||
</ItemGroup> | ||
</Project> |
63 changes: 63 additions & 0 deletions
63
.../perf/Azure.Containers.ContainerRegistry.Perf/Infrastructure/ContainerRegistryPerfTest.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,63 @@ | ||
// Copyright (c) Microsoft Corporation. All rights reserved. | ||
// Licensed under the MIT License. | ||
|
||
using System; | ||
using System.Collections.Generic; | ||
using System.Linq; | ||
using System.Text; | ||
using System.Threading.Tasks; | ||
using Azure.Test.Perf; | ||
using Microsoft.Azure.Management.ContainerRegistry; | ||
using Microsoft.Azure.Management.ContainerRegistry.Models; | ||
using Microsoft.Azure.Management.ResourceManager.Fluent; | ||
using Microsoft.Azure.Management.ResourceManager.Fluent.Authentication; | ||
using Task = System.Threading.Tasks.Task; | ||
|
||
namespace Azure.Containers.ContainerRegistry.Perf | ||
{ | ||
public abstract class ContainerRegistryPerfTest : PerfTest<PerfOptions> | ||
{ | ||
public ContainerRegistryPerfTest(PerfOptions options) : base(options) | ||
{ | ||
} | ||
|
||
public async Task ImportImageAsync(string registry, string repository, string tag) | ||
{ | ||
await ImportImageAsync(registry, repository, new List<string>() { tag }); | ||
} | ||
|
||
public async Task ImportImageAsync(string registry, string repository, List<string> tags) | ||
{ | ||
var credential = new AzureCredentials( | ||
new ServicePrincipalLoginInformation | ||
{ | ||
ClientId = PerfTestEnvironment.Instance.ClientId, | ||
ClientSecret = PerfTestEnvironment.Instance.ClientSecret, | ||
}, | ||
PerfTestEnvironment.Instance.TenantId, | ||
AzureEnvironment.AzureGlobalCloud); | ||
|
||
var managementClient = new ContainerRegistryManagementClient(credential.WithDefaultSubscription(PerfTestEnvironment.Instance.SubscriptionId)); | ||
managementClient.SubscriptionId = PerfTestEnvironment.Instance.SubscriptionId; | ||
|
||
var importSource = new ImportSource | ||
{ | ||
SourceImage = repository, | ||
RegistryUri = "registry.hub.docker.com" | ||
}; | ||
|
||
var targetTags = tags.Select(tag => $"{repository}:{tag}"); | ||
|
||
await managementClient.Registries.ImportImageAsync( | ||
resourceGroupName: PerfTestEnvironment.Instance.ResourceGroup, | ||
registryName: registry, | ||
parameters: | ||
new ImportImageParameters | ||
{ | ||
Mode = ImportMode.Force, | ||
Source = importSource, | ||
TargetTags = targetTags.ToList() | ||
}); | ||
} | ||
} | ||
} |
31 changes: 31 additions & 0 deletions
31
...gistry/perf/Azure.Containers.ContainerRegistry.Perf/Infrastructure/PerfTestEnvironment.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
// Copyright (c) Microsoft Corporation. All rights reserved. | ||
// Licensed under the MIT License. | ||
|
||
using System; | ||
using Azure.Core.TestFramework; | ||
|
||
namespace Azure.Containers.ContainerRegistry.Perf | ||
{ | ||
/// <summary> | ||
/// Represents the ambient environment in which the test suite is being run, offering access to information such as environment variables. | ||
/// </summary> | ||
internal sealed class PerfTestEnvironment : TestEnvironment | ||
{ | ||
/// <summary> | ||
/// The shared instance of the <see cref="PerfTestEnvironment"/> to be used during test runs. | ||
/// </summary> | ||
public static PerfTestEnvironment Instance { get; } = new PerfTestEnvironment(); | ||
|
||
/// <summary> | ||
/// The endpoint of the Container Registry resource to test against. | ||
/// </summary> | ||
/// <value>The endpoint, read from the "CONTAINERREGISTRY_ENDPOINT" environment variable.</value> | ||
public string Endpoint => GetVariable("CONTAINERREGISTRY_ENDPOINT"); | ||
|
||
/// <summary> | ||
/// The name of the registry to test against. | ||
/// </summary> | ||
/// <value>The registry name, read from the "CONTAINERREGISTRY_REGISTRY_NAME" environment variable.</value> | ||
public string Registry => GetVariable("CONTAINERREGISTRY_REGISTRY_NAME"); | ||
} | ||
} |
7 changes: 7 additions & 0 deletions
7
...zure.Containers.ContainerRegistry/perf/Azure.Containers.ContainerRegistry.Perf/Program.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
// Copyright (c) Microsoft Corporation. All rights reserved. | ||
// Licensed under the MIT License. | ||
|
||
using System.Reflection; | ||
using Azure.Test.Perf; | ||
|
||
await PerfProgram.Main(Assembly.GetEntryAssembly(), args); |
15 changes: 15 additions & 0 deletions
15
...ainers.ContainerRegistry/perf/Azure.Containers.ContainerRegistry.Perf/README.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
# Azure Container Registry performance tests | ||
|
||
This folder contains performance tests for the [Azure Container Registry client library for .NET](https://github.com/Azure/azure-sdk-for-net/tree/master/sdk/containerregistry/Azure.Containers.ContainerRegistry) and its associated ecosystem. The artifacts in this library are intended to be used primarily with the Azure SDK engineering system's testing infrastructure, but may also be run as stand-alone applications from the command line. | ||
|
||
## Contributing | ||
|
||
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com. | ||
|
||
When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA. | ||
|
||
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/). For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments. | ||
|
||
Please see our [contributing guide](https://github.com/Azure/azure-sdk-for-net/blob/master/CONTRIBUTING.md) for more information. | ||
|
||
![Impressions](https://azure-sdk-impressions.azurewebsites.net/api/impressions/azure-sdk-for-net%2Fsdk%2Fcontainerregistry%2FAzure.Containers.ContainerRegistry.Perf%2FREADME.png) |
55 changes: 55 additions & 0 deletions
55
...ContainerRegistry/perf/Azure.Containers.ContainerRegistry.Perf/Scenarios/ListArtifacts.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,55 @@ | ||
// Copyright (c) Microsoft Corporation. All rights reserved. | ||
// Licensed under the MIT License. | ||
|
||
using System; | ||
using System.Threading; | ||
using System.Threading.Tasks; | ||
using Azure.Test.Perf; | ||
|
||
namespace Azure.Containers.ContainerRegistry.Perf | ||
{ | ||
public sealed class ListArtifacts : ContainerRegistryPerfTest | ||
{ | ||
private readonly ContainerRegistryClient _client; | ||
private ContainerRepository _repository; | ||
|
||
public ListArtifacts(PerfOptions options) : base(options) | ||
{ | ||
_client = new ContainerRegistryClient(new Uri(PerfTestEnvironment.Instance.Endpoint), PerfTestEnvironment.Instance.Credential); | ||
} | ||
|
||
public override async Task GlobalSetupAsync() | ||
{ | ||
// Global setup code that runs once at the beginning of test execution. | ||
await base.GlobalSetupAsync(); | ||
|
||
string repository = $"library/node"; | ||
string tag = "test-perf"; | ||
|
||
await ImportImageAsync(PerfTestEnvironment.Instance.Registry, repository, tag); | ||
} | ||
|
||
public override async Task SetupAsync() | ||
{ | ||
await base.SetupAsync(); | ||
|
||
_repository = _client.GetRepository($"library/node"); | ||
JoshLove-msft marked this conversation as resolved.
Show resolved
Hide resolved
|
||
} | ||
|
||
public override void Run(CancellationToken cancellationToken) | ||
{ | ||
foreach (var manifest in _repository.GetManifests()) | ||
{ | ||
_client.GetArtifact($"library/node", manifest.Digest); | ||
} | ||
} | ||
|
||
public override async Task RunAsync(CancellationToken cancellationToken) | ||
{ | ||
await foreach (var manifest in _repository.GetManifestsAsync()) | ||
{ | ||
_client.GetArtifact($"library/node", manifest.Digest); | ||
} | ||
} | ||
} | ||
} |
36 changes: 36 additions & 0 deletions
36
...tainerRegistry/perf/Azure.Containers.ContainerRegistry.Perf/Scenarios/ListRepositories.cs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
// Copyright (c) Microsoft Corporation. All rights reserved. | ||
// Licensed under the MIT License. | ||
|
||
using System; | ||
using System.Threading; | ||
using System.Threading.Tasks; | ||
using Azure.Test.Perf; | ||
|
||
namespace Azure.Containers.ContainerRegistry.Perf | ||
{ | ||
public sealed class ListRepositories : ContainerRegistryPerfTest | ||
{ | ||
private readonly ContainerRegistryClient _client; | ||
|
||
public ListRepositories(PerfOptions options) : base(options) | ||
{ | ||
_client = new ContainerRegistryClient(new Uri(PerfTestEnvironment.Instance.Endpoint), PerfTestEnvironment.Instance.Credential); | ||
} | ||
|
||
public override void Run(CancellationToken cancellationToken) | ||
{ | ||
foreach (var repositoryName in _client.GetRepositoryNames()) | ||
{ | ||
_client.GetRepository(repositoryName); | ||
} | ||
} | ||
|
||
public override async Task RunAsync(CancellationToken cancellationToken) | ||
{ | ||
await foreach (var repositoryName in _client.GetRepositoryNamesAsync()) | ||
{ | ||
_client.GetRepository(repositoryName); | ||
} | ||
} | ||
} | ||
} |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can the ContainerRegistryEnvironment be used instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It uses recorded variables? Honestly, I was following the README and Search and FormRecognizer examples. Beyond a small amount of code duplication, is it a concern?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, I don't think it is a concern - I was mostly curious why there was a separate TestEnvironment. I see that you don't actually have a dependency on your test project from here so it may not make sense to do so.
FWIW, I don't think it would matter that the environment uses Recorded variables as the
RecordedTestMode? Mode
property would end up being null (I think).