Skip to content

Commit

Permalink
[Key Vault] Add perf tests for certificates, keys, and secrets (#17073)
Browse files Browse the repository at this point in the history
  • Loading branch information
mccoyp authored Mar 5, 2021
1 parent c90a60d commit 1025c0b
Show file tree
Hide file tree
Showing 9 changed files with 294 additions and 0 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# azure-keyvault-certificates Performance Tests

In order to run the performance tests, the `azure-devtools` package must be installed. This is done as part of the
`dev_requirements` install. Start by creating a new virtual environment for your perf tests. This will need to be a
Python 3 environment, preferably >=3.7.

### Setup for test resources

The following environment variables will need to be set for the tests to access the live resources:

```
AZURE_TENANT_ID=<tenant ID of testing service principal>
AZURE_CLIENT_ID=<client ID of testing service principal>
AZURE_CLIENT_SECRET=<client secret of testing service principal>
AZURE_KEYVAULT_URL=<URL of the testing key vault>
```

### Setup for perf test runs

```cmd
(env) ~/azure-keyvault-certificates> pip install -r dev_requirements.txt
(env) ~/azure-keyvault-certificates> pip install -e .
```

## Test commands

Once `azure-devtools` is installed, you will have access to the `perfstress` command line tool, which will scan the
current module for runnable perf tests. Only a specific test can be run at a time (i.e. there is no "run all" feature).

```cmd
(env) ~/azure-keyvault-certificates> cd tests/perfstress_tests/
(env) ~/azure-keyvault-certificates/tests> perfstress
```
Using the `perfstress` command alone will list the available perf tests found.

### Common perf command line options
These options are available for all perf tests:
- `--duration=10` Number of seconds to run as many operations (the "run" function) as possible. Default is 10.
- `--iterations=1` Number of test iterations to run. Default is 1.
- `--parallel=1` Number of tests to run in parallel. Default is 1.
- `--warm-up=5` Number of seconds to spend warming up the connection before measuring begins. Default is 5.
- `--sync` Whether to run the tests in sync or async. Default is False (async).
- `--no-cleanup` Whether to keep newly created resources after test run. Default is False (resources will be deleted).

## Example command
```cmd
(env) ~/azure-keyvault-certificates/tests> perfstress GetCertificateTest
```
Empty file.
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# ------------------------------------
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
# ------------------------------------

from azure_devtools.perfstress_tests import PerfStressTest
from azure.identity import EnvironmentCredential
from azure.identity.aio import EnvironmentCredential as AsyncEnvironmentCredential
from azure.keyvault.certificates import CertificateClient, CertificatePolicy
from azure.keyvault.certificates.aio import CertificateClient as AsyncCertificateClient


class GetCertificateTest(PerfStressTest):

def __init__(self, arguments):
super().__init__(arguments)

# Auth configuration
self.credential = EnvironmentCredential()
self.async_credential = AsyncEnvironmentCredential()

# Create clients
vault_url = self.get_from_env("AZURE_KEYVAULT_URL")
self.client = CertificateClient(vault_url, self.credential)
self.async_client = AsyncCertificateClient(vault_url, self.async_credential)

async def global_setup(self):
"""The global setup is run only once."""
await super().global_setup()
await self.async_client.create_certificate("livekvtestperfcert", CertificatePolicy.get_default())

async def global_cleanup(self):
"""The global cleanup is run only once."""
await self.async_client.delete_certificate("livekvtestperfcert")
await self.async_client.purge_deleted_certificate("livekvtestperfcert")
await super().global_cleanup()

async def close(self):
"""This is run after cleanup."""
await self.async_client.close()
await self.async_credential.close()
await super().close()

def run_sync(self):
"""The synchronous perf test."""
self.client.get_certificate("livekvtestperfcert")

async def run_async(self):
"""The asynchronous perf test."""
await self.async_client.get_certificate("livekvtestperfcert")
48 changes: 48 additions & 0 deletions sdk/keyvault/azure-keyvault-keys/tests/perfstress_tests/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# azure-keyvault-keys Performance Tests

In order to run the performance tests, the `azure-devtools` package must be installed. This is done as part of the
`dev_requirements` install. Start by creating a new virtual environment for your perf tests. This will need to be a
Python 3 environment, preferably >=3.7.

### Setup for test resources

The following environment variables will need to be set for the tests to access the live resources:

```
AZURE_TENANT_ID=<tenant ID of testing service principal>
AZURE_CLIENT_ID=<client ID of testing service principal>
AZURE_CLIENT_SECRET=<client secret of testing service principal>
AZURE_KEYVAULT_URL=<URL of the testing key vault>
```

### Setup for perf test runs

```cmd
(env) ~/azure-keyvault-keys> pip install -r dev_requirements.txt
(env) ~/azure-keyvault-keys> pip install -e .
```

## Test commands

Once `azure-devtools` is installed, you will have access to the `perfstress` command line tool, which will scan the
current module for runnable perf tests. Only a specific test can be run at a time (i.e. there is no "run all" feature).

```cmd
(env) ~/azure-keyvault-keys> cd tests/perfstress_tests/
(env) ~/azure-keyvault-keys/tests> perfstress
```
Using the `perfstress` command alone will list the available perf tests found.

### Common perf command line options
These options are available for all perf tests:
- `--duration=10` Number of seconds to run as many operations (the "run" function) as possible. Default is 10.
- `--iterations=1` Number of test iterations to run. Default is 1.
- `--parallel=1` Number of tests to run in parallel. Default is 1.
- `--warm-up=5` Number of seconds to spend warming up the connection before measuring begins. Default is 5.
- `--sync` Whether to run the tests in sync or async. Default is False (async).
- `--no-cleanup` Whether to keep newly created resources after test run. Default is False (resources will be deleted).

## Example command
```cmd
(env) ~/azure-keyvault-keys/tests> perfstress GetKeyTest
```
Empty file.
50 changes: 50 additions & 0 deletions sdk/keyvault/azure-keyvault-keys/tests/perfstress_tests/get_key.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# ------------------------------------
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
# ------------------------------------

from azure_devtools.perfstress_tests import PerfStressTest
from azure.identity import EnvironmentCredential
from azure.identity.aio import EnvironmentCredential as AsyncEnvironmentCredential
from azure.keyvault.keys import KeyClient
from azure.keyvault.keys.aio import KeyClient as AsyncKeyClient


class GetKeyTest(PerfStressTest):

def __init__(self, arguments):
super().__init__(arguments)

# Auth configuration
self.credential = EnvironmentCredential()
self.async_credential = AsyncEnvironmentCredential()

# Create clients
vault_url = self.get_from_env("AZURE_KEYVAULT_URL")
self.client = KeyClient(vault_url, self.credential)
self.async_client = AsyncKeyClient(vault_url, self.async_credential)

async def global_setup(self):
"""The global setup is run only once."""
await super().global_setup()
await self.async_client.create_rsa_key("livekvtestperfkey")

async def global_cleanup(self):
"""The global cleanup is run only once."""
await self.async_client.delete_key("livekvtestperfkey")
await self.async_client.purge_deleted_key("livekvtestperfkey")
await super().global_cleanup()

async def close(self):
"""This is run after cleanup."""
await self.async_client.close()
await self.async_credential.close()
await super().close()

def run_sync(self):
"""The synchronous perf test."""
self.client.get_key("livekvtestperfkey")

async def run_async(self):
"""The asynchronous perf test."""
await self.async_client.get_key("livekvtestperfkey")
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# azure-keyvault-secrets Performance Tests

In order to run the performance tests, the `azure-devtools` package must be installed. This is done as part of the
`dev_requirements` install. Start by creating a new virtual environment for your perf tests. This will need to be a
Python 3 environment, preferably >=3.7.

### Setup for test resources

The following environment variables will need to be set for the tests to access the live resources:

```
AZURE_TENANT_ID=<tenant ID of testing service principal>
AZURE_CLIENT_ID=<client ID of testing service principal>
AZURE_CLIENT_SECRET=<client secret of testing service principal>
AZURE_KEYVAULT_URL=<URL of the testing key vault>
```

### Setup for perf test runs

```cmd
(env) ~/azure-keyvault-secrets> pip install -r dev_requirements.txt
(env) ~/azure-keyvault-secrets> pip install -e .
```

## Test commands

Once `azure-devtools` is installed, you will have access to the `perfstress` command line tool, which will scan the
current module for runnable perf tests. Only a specific test can be run at a time (i.e. there is no "run all" feature).

```cmd
(env) ~/azure-keyvault-secrets> cd tests/perfstress_tests/
(env) ~/azure-keyvault-secrets/tests> perfstress
```
Using the `perfstress` command alone will list the available perf tests found.

### Common perf command line options
These options are available for all perf tests:
- `--duration=10` Number of seconds to run as many operations (the "run" function) as possible. Default is 10.
- `--iterations=1` Number of test iterations to run. Default is 1.
- `--parallel=1` Number of tests to run in parallel. Default is 1.
- `--warm-up=5` Number of seconds to spend warming up the connection before measuring begins. Default is 5.
- `--sync` Whether to run the tests in sync or async. Default is False (async).
- `--no-cleanup` Whether to keep newly created resources after test run. Default is False (resources will be deleted).

## Example command
```cmd
(env) ~/azure-keyvault-secrets/tests> perfstress GetSecretTest
```
Empty file.
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# ------------------------------------
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.
# ------------------------------------

from azure_devtools.perfstress_tests import PerfStressTest
from azure.identity import EnvironmentCredential
from azure.identity.aio import EnvironmentCredential as AsyncEnvironmentCredential
from azure.keyvault.secrets import SecretClient
from azure.keyvault.secrets.aio import SecretClient as AsyncSecretClient


class GetSecretTest(PerfStressTest):

def __init__(self, arguments):
super().__init__(arguments)

# Auth configuration
self.credential = EnvironmentCredential()
self.async_credential = AsyncEnvironmentCredential()

# Create clients
vault_url = self.get_from_env("AZURE_KEYVAULT_URL")
self.client = SecretClient(vault_url, self.credential)
self.async_client = AsyncSecretClient(vault_url, self.async_credential)

async def global_setup(self):
"""The global setup is run only once."""
await super().global_setup()
await self.async_client.set_secret("livekvtestperfsecret", "secret-value")

async def global_cleanup(self):
"""The global cleanup is run only once."""
await self.async_client.delete_secret("livekvtestperfsecret")
await self.async_client.purge_deleted_secret("livekvtestperfsecret")
await super().global_cleanup()

async def close(self):
"""This is run after cleanup."""
await self.async_client.close()
await self.async_credential.close()
await super().close()

def run_sync(self):
"""The synchronous perf test."""
self.client.get_secret("livekvtestperfsecret")

async def run_async(self):
"""The asynchronous perf test."""
await self.async_client.get_secret("livekvtestperfsecret")

0 comments on commit 1025c0b

Please sign in to comment.