Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update TextAnalytics to enable live testing in sovereign clouds for multiple services #22461

Merged
merged 1 commit into from
Jan 18, 2022
Merged

Update TextAnalytics to enable live testing in sovereign clouds for multiple services #22461

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 26 additions & 8 deletions sdk/textanalytics/azure-ai-textanalytics/tests/test_analyze.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
from unittest import mock
from azure.core.exceptions import HttpResponseError, ClientAuthenticationError
from azure.core.credentials import AzureKeyCredential
from testcase import TextAnalyticsTest, TextAnalyticsPreparer
from testcase import TextAnalyticsTest, TextAnalyticsPreparer, is_public_cloud
from testcase import TextAnalyticsClientPreparer as _TextAnalyticsClientPreparer
from devtools_testutils import recorded_by_proxy, set_bodiless_matcher
from azure.ai.textanalytics import (
Expand Down Expand Up @@ -47,6 +47,17 @@
# pre-apply the client_cls positional argument so it needn't be explicitly passed below
TextAnalyticsClientPreparer = functools.partial(_TextAnalyticsClientPreparer, TextAnalyticsClient)

TextAnalyticsCustomPreparer = functools.partial(
TextAnalyticsPreparer,
textanalytics_custom_text_endpoint="https://fakeendpoint.cognitiveservices.azure.com",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are these variables moved from the testcase to the test files?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I'm not mistaken, they needed to be separated since these env vars don't exist in the UsGov/China cloud.

Copy link
Member Author

@v-xuto v-xuto Jan 13, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kristapratico @iscai-msft If I don't remove these variables, I'll get an error when rerun textanalytics weekly pipeline. Error message: azure_devtools.scenario_tests.preparers:preparers.py:82 Preparer failure when creating resource PowerShellPreparer for test <test_auth.TestAuth object at 0x111e26190>: 'TEXTANALYTICS_CUSTOM_TEXT_ENDPOINT'
For more details please check here.

For example test_auth.py::TestAuth::test_active_directory_auth, I have debugged this test locally. When the code is executed to devtools_testutils/powershell_preparer.py , it will get the corresponding values of these variables ( in TextAnalyticsPreparer) from the environment variables one by one, if not, an exception will be thrown. Because the values of these variables are not passed in UsGov/China cloud, the test will fail before the testcase actually runs.

So I moved variables from the testcase to the test files. Or do you have better suggestions?

textanalytics_custom_text_key="fakeZmFrZV9hY29jdW50X2tleQ==",
textanalytics_single_category_classify_project_name="single_category_classify_project_name",
textanalytics_single_category_classify_deployment_name="single_category_classify_deployment_name",
textanalytics_multi_category_classify_project_name="multi_category_classify_project_name",
textanalytics_multi_category_classify_deployment_name="multi_category_classify_deployment_name",
textanalytics_custom_entities_project_name="custom_entities_project_name",
textanalytics_custom_entities_deployment_name="custom_entities_deployment_name",
)

class TestAnalyze(TextAnalyticsTest):

Expand Down Expand Up @@ -627,7 +638,8 @@ def test_too_many_documents(self, client):
)
assert excinfo.value.status_code == 400

@TextAnalyticsPreparer()
@pytest.mark.skipif(not is_public_cloud(), reason='Usgov and China Cloud are not supported')
@TextAnalyticsCustomPreparer()
@recorded_by_proxy
def test_disable_service_logs(
self,
Expand Down Expand Up @@ -989,7 +1001,8 @@ def test_extract_summary_partial_results(self, client):
assert not document_results[1][0].is_error
assert isinstance(document_results[1][0], ExtractSummaryResult)

@TextAnalyticsPreparer()
@pytest.mark.skipif(not is_public_cloud(), reason='Usgov and China Cloud are not supported')
@TextAnalyticsCustomPreparer()
@recorded_by_proxy
def test_single_category_classify(
self,
Expand Down Expand Up @@ -1028,7 +1041,8 @@ def test_single_category_classify(
assert result.classification.category
assert result.classification.confidence_score

@TextAnalyticsPreparer()
@pytest.mark.skipif(not is_public_cloud(), reason='Usgov and China Cloud are not supported')
@TextAnalyticsCustomPreparer()
@recorded_by_proxy
def test_multi_category_classify(
self,
Expand Down Expand Up @@ -1068,7 +1082,8 @@ def test_multi_category_classify(
assert classification.category
assert classification.confidence_score

@TextAnalyticsPreparer()
@pytest.mark.skipif(not is_public_cloud(), reason='Usgov and China Cloud are not supported')
@TextAnalyticsCustomPreparer()
@recorded_by_proxy
def test_recognize_custom_entities(
self,
Expand Down Expand Up @@ -1216,7 +1231,8 @@ def test_analyze_continuation_token(self, client):

initial_poller.wait() # necessary so azure-devtools doesn't throw assertion error

@TextAnalyticsPreparer()
@pytest.mark.skipif(not is_public_cloud(), reason='Usgov and China Cloud are not supported')
@TextAnalyticsCustomPreparer()
def test_generic_action_error_no_target(
self,
textanalytics_custom_text_endpoint,
Expand Down Expand Up @@ -1276,7 +1292,8 @@ def test_generic_action_error_no_target(
).result())
assert e.value.message == "(InternalServerError) 1 out of 3 job tasks failed. Failed job tasks : v3.2-preview.2/custom/entities/general."

@TextAnalyticsPreparer()
@pytest.mark.skipif(not is_public_cloud(), reason='Usgov and China Cloud are not supported')
@TextAnalyticsCustomPreparer()
def test_action_errors_with_targets(
self,
textanalytics_custom_text_endpoint,
Expand Down Expand Up @@ -1366,7 +1383,8 @@ def test_action_errors_with_targets(
assert result.error.code == "InvalidRequest"
assert result.error.message == "Some error" + str(idx) # confirms correct doc error order

@TextAnalyticsPreparer()
@pytest.mark.skipif(not is_public_cloud(), reason='Usgov and China Cloud are not supported')
@TextAnalyticsCustomPreparer()
def test_action_job_failure(
self,
textanalytics_custom_text_endpoint,
Expand Down
35 changes: 27 additions & 8 deletions sdk/textanalytics/azure-ai-textanalytics/tests/test_analyze_async.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

from azure.core.exceptions import HttpResponseError, ClientAuthenticationError
from azure.core.credentials import AzureKeyCredential
from testcase import TextAnalyticsPreparer
from testcase import TextAnalyticsPreparer, is_public_cloud
from testcase import TextAnalyticsClientPreparer as _TextAnalyticsClientPreparer
from devtools_testutils import set_bodiless_matcher
from devtools_testutils.aio import recorded_by_proxy_async
Expand Down Expand Up @@ -50,6 +50,18 @@
# pre-apply the client_cls positional argument so it needn't be explicitly passed below
TextAnalyticsClientPreparer = functools.partial(_TextAnalyticsClientPreparer, TextAnalyticsClient)

TextAnalyticsCustomPreparer = functools.partial(
TextAnalyticsPreparer,
textanalytics_custom_text_endpoint="https://fakeendpoint.cognitiveservices.azure.com",
textanalytics_custom_text_key="fakeZmFrZV9hY29jdW50X2tleQ==",
textanalytics_single_category_classify_project_name="single_category_classify_project_name",
textanalytics_single_category_classify_deployment_name="single_category_classify_deployment_name",
textanalytics_multi_category_classify_project_name="multi_category_classify_project_name",
textanalytics_multi_category_classify_deployment_name="multi_category_classify_deployment_name",
textanalytics_custom_entities_project_name="custom_entities_project_name",
textanalytics_custom_entities_deployment_name="custom_entities_deployment_name",
)

def get_completed_future(result=None):
future = asyncio.Future()
future.set_result(result)
Expand Down Expand Up @@ -682,7 +694,8 @@ async def test_too_many_documents(self, client):
)).result()
assert excinfo.value.status_code == 400

@TextAnalyticsPreparer()
@pytest.mark.skipif(not is_public_cloud(), reason='Usgov and China Cloud are not supported')
@TextAnalyticsCustomPreparer()
@recorded_by_proxy_async
async def test_disable_service_logs(
self,
Expand Down Expand Up @@ -1058,7 +1071,8 @@ async def test_extract_summary_partial_results(self, client):
assert not document_results[1][0].is_error
assert isinstance(document_results[1][0], ExtractSummaryResult)

@TextAnalyticsPreparer()
@pytest.mark.skipif(not is_public_cloud(), reason='Usgov and China Cloud are not supported')
@TextAnalyticsCustomPreparer()
@recorded_by_proxy_async
async def test_single_category_classify(
self,
Expand Down Expand Up @@ -1100,7 +1114,8 @@ async def test_single_category_classify(
assert result.classification.category
assert result.classification.confidence_score

@TextAnalyticsPreparer()
@pytest.mark.skipif(not is_public_cloud(), reason='Usgov and China Cloud are not supported')
@TextAnalyticsCustomPreparer()
@recorded_by_proxy_async
async def test_multi_category_classify(
self,
Expand Down Expand Up @@ -1144,7 +1159,8 @@ async def test_multi_category_classify(
assert classification.category
assert classification.confidence_score

@TextAnalyticsPreparer()
@pytest.mark.skipif(not is_public_cloud(), reason='Usgov and China Cloud are not supported')
@TextAnalyticsCustomPreparer()
@recorded_by_proxy_async
async def test_recognize_custom_entities(
self,
Expand Down Expand Up @@ -1303,7 +1319,8 @@ async def test_analyze_continuation_token(self, client):

await initial_poller.wait() # necessary so azure-devtools doesn't throw assertion error

@TextAnalyticsPreparer()
@pytest.mark.skipif(not is_public_cloud(), reason='Usgov and China Cloud are not supported')
@TextAnalyticsCustomPreparer()
@recorded_by_proxy_async
async def test_generic_action_error_no_target(
self,
Expand Down Expand Up @@ -1367,7 +1384,8 @@ async def test_generic_action_error_no_target(
results.append(resp)
assert e.value.message == "(InternalServerError) 1 out of 3 job tasks failed. Failed job tasks : v3.2-preview.2/custom/entities/general."

@TextAnalyticsPreparer()
@pytest.mark.skipif(not is_public_cloud(), reason='Usgov and China Cloud are not supported')
@TextAnalyticsCustomPreparer()
@recorded_by_proxy_async
async def test_action_errors_with_targets(
self,
Expand Down Expand Up @@ -1463,7 +1481,8 @@ async def test_action_errors_with_targets(
assert result.error.code == "InvalidRequest"
assert result.error.message == "Some error" + str(idx) # confirms correct doc error order

@TextAnalyticsPreparer()
@pytest.mark.skipif(not is_public_cloud(), reason='Usgov and China Cloud are not supported')
@TextAnalyticsCustomPreparer()
@recorded_by_proxy_async
async def test_action_job_failure(
self,
Expand Down
6 changes: 5 additions & 1 deletion sdk/textanalytics/azure-ai-textanalytics/tests/test_auth.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,14 +7,18 @@
from testcase import TextAnalyticsTest, TextAnalyticsPreparer
from azure.ai.textanalytics import TextAnalyticsClient
from azure.core.credentials import AzureKeyCredential
import os


class TestAuth(TextAnalyticsTest):

@pytest.mark.live_test_only
@TextAnalyticsPreparer()
def test_active_directory_auth(self, textanalytics_test_endpoint):
token = self.get_credential(TextAnalyticsClient)
text_analytics = TextAnalyticsClient(textanalytics_test_endpoint, token)
text_analytics_endpoint_suffix = os.environ.get("TEXTANALYTICS_ENDPOINT_SUFFIX",".cognitiveservices.azure.com")
credential_scopes = ["https://{}/.default".format(text_analytics_endpoint_suffix[1:])]
text_analytics = TextAnalyticsClient(textanalytics_test_endpoint, token, credential_scopes=credential_scopes)

docs = [{"id": "1", "text": "I should take my cat to the veterinarian."},
{"id": "2", "text": "Este es un document escrito en Español."},
Expand Down
5 changes: 4 additions & 1 deletion sdk/textanalytics/azure-ai-textanalytics/tests/test_auth_async.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
from azure.core.credentials import AzureKeyCredential
from testcase import TextAnalyticsPreparer
from testcase import TextAnalyticsTest
import os


class TestAuth(TextAnalyticsTest):
Expand All @@ -16,7 +17,9 @@ class TestAuth(TextAnalyticsTest):
@TextAnalyticsPreparer()
async def test_active_directory_auth(self, textanalytics_test_endpoint):
token = self.get_credential(TextAnalyticsClient, is_async=True)
text_analytics = TextAnalyticsClient(textanalytics_test_endpoint, token)
text_analytics_endpoint_suffix = os.environ.get("TEXTANALYTICS_ENDPOINT_SUFFIX",".cognitiveservices.azure.com")
credential_scopes = ["https://{}/.default".format(text_analytics_endpoint_suffix[1:])]
text_analytics = TextAnalyticsClient(textanalytics_test_endpoint, token, credential_scopes=credential_scopes)

docs = [{"id": "1", "text": "I should take my cat to the veterinarian."},
{"id": "2", "text": "Este es un document escrito en Español."},
Expand Down
12 changes: 4 additions & 8 deletions sdk/textanalytics/azure-ai-textanalytics/tests/testcase.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,19 +22,15 @@
from devtools_testutils import PowerShellPreparer, AzureRecordedTestCase


def is_public_cloud():
return (".microsoftonline.com" in os.getenv('AZURE_AUTHORITY_HOST', ''))


TextAnalyticsPreparer = functools.partial(
PowerShellPreparer,
'textanalytics',
textanalytics_test_endpoint="https://fakeendpoint.cognitiveservices.azure.com",
textanalytics_test_api_key="fakeZmFrZV9hY29jdW50X2tleQ==",
textanalytics_custom_text_endpoint="https://fakeendpoint.cognitiveservices.azure.com",
textanalytics_custom_text_key="fakeZmFrZV9hY29jdW50X2tleQ==",
textanalytics_single_category_classify_project_name="single_category_classify_project_name",
textanalytics_single_category_classify_deployment_name="single_category_classify_deployment_name",
textanalytics_multi_category_classify_project_name="multi_category_classify_project_name",
textanalytics_multi_category_classify_deployment_name="multi_category_classify_deployment_name",
textanalytics_custom_entities_project_name="custom_entities_project_name",
textanalytics_custom_entities_deployment_name="custom_entities_deployment_name",
)


Expand Down
12 changes: 8 additions & 4 deletions sdk/textanalytics/test-resources.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,16 +42,16 @@
"description": "The application client secret used to run tests."
}
},
"textAnalyticsEndpointSuffix": {
"defaultValue": ".cognitiveservices.azure.com/",
"cognitiveServicesEndpointSuffix": {
"defaultValue": ".cognitiveservices.azure.com",
"type": "String"
},
}
},
"variables": {
"authorizationApiVersion": "2018-09-01-preview",
"textAnalyticsBaseName": "[concat('textanalytics', parameters('baseName'))]",
"cognitiveApiVersion": "2017-04-18",
"azureTextAnalyticsUrl": "[concat('https://', variables('textAnalyticsBaseName'), parameters('textAnalyticsEndpointSuffix'))]",
"azureTextAnalyticsUrl": "[concat('https://', variables('textAnalyticsBaseName'), parameters('cognitiveServicesEndpointSuffix'))]",
"cognitiveServiceUserRoleId": "[concat('/subscriptions/', subscription().subscriptionId, '/providers/Microsoft.Authorization/roleDefinitions/a97b65f3-24c7-4388-baec-2e87135dc908')]"
},
"resources": [
Expand Down Expand Up @@ -101,6 +101,10 @@
"TEXTANALYTICS_TEST_ENDPOINT": {
"type": "string",
"value": "[variables('azureTextAnalyticsUrl')]"
},
"TEXTANALYTICS_ENDPOINT_SUFFIX": {
"type": "string",
"value": "[parameters('cognitiveServicesEndpointSuffix')]"
}
}
}
5 changes: 3 additions & 2 deletions sdk/textanalytics/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,13 @@ stages:
SubscriptionConfigurations:
- $(sub-config-azure-cloud-test-resources)
- $(sub-config-text-analytics-azure-cloud-test-resources)
MatrixReplace:
- TestSamples=.*/true
UsGov:
SubscriptionConfiguration: $(sub-config-gov-test-resources)
China:
SubscriptionConfiguration: $(sub-config-cn-test-resources)
MatrixReplace:
- TestSamples=.*/true
SupportedClouds: 'Public,UsGov,China'
kristapratico marked this conversation as resolved.
Show resolved Hide resolved
EnvVars:
AZURE_SUBSCRIPTION_ID: $(TEXTANALYTICS_SUBSCRIPTION_ID)
AZURE_TENANT_ID: $(TEXTANALYTICS_TENANT_ID)
Expand Down