[Text Analytics] "Invalid URL '$(js-textanalytics-test-service-endpoint)/text/analytics/v3.2-preview.2/analyze': No schema supplied" in Public, USGOV and China clouds

[v-xuto]

We are running live Tests against other clouds like US Gov and Azure China Cloud. The goal is to check whether new azure sdk package work with other clouds or not.

Error Description:
When running the test test_analyze.py::TestAnalyze::test_disable_service_logs in weekly pipeline, it runs failed and the error message is shown as following. In addition, I ran this test in the local and nightly pipelines, and the result of both is pass. For more details on this issue, please check here:

Expected Behavior:
This service request is a valid url.
Maybe we need to confirm whether the value of the environment variable js-textanalytics-test-service-endpoint (textanalytics/tests.yml) in weekly pipeline is correct and exists.

@benbp, @jameszliao-msft, @lmazuel, @meeraharidasa , @mayurid, @abhahn for notification.

[benbp]

These appear to be references to a static test resource via devops variables. Any time these are here it means that the tests will not work by default for sovereign clouds, and the static test resource needs to either be replicated across the sovereign clouds by the owning team OR updated to be provisioned dynamically. FYI @kristapratico

The reason you are seeing the current error (where it fails to expand the variable) is because this variable group has not been authorized for use by the weekly pipeline (but doing so won't fix the underlying issue).

[v-xuto]

@benbp So how are we currently going to fundamentally solve this problem? Do you have any idea? Or shall we first authorize this variable group to the weekly pipeline for sovereign testing?

[kristapratico]

These vars do refer to static test resources which are only available in the Public cloud (service constraint) - so provisioning dynamically will not quite work yet for sovereign clouds. I believe you could authorize the weekly pipeline to use the variable group since it will just use an endpoint/key combo from the Public cloud vars for these specific tests. Otherwise we can skip these tests in sovereign clouds for now.

[benbp]

@kristapratico I'd rather not make cross-cloud service requests if we can avoid it (from a test coverage/isolation standpoint). Would you be able to help copy over the static resources to the UsGov and China clouds? I can help with access.

[kristapratico]

@kristapratico I'd rather not make cross-cloud service requests if we can avoid it (from a test coverage/isolation standpoint). Would you be able to help copy over the static resources to the UsGov and China clouds? I can help with access.

@benbp That's fair. The issue is that these static resources pertain to features of the service that only are supported in westus2 and westeurope at the moment. The particular endpoint/key passed with env vars has trained and deployed models under the account in the Public cloud which we test from. We're unable to create these models in different clouds until the service supports it.

[benbp]

@kristapratico ah ok, in that case I'd say we should skip these tests in sovereign clouds until the feature set is supported.

[v-xuto]

@kristapratico @benbp Thanks for your help. I will skip these tests related to variable group (like js-single-category-classify-project-name) in UsGov and China clouds.

1. Please help authorize the weekly pipeline to use this variable group in textanalytics/tests.yml.
2. I will skip the following tests in UsGov and China, please help confirm whether it is correct or missing.

• test_analyze.py::TestAnalyze::test_disable_service_logs
• test_analyze.py::TestAnalyze::test_single_category_classify
• test_analyze.py::TestAnalyze::test_multi_category_classify
• test_analyze.py::TestAnalyze::test_recognize_custom_entities
• test_analyze.py::TestAnalyze::test_generic_action_error_no_target
• test_analyze.py::TestAnalyze::test_action_errors_with_targets
• test_analyze.py::TestAnalyze::test_action_job_failure
• test_analyze_async.py::TestAnalyzeAsync::test_disable_service_logs
• test_analyze_async.py::TestAnalyzeAsync::test_single_category_classify
• test_analyze_async.py::TestAnalyzeAsync::test_multi_category_classify
• test_analyze_async.py::TestAnalyzeAsync::test_recognize_custom_entities
• test_analyze_async.py::TestAnalyzeAsync::test_generic_action_error_no_target
• test_analyze_async.py::TestAnalyzeAsync::test_action_errors_with_targets
• test_analyze_async.py::TestAnalyzeAsync::test_action_job_failure

[kristapratico]

@v-xuto it might be best to skip on the absence of textanalytics_custom_text_endpoint since the env var associated is used in all the tests that will need to be skipped.

1. Like Ben said, we shouldn't need to authorize the variable group for this pipeline since we don't want to mix cross-cloud requests. Let me know if you're still running into issues with this, though.
2. That set of tests looks correct to me.

[benbp]

@kristapratico because the variables are still referenced in tests.yml, devops will try to download them hence the access error.

I recommend we build out a subscription configuration overlay for text analytics and migrate these values out of the variable group, that way we know the values won't be defined at all for sovereign tests and the test skips can handle that appropriately. I can take this on.

[kristapratico]

@benbp okay that sounds good! let me know how I can help

[benbp]

@kristapratico @v-xuto I have a PR up at #22376 which updates the yaml to support sovereign cloud testing and removes the dependency on the current variable group. Once that's merged in I can remove the old variable group reference in the pipeline as well.

@kristapratico if you can double check the contents of the config where I added the env variables and add anyone else relevant to the access policies + as vault owners that would be great. (Docs on the schema and how it gets set up are here).

Vault: https://ms.portal.azure.com/#@microsoft.onmicrosoft.com/resource/subscriptions/2cd617ea-1866-46b1-90e3-fffb087ebf9b/resourceGroups/LiveTestSecrets/providers/Microsoft.KeyVault/vaults/TestSecrets-textanalytic/overview

Or:

az keyvault secret show --vault-name testsecrets-textanalytic -n sub-config-text-analytics-azure-cloud-test-resources --query 'value'

P.S. Sadly I had to name the keyvault TestSecrets-TextAnalytic because the 24 character vault name limit made it to long for "Analytics" plural...

[kristapratico]

@benbp thanks for the help! I approved the PR.

[benbp]

Ok the subscription configuration for public and the tests.yml config update has been made. I authorized the weekly pipeline to be able to load it for the public stage, and the env variables should not be defined for sovereign environments now.

[v-xuto]

@benbp @kristapratico Maybe we need add envs AZURE_TEXT_ANALYTICS_ENDPOINT and AZURE_TEXT_ANALYTICS_KEY to (textanalytics/tests.yml). Because, I rerun the weekly pipeline, got an error in section Test Samples on UsGov and China clouds. For more details please check here.
sample_analyze_healthcare_entities.py error message: KeyError: 'AZURE_TEXT_ANALYTICS_ENDPOINT'.

Besides, can I update the code to the following to skip the samples (like sample_recognize_custom_entities.py) that uses the parameter textanalytics_custom_text_endpoint? Or do you have any better suggestion to skip these samples in UsGov and China?

def is_public_cloud():
    return (".microsoftonline.com" in os.getenv('AZURE_AUTHORITY_HOST', ''))

if __name__ == "__main__":
    if is_public_cloud():
        sample_recognize_custom_entities()

The relevant samples are: sample_multi_category_classify.py(asnyc), sample_recognize_custom_entities.py(asnyc), sample_single_category_classify.py(asnyc)

[kristapratico]

Hi @v-xuto, you can skip samples for UsGov and China. Just move the

MatrixReplace:
  - TestSamples=.*/true

under the PublicCloud config. For example: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/translation/tests.yml#L12-L13

This will skip samples for those clouds and we won't have to add any code to the sample files themselves.

[v-xuto]

This issue has been fixed and I will close it.