Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DefaultAzureCredential not working with Managed Identity (aka "Service Principal") #20628

Closed
gro1m opened this issue Dec 6, 2021 · 2 comments
Closed

DefaultAzureCredential not working with Managed Identity (aka "Service Principal") #20628

gro1m opened this issue Dec 6, 2021 · 2 comments
Assignees
@jiasli
Labels
Account az login/account customer-reported Issues that are reported by GitHub users external to the Azure organization. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that
Milestone
Dec 2021 (2022-01...

Comments

@gro1m
Copy link

gro1m commented Dec 6, 2021

az feedback auto-generates most of the information requested below, as of CLI version 2.0.62

Describe the bug
DefaultAzureCredential cannot login with credentials of managed identity after logging it in.

To Reproduce

COMMANDS:

az login --identity -u "<managed identity attached to machine learning compute cluster>"
dvc pull #needs DefaultAzureCredential class

ERROR MESSAGE:

2021-12-06 17:14:49,273 ERROR: unexpected error - DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
	AzureCliCredential: strptime() argument 1 must be str, not None
To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
------------------------------------------------------------
Traceback (most recent call last):
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/dvc/command/base.py", line 45, in do_run
    return self.run()
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/dvc/command/data_sync.py", line 30, in run
    stats = self.repo.pull(
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/dvc/repo/__init__.py", line 50, in wrapper
    return f(repo, *args, **kwargs)
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/dvc/repo/pull.py", line 29, in pull
    processed_files_count = self.fetch(
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/dvc/repo/__init__.py", line 50, in wrapper
    return f(repo, *args, **kwargs)
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/dvc/repo/fetch.py", line 67, in fetch
    d, f = _fetch(
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/dvc/repo/fetch.py", line 87, in _fetch
    downloaded += repo.cloud.pull(obj_ids, **kwargs)
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/dvc/data_cloud.py", line 114, in pull
    return transfer(
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/dvc/objects/transfer.py", line 153, in transfer
    status = compare_status(src, dest, obj_ids, check_deleted=False, **kwargs)
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/dvc/objects/status.py", line 166, in compare_status
    src_exists, src_missing = status(
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/dvc/objects/status.py", line 132, in status
    odb.hashes_exist(hashes, name=str(odb.path_info), **kwargs)
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/dvc/objects/db/base.py", line 472, in hashes_exist
    remote_size, remote_hashes = self._estimate_remote_size(hashes, name)
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/dvc/objects/db/base.py", line 274, in _estimate_remote_size
    remote_hashes = set(hashes)
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/dvc/objects/db/base.py", line 228, in _hashes_with_limit
    for hash_ in self.list_hashes(prefix, progress_callback):
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/dvc/objects/db/base.py", line 218, in list_hashes
    for path in self._list_paths(prefix, progress_callback):
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/dvc/objects/db/base.py", line 198, in _list_paths
    for file_info in self.fs.walk_files(path_info, prefix=prefix):
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/dvc/fs/fsspec_wrapper.py", line 181, in walk_files
    for file in self.find(path_info, **kwargs):
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/dvc/fs/fsspec_wrapper.py", line 276, in find
    files = self.fs.find(
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/adlfs/spec.py", line 965, in find
    return sync(
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/fsspec/asyn.py", line 71, in sync
    raise return_result
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/adlfs/spec.py", line 999, in _find
    infos = await self._details([b async for b in blobs])
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/adlfs/spec.py", line 999, in <listcomp>
    infos = await self._details([b async for b in blobs])
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/azure/core/async_paging.py", line 154, in __anext__
    return await self.__anext__()
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/azure/core/async_paging.py", line 157, in __anext__
    self._page = await self._page_iterator.__anext__()
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/azure/core/async_paging.py", line 99, in __anext__
    self._response = await self._get_next(self.continuation_token)
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/azure/storage/blob/aio/_list_blobs_helper.py", line 78, in _get_next_cb
    process_storage_error(error)
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/azure/storage/blob/_shared/response_handlers.py", line 89, in process_storage_error
    raise storage_error
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/azure/storage/blob/aio/_list_blobs_helper.py", line 71, in _get_next_cb
    return await self._command(
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/azure/storage/blob/_generated/aio/operations/_container_operations.py", line 1456, in list_blob_flat_segment
    pipeline_response = await self._client._pipeline.run(request, stream=False, **kwargs)
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/azure/core/pipeline/_base_async.py", line 215, in run
    return await first_node.send(pipeline_request)
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  [Previous line repeated 3 more times]
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/azure/core/pipeline/_base_async.py", line 81, in send
    await _await_result(self._policy.on_request, request)
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/azure/core/pipeline/_tools_async.py", line 32, in await_result
    return await result  # type: ignore
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/azure/core/pipeline/policies/_authentication_async.py", line 37, in on_request
    self._token = await self._credential.get_token(*self._scopes)  # type: ignore
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/azure/identity/aio/_credentials/default.py", line 143, in get_token
    return await super().get_token(*scopes, **kwargs)
  File "/azureml-envs/azureml_634aebf1e3b2ee06023cf87c0fb077a8/lib/python3.8/site-packages/azure/identity/aio/_credentials/chained.py", line 84, in get_token
    raise ClientAuthenticationError(message=message)
azure.core.exceptions.ClientAuthenticationError: DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
	AzureCliCredential: strptime() argument 1 must be str, not None
To mitigate this issue, please refer to the troubleshooting guidelines here at https://aka.ms/azsdk/python/identity/defaultazurecredential/troubleshoot.
------------------------------------------------------------
2021-12-06 17:14:49,354 DEBUG: Version info for developers:
DVC version: 2.8.3 (conda)
---------------------------------
Platform: Python 3.8.12 on Linux-5.0.0-1036-azure-x86_64-with-glibc2.17
Supports:
	azure (adlfs = 2021.10.0, knack = 0.6.3, azure-identity = 1.7.1),
	webhdfs (fsspec = 2021.10.1),
	http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
	https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6)
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: azure
Workspace directory: ext4 on /dev/sdb1
Repo: dvc, git

Expected behavior
Successful authentication

Environment summary
Commands are executed on Azure Machine Learning Compute Cluster.

az version: 2.30.0 (LATEST)
dvc: 2.8.3
Operating System: Ubuntu Bionic (18.04)
Image: "mcr.microsoft.com/azureml/openmpi3.1.2-ubuntu18.04"
Environment:

name: "<envname>"
dependencies:
  - python=3.8.*
  - pandas=1.3.*
  - pyaml=20.4.*
  - ipykernel=6.4.*
  - requests=2.26.*
  - pylint=2.11.*
  - conda-forge::papermill=2.3.*
  - conda-forge::scikit-learn=1.0.*
  - conda-forge::dvc-azure=2.8.*
  - conda-forge::aiohttp=3.8.*
  - conda-forge::cchardet=2.1.*

Additional context
@ghost ghost added needs-triage This is a new issue that needs to be triaged to the appropriate team. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that customer-reported Issues that are reported by GitHub users external to the Azure organization. labels Dec 6, 2021
@gro1m gro1m mentioned this issue Dec 6, 2021
Closed
@jiasli
Copy link
Member

jiasli commented Dec 7, 2021

This issue is fixed by #20219 in 2.31.0. Please update to the latest Azure CLI.

@jiasli jiasli closed this as completed Dec 7, 2021
@jiasli jiasli added this to the Dec 2021 (2022-01-04) milestone Dec 7, 2021
@jiasli jiasli self-assigned this Dec 7, 2021
@jiasli jiasli added the Account az login/account label Dec 7, 2021
@ghost ghost removed the needs-triage This is a new issue that needs to be triaged to the appropriate team. label Dec 7, 2021
@gro1m
Copy link
Author

gro1m commented Dec 7, 2021

Thanks @jiasli, it works now 👍

@gro1m gro1m mentioned this issue Dec 13, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jiasli jiasli

Labels
Account az login/account customer-reported Issues that are reported by GitHub users external to the Azure organization. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that
Projects
None yet
Milestone
Dec 2021 (2022-01-04)
Development

No branches or pull requests
2 participants
@jiasli @gro1m