azure-storage-queue installed but ImportError: Azure Storage Queues transport requires the azure-storage-queue library

[lucafaggianelli]

Checklist

• I have verified that the issue exists against the master branch of Celery.
• This has already been asked to the discussion group first.
• I have read the relevant section in the
  contribution guide
  on reporting bugs.
• I have checked the issues list
  for similar or identical bug reports.
• I have checked the pull requests list
  for existing proposed fixes.
• I have checked the commit log
  to find out if the bug was already fixed in the master branch.
• I have included all related issues and possible duplicate issues
  in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

• I have included the output of celery -A proj report in the issue.
  (if you are not able to do this, then at least specify the Celery
  version affected).
• I have verified that the issue exists against the master branch of Celery.
• I have included the contents of pip freeze in the issue.
• I have included all the versions of all the external dependencies required
  to reproduce this bug.

Optional Debugging Information

• I have tried reproducing the issue on more than one Python version
  and/or implementation.
• I have tried reproducing the issue on more than one message broker and/or
  result backend.
• I have tried reproducing the issue on more than one version of the message
  broker and/or result backend.
• I have tried reproducing the issue on more than one operating system.
• I have tried reproducing the issue on more than one workers pool.
• I have tried reproducing the issue with autoscaling, retries,
  ETA/Countdown & rate limits disabled.
• I have tried reproducing the issue after downgrading
  and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

• None

Possible Duplicates

• None

Environment & Settings

Celery version: 4.4.6 (cliffs)

celery report Output:

ImportError: Azure Storage Queues transport requires the azure-storage-queue library

Steps to Reproduce

Required Dependencies

• Minimal Python Version: N/A or Unknown
• Minimal Celery Version: N/A or Unknown
• Minimal Kombu Version: N/A or Unknown
• Minimal Broker Version: N/A or Unknown
• Minimal Result Backend Version: N/A or Unknown
• Minimal OS and/or Kernel Version: N/A or Unknown
• Minimal Broker Client Version: N/A or Unknown
• Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output:

amqp==2.6.0
astroid==2.4.2
attrs==19.3.0
azure-common==1.1.25
azure-core==1.8.1
azure-identity==1.4.0
azure-keyvault-secrets==4.2.0
azure-storage-common==2.1.0
azure-storage-queue==12.1.3
billiard==3.6.3.0
bnd-json-schemas==0.3.0
boto3==1.14.51
botocore==1.17.51
celery==4.4.6
certifi==2020.6.20
cffi==1.14.0
chardet==3.0.4
colorama==0.4.3
coverage==5.3
cryptography==3.1.1
dnspython==2.0.0
docutils==0.15.2
falcon==2.0.0
future==0.18.2
gevent==20.4.0
greenlet==0.4.16
gunicorn==20.0.4
idna==2.10
isodate==0.6.0
isort==4.3.21
jmespath==0.10.0
jsonschema==3.2.0
kombu==4.6.11
lazy-object-proxy==1.4.3
librabbitmq==2.0.0
mccabe==0.6.1
msal==1.5.0
msal-extensions==0.2.2
msrest==0.6.19
oauthlib==3.1.0
portalocker==1.7.1
pycparser==2.20
pycurl==7.43.0.5
PyJWT==1.7.1
pylint==2.5.3
pymongo==3.11.0
pyrsistent==0.16.0
python-dateutil==2.8.1
python-dotenv==0.14.0
python-falcon==0.1.5
pytz==2020.1
pywin32==228
requests==2.23.0
requests-oauthlib==1.3.0
s3transfer==0.3.3
six==1.15.0
tabulate==0.8.7
toml==0.10.1
urllib3==1.25.9
vine==1.3.0
waitress==1.4.4
wrapt==1.12.1

Other Dependencies

N/A

Minimally Reproducible Test Case

from celery import Celery

celery_config['broker_url'] = f"azurestoragequeues://:{AZ_QUEUE_KEY}@{AZ_QUEUE_NAME}/testqueue"

app = Celery('worker')
app.conf.update(**celery_config)

Expected Behavior

To work

Actual Behavior

I have this exception:

ImportError: Azure Storage Queues transport requires the azure-storage-queue library

even if azure library is installed. Taking a look at the source code of the Celery-Azure integration (https://docs.celeryproject.org/projects/kombu/en/stable/_modules/kombu/transport/azurestoragequeues.html) it seems that it tries to import:

from azure.storage.queue import QueueService

but the service has been renamed to QueueServiceClient in the latest version (v12) of azure-storage-queue library.
Downgrading azure-storage-queue to v2, celery works but it performs a couple of GET requests per second to check for new messages, that is, it seems to poll the queue.

[thedrow]

Thanks for the report. Feel free to help us add compatibility to the newer versions of azure-storage-queue.
PRs are always welcome.

[open-collective-bot]

Hey @lucafaggianelli 👋,
Thank you for opening an issue. We will get back to you as soon as we can.
Also, check out our Open Collective and consider backing us - every little helps!

We also offer priority support for our sponsors.
If you require immediate assistance please consider sponsoring us.