azure storage blob download: ConnectionResetError(104, 'Connection reset by peer')

[fengzhuye]

Hi

I installed the azure-storage-blob python sdk(12.8.1) from pip.
And setup a cron job to read all the blob files from the container, however, i find many error message from the log:

Unable to stream download: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

Why the connection will be reset by azure?
Any feedback will be appreciated.

Thanks
Wei

---

文档详细信息

⚠ 请勿编辑此部分。 docs.microsoft.com ➟ GitHub 问题链接必须具有此部分。

• ID: f60c47f1-e4d8-f68e-5f9c-460c4fdc6da8
• Version Independent ID: 172e9f9b-25fb-2121-cb72-d383cc21763a
• Content: Azure Storage Blobs client library for Python
• Content Source: docs-ref-services/latest/storage-blob-readme.md
• Service: storagestorage
• Product: azure
• Technology: azure
• GitHub Login: @maggiepint
• Microsoft Alias: magpint

[xiangyan99]

Thanks for the feedback, we’ll investigate asap.

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

Issue Details

---

Hi

I installed the azure-storage-blob python sdk(12.8.1) from pip.
And setup a cron job to read all the blob files from the container, however, i find many error message from the log:

Unable to stream download: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))

Why the connection will be reset by azure?
Any feedback will be appreciated.

Thanks
Wei

---

文档详细信息

⚠ 请勿编辑此部分。 docs.microsoft.com ➟ GitHub 问题链接必须具有此部分。

• ID: f60c47f1-e4d8-f68e-5f9c-460c4fdc6da8
• Version Independent ID: 172e9f9b-25fb-2121-cb72-d383cc21763a
• Content: Azure Storage Blobs client library for Python
• Content Source: docs-ref-services/latest/storage-blob-readme.md
• Service: storagestorage
• Product: azure
• Technology: azure
• GitHub Login: @maggiepint
• Microsoft Alias: magpint

Author:	fengzhuye
Assignees:	-
Labels:	Client, Service Attention, Storage, bug, customer-reported
Milestone:	-

[xiafu-msft]

Hi @fengzhuye

Thanks for reporting this. Is there any extra info in the stacktrace? Can you paste them here?
Also, did you check if the blob was downloaded successfully finally, ideally we will auto retry it if the error is caught properly. Can you tell if the requests are retired based on the stacktrace?

Thanks for helping on our investigation!

[xiafu-msft]

Hi @fengzhuye

Can you check which version of azure-storage-blob sdk you are using?
if it's not >=12.8.1 can you upgrade to that and see if that can help?

[fengzhuye]

Hi @xiafu-msft

Thanks for the reply. I am currently using version 12.8.1

Only the log message shows up, there is no stacktrace or other info.

It seems the exception is already handled by the sdk. I am not aware of this error in my program.
However, it seems the blob is downloaded successfully in this situation. Not sure whether the retry logic is triggerd not.

[xiafu-msft]

Hi @fengzhuye

Sounds like this is not a bug, storage retry 3 times on this error, the sdk is working as expected if your blob is downloaded successfully.

Let me know if you have any concern!

[fengzhuye]

Hi @xiafu-msft

It's that possible "ConnectionResetError(104, 'Connection reset by peer'))" will happens 3 times?
I want to know why this error happens, or what can I do to avoid this kind of error?

Sorry about the bothering.

[jaymegordo]

I also get this error frequently, ~2 times per day, while running code on a 15min interval on an azure functions app. The error never occurs while testing locally though. Is there a way to increase the retries?

[amishra-dev]

@tasherif-msft we should have a sample about how to setup retry policy and modify it

[gmotasilva]

Is there any update on that? Any workaround? I'm having this errors intermittently.

[amishra-dev]

@jalauzon-msft Jacob was working on this. Jacob can you please post an update here?

[jalauzon-msft]

Hi all, providing an update to everyone on the thread. @fengzhuye @jaymegordo @gmotasilva

Connection Reset errors come from the service and there is not much that can be done to eliminate them from the client-side. The best that can be done from the client is to have retries when this error occurs. The SDK will automatically retry these errors 3 times with an exponential backoff policy.

If you are seeing these errors, but retries are succeeding, then there is not much more that can be done from the client side.

If, however, your operations are failing with this error after the default 3 retries there are a couple of options.

• You can open a support ticket with your Storage account to have the service team investigate the connection reset errors.
• You can try increasing the number of retries by providing your own custom retry policy. This can be achieved with by providing a retry policy when creating any client (blob, datalake, file share, queue).

from azure.storage.blob import ExponentialRetry, BlobServiceClient

retry_policy = ExponentialRetry(
    retry_total=5
)

service_client = BlobServiceClient(
    'https://myaccount.blob.core.windows.net',
    retry_policy=retry_policy
)

The default value for retry_total is 3 so you could try 5, for example. There are also other parameters such as initial_backoff and increment_base that can be tweaked to change the time in between retries.

We are also currently investigating why this issue seems to be happening more often in our Python SDK, when compared to our other languages. From this investigation, we may end up tweaking the default retry values or something similar. We also plan to start an internal discussion with the service team on this issue.

[jalauzon-msft]

Hi all, while investigating #22784 it seems there was, at one point, an issue where blob download was not properly retrying connection reset errors. I think this was addressed in a later release of azure-storage-blob or (more like) azure-core so updating to latest may help with error in blob download specifically.

[ghost]

Hi @fengzhuye. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

[cheryl-catalog]

/unresolved
I am seeing this error as well:

2022-05-27T21:51:42+0000 next():186 Unable to stream download: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
2022-05-27T21:51:42+0000 wrapped():79 Download stream interrupted.

pip3 freeze | grep azure
azure-common==1.1.28
azure-core==1.24.0
azure-storage-blob==12.12.0
azure-storage-common==2.1.0
azure-storage-file==2.1.0
azure-storage-file-share==12.8.0

I will try the suggested tweaks above and see how it goes. I just started see this error this week

[ghost]

Hi cheryl-catalog, only the original author of the issue can ask that it be unresolved. Please open a new issue with your scenario and details if you would like to discuss this topic with the team.

[cheryl-catalog]

/unresolve

[ghost]

Hi cheryl-catalog, only the original author of the issue can ask that it be unresolved. Please open a new issue with your scenario and details if you would like to discuss this topic with the team.

[ghost]

Hi @fengzhuye, since you haven’t asked that we “/unresolve” the issue, we’ll close this out. If you believe further discussion is needed, please add a comment “/unresolve” to reopen the issue.