Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Task factory "task per file" fails with source "containerUrl" #122

Open
tojozefi opened this issue Apr 29, 2020 · 2 comments
Open

Task factory "task per file" fails with source "containerUrl" #122

tojozefi opened this issue Apr 29, 2020 · 2 comments

Comments

@tojozefi
Copy link

Using task factory "task per file" with source defined as container URL fails with error:
AttributeError: 'SplitResult' object has no attribute 'pathname'

Here is extract from the job template:
{
(...)
"parameters":{
(...)
"inputContainerUrl":{
"type":"string",
"defaultValue":"https://tojobatch.blob.core.windows.net/fgrp-sequencer-input?st=2020-04-29T20%3A20%3A06Z&se=2020-05-30T20%3A20%3A00Z&sp=rl&sv=2018-03-28&sr=c&sig=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
},
(...)
"job":{
(...)
"taskFactory":{
"type":"taskPerFile",
"source":{
"containerUrl": "[parameters('inputContainerUrl')]"
},
(...)

The same template works ok when using source with "fileGroup" parameter.

Software level:
$ az --version
azure-cli 2.4.0 *
command-modules-nspkg 2.0.3
core 2.4.0 *
nspkg 3.0.4
telemetry 1.0.4
Extensions:
image-copy-extension 0.0.9
azure-batch-cli-extensions 5.0.1

@bgklein
Copy link
Contributor

bgklein commented May 5, 2020

I believe this section of code was never actually functional:

if resource_file.source.container_url:
# Input data storage in arbitrary container
uri = urlsplit(resource_file.source.container_url)
container = uri.pathname.split('/')[1]
blobs = self.list_container_contents(resource_file.source, container, storage_client)
return convert_blobs_to_resource_files(blobs, resource_file)
(or at least not for the last three years). According to urllib.parse.urlsplit documentation for both py2 and py3 and manually executing these 5 lines on your url it looks to be path instead of pathname.

I'll look to update this in our next version as it really should be using our native support for containers now anyways rather than manually listing the contents (https://docs.microsoft.com/en-us/python/api/azure-batch/azure.batch.models.resourcefile?view=azure-python).

@tojozefi
Copy link
Author

tojozefi commented Sep 9, 2020

Any plans on solving this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants