There should be a built-in way to list prefixes (directories) in a bucket

[lrowe]

The REST API provides a way to list prefixes (directories) in a bucket but this does not seem to be supported by the client library.

According to #192 the iterator returned by list_blobs has a prefixes field which is filled in as you iterate over the blob. This should be better documented, since it is only mentioned in passing:

include_trailing_delimiter (boolean) – (Optional) If true, objects that end in exactly one instance of delimiter will have their metadata included in items in addition to prefixes.

https://googleapis.dev/python/storage/latest/client.html#google.cloud.storage.client.Client.list_blobs

As well as a method to list prefixes alone, it would be helpful to have an iterator that returned both prefixes and objects in order for produce ordered listings.

Workaround found on Stack Overflow. https://stackoverflow.com/a/59008580

def list_prefixes(client, bucket_name, prefix, delimiter):
    # Adapted from https://stackoverflow.com/a/59008580
    from google.api_core import page_iterator
    return page_iterator.HTTPIterator(
        client=client,
        api_request=client._connection.api_request,
        path=f"/b/{bucket_name}/o",
        items_key="prefixes",
        item_to_value=lambda iterator, item: item,
        extra_params={
            "projection": "noAcl",
            "prefix": prefix,
            "delimiter": delimiter,
        },
    )

[frankyn]

Hi @lrowe,
Thanks for filing the issue.

Would it help to document an example instead pydocs? Here's an example:

https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/storage/cloud-client/storage_list_files_with_prefix.py#L49-L65

[lrowe]

@frankyn An example would be helpful too, but I think the prefixes property should be included the API docs since they are expected to be exhaustive.

Perhaps the :returns: section of list_blobs could be expanded to mention the special prefixes property of the iterator, see list_blobs docstring:

        :rtype: :class:`~google.api_core.page_iterator.Iterator`
        :returns: Iterator of all :class:`~google.cloud.storage.blob.Blob`
                  in this bucket matching the arguments.

[lrowe]

I do worry though that the current list_blobs Iterator.prefixes set might be problematic when there are a very large number of prefixes since it would grow unbounded.

[frankyn]

Thanks for clarifying @lrowe, how many prefixes are you expecting right now? We'd need to run tests to see how the library behaves.

@HemangChothani could you update your PR to provide better documentation instead for now?

[lrowe]

I ran into this while building a Cloud Function storage trigger to generate directory listings in a bucket exposed as a static website containing output from programmatically generated analysis runs. Some of these 'directories' have ~50,000 subdirectories and I can imagine we may have cases in the future with an order of magnitude more and I will need to split the listing over multiple pages in order to keep it usable.

[frankyn]

Python documentation for list_blobs() needs to be updated to clarify that there's a property for prefixes on the iterator returned. Right now documentation only states that an Iterator type is returned.

python-storage/google/cloud/storage/client.py

Lines 832 to 834 in fe87755

	Returns:
	Iterator of all :class:`~google.cloud.storage.blob.Blob`
	in this bucket matching the arguments.

[amirbtb]

Hi,

I believe there is an issue with the prefixes property of the iterator.
the set returned is empty right after I call the client

result = client.list_blobs(
    bucket_or_name='my-bucket',
    prefix='',
    delimiter='/'
)

result.prefixes
# set()

But when I loop over the result iterator I can access the prefixes

result = client.list_blobs(
    bucket_or_name='my-bucket',
    prefix='',
    delimiter='/'
)

for element in result.prefixes:
    pass

result.prefixes
# {'folder1/', 'folder2/'}

It would be great to be have a built-in way to list prefixes directly without listing files, since it can be expensive in time and compute.

Thank you in advance (and Merry Christmas 🎄)