Skip to content

Commit

Permalink
feat: Add "transfer_manager" module for concurrent uploads and downlo…
Browse files Browse the repository at this point in the history
…ads, as a preview feature (#943)

* checkpoint before design doc impl

* checkpoint

* more tests

* code and tests for transfer manager complete

* proactively close temp files when finished reading

* respond to comments; destroy tmp files as they are consumed

* Add system tests, docstrings, address feedback

* Respond to review comments

* verify md5 hash of downloaded file in test

* lint

* default empty strings for root arguments

* fix bug with blob constructor

* add warning about files not being deleted if their downloads fail

* docs: Add samples to multithread branch (#918)

* add samples, tests pending

* add snippet tests

* snippet and snippets_test.py linting

* snippets; recursive directory creation; rename some params

* Add directory upload snippet

* fix: remove chunked downloads; change max_workers to threads

* update snippets to add thread info

* fix snippets test issue due to change in dependency

* snippet nomenclature

* fix samples for real this time
  • Loading branch information
andrewsg authored Dec 6, 2022
1 parent 1ca0233 commit 9998a5e
Show file tree
Hide file tree
Showing 7 changed files with 1,210 additions and 19 deletions.
1 change: 1 addition & 0 deletions google/cloud/storage/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""Constants used across google.cloud.storage modules."""

# Storage classes
Expand Down
6 changes: 4 additions & 2 deletions google/cloud/storage/fileio.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.

"""Support for file-like I/O."""
"""Module for file-like access of blobs, usually invoked via Blob.open()."""

import io
import warnings
Expand Down Expand Up @@ -101,10 +101,12 @@ class BlobReader(io.BufferedIOBase):
- ``if_metageneration_match``
- ``if_metageneration_not_match``
- ``timeout``
Note that download_kwargs are also applied to blob.reload(), if a reload
is needed during seek().
"""

def __init__(self, blob, chunk_size=None, retry=DEFAULT_RETRY, **download_kwargs):
"""docstring note that download_kwargs also used for reload()"""
for kwarg in download_kwargs:
if kwarg not in VALID_DOWNLOAD_KWARGS:
raise ValueError(
Expand Down
Loading

0 comments on commit 9998a5e

Please sign in to comment.