Retry after failure to download or failure to open files (#4609)

* add retry logic, tests when extracting tarfile fails

* fixed bug with not catching empty responses

* specify compression type

* WIP test

* more testing work

* fixed up unit test

* add changelog

* Add more comments!

* clarify why we do the json() check for None

automatic commit by git-black, original commits:
  d9cfeb1

CHANGELOG.md

@@ -26,7 +26,7 @@ Contributors:
 - Projects created using `dbt init` now have the correct `seeds` directory created (instead of `data`) ([#4588](https://github.com/dbt-labs/dbt-core/issues/4588), [#4599](https://github.com/dbt-labs/dbt-core/pull/4589))
 - Don't require a profile for dbt deps and clean commands ([#4554](https://github.com/dbt-labs/dbt-core/issues/4554), [#4610](https://github.com/dbt-labs/dbt-core/pull/4610))
 - Select modified.body works correctly when new model added([#4570](https://github.com/dbt-labs/dbt-core/issues/4570), [#4631](https://github.com/dbt-labs/dbt-core/pull/4631))
-- Fix bug in retry logic for bad response from hub and when there is a bad git tarball download. ([#4577](https://github.com/dbt-labs/dbt-core/issues/4577), [#4579](https://github.com/dbt-labs/dbt-core/issues/4579), [#4609](https://github.com/dbt-labs/dbt-core/pull/4609)) 
+- Fix bug in retry logic for bad response from hub and when there is a bad git tarball download. ([#4577](https://github.com/dbt-labs/dbt-core/issues/4577), [#4579](https://github.com/dbt-labs/dbt-core/issues/4579), [#4609](https://github.com/dbt-labs/dbt-core/pull/4609))
 - Restore previous log level (DEBUG) when a test depends on a disabled resource. Still WARN if the resource is missing ([#4594](https://github.com/dbt-labs/dbt-core/issues/4594), [#4647](https://github.com/dbt-labs/dbt-core/pull/4647))
 - Add project name validation to `dbt init` ([#4490](https://github.com/dbt-labs/dbt-core/issues/4490),[#4536](https://github.com/dbt-labs/dbt-core/pull/4536))
 - Support click versions in the v7.x series ([#4681](https://github.com/dbt-labs/dbt-core/pull/4681))

core/dbt/clients/system.py

@@ -469,7 +469,7 @@ def rename(from_path: str, to_path: str, force: bool = False) -> None:
 def untar_package(tar_path: str, dest_dir: str, rename_to: Optional[str] = None) -> None:
     tar_path = convert_path(tar_path)
     tar_dir_name = None
-    with tarfile.open(tar_path, 'r:gz') as tarball:
+    with tarfile.open(tar_path, "r:gz") as tarball:
         tarball.extractall(dest_dir)
         tar_dir_name = os.path.commonprefix(tarball.getnames())
     if rename_to:

core/dbt/deps/registry.py

@@ -74,11 +74,7 @@ def install(self, project, renderer):
         package_name = self.get_project_name(project, renderer)
 
         download_untar_fn = functools.partial(
-            self.download_and_untar,
-            download_url,
-            tar_path,
-            deps_path,
-            package_name
+            self.download_and_untar, download_url, tar_path, deps_path, package_name
         )
         connection_exception_retry(download_untar_fn, 5)
 

core/dbt/exceptions.py

@@ -786,6 +786,7 @@ class ConnectionException(Exception):
     There was a problem with the connection that returned a bad response,
     timed out, or resulted in a file that is corrupt.
     """
+
     pass