[Bug] dbt deps should keep retrying if registry response is None

[crystalro0]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Running dbt deps in a nightly job failed after retrying downloading the packages. The Max retries is listed as 5 and was currently at 0.

Expected Behavior

The download is retried until the max retries is reached or the package is downloaded successfully.

Steps To Reproduce

This cannot be reproduced on demand. It is an intermittent failure case for dbt deps

Relevant log output

Console:
2021-10-28 09:23:31.847397 (MainThread): Making package registry request: GET https://hub.getdbt.com/api/v1/dbt-labs/logging.json
2021-10-28 09:23:31.861303 (MainThread): Retrying external call. Attempt: 0 Max attempts: 5
2021-10-28 09:23:32.862660 (MainThread): Making package registry request: GET https://hub.getdbt.com/api/v1/dbt-labs/logging.json
2021-10-28 09:23:32.879183 (MainThread): Response from registry: GET https://hub.getdbt.com/api/v1/dbt-labs/logging.json 200
2021-10-28 09:23:32.881613 (MainThread): Flushing usage events
2021-10-28 09:23:32.905059 (MainThread): argument of type 'NoneType' is not iterable


Datadog: 
Traceback (most recent call last):
  File "/usr/src/app/sinter/clients/dbt.py", line 1164, in call
    dbt_main.handle(command + extra_args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 159, in handle
    res, success = handle_and_check(args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 205, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 258, in run_from_args
    results = task.run()
  File "/usr/local/lib/python3.8/dist-packages/dbt/task/deps.py", line 53, in run
    final_deps = resolve_packages(packages, self.config)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/resolver.py", line 137, in resolve_packages
    target = final[package].resolved().fetch_metadata(config, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/registry.py", line 129, in resolved
    available = registry.get_available_versions(self.package)
  File "/usr/local/lib/python3.8/dist-packages/dbt/clients/registry.py", line 78, in get_available_versions
    response = package(name)
  File "/usr/local/lib/python3.8/dist-packages/dbt/clients/registry.py", line 55, in package
    if ('redirectnamespace' in response) or ('redirectname' in response):
TypeError: argument of type 'NoneType' is not iterable

Environment

- dbt: 0.21.0

What database are you using dbt with?

snowflake

Additional Context

No response

[barberscott]

It does look like we get a response here:

2021-10-28 09:23:32.879183 (MainThread): Response from registry: GET https://hub.getdbt.com/api/v1/dbt-labs/logging.json 200

but that it's potentially empty or malformed, leading to response = None. Do we need to check for an empty response and retry in that case as well?

[jtcohen6]

@crystalro0 Thanks for the report!

@barberscott Your read here seems right on. It's reasonable to expect that dbt should continue retrying until it returns a response that isn't None, not just any successful response.

I think the change for that should happen somewhere in the weeds of _get, _get_with_retries, or connection_exception_retry. While that code is a bit thorny, it's fairly well self-contained. The fix could be a one-liner. I'm going to call this a good first issue, and I'd welcome a PR for it.

[b-per]

Hi! I'd be happy to work on this one unless @barberscott wants to create a PR himself.