[CT-77] [Bug] dbt deps should keep retrying if registry response is malformed (bug regression)

[BrJan]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Running dbt deps in a nightly job failed after retrying downloading the packages. The Max retries is listed as 5 and was currently at 0.

Expected Behavior

The download is retried until the max retries is reached or the package is downloaded successfully.

Steps To Reproduce

This cannot be reproduced on demand. It is an intermittent failure case for dbt deps

Relevant log output

2022-01-20 10:07:06.950387 (MainThread): 10:07:06  Making package registry request: GET https://hub.getdbt.com/api/v1/fivetran/fivetran_utils.json
2022-01-20 10:07:07.058992 (MainThread): 10:07:07  Response from registry: GET https://hub.getdbt.com/api/v1/fivetran/fivetran_utils.json 200
2022-01-20 10:07:07.059729 (MainThread): 10:07:07  Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7efcdd10f580>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7efcdd10f8b0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x7efcdd10f4f0>]}
2022-01-20 10:07:07.060251 (MainThread): 10:07:07  Flushing usage events
2022-01-20 10:07:07.083491 (MainThread): argument of type 'NoneType' is not iterable

Datadog:
Traceback (most recent call last):
  File "/usr/src/app/sinter/clients/dbt.py", line 1200, in call
    dbt_main.handle(command + extra_args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 159, in handle
    res, success = handle_and_check(args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 205, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 258, in run_from_args
    results = task.run()
  File "/usr/local/lib/python3.8/dist-packages/dbt/task/deps.py", line 53, in run
    final_deps = resolve_packages(packages, self.config)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/resolver.py", line 137, in resolve_packages
    target = final[package].resolved().fetch_metadata(config, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/registry.py", line 129, in resolved
    available = registry.get_available_versions(self.package)
  File "/usr/local/lib/python3.8/dist-packages/dbt/clients/registry.py", line 78, in get_available_versions
    response = package(name)
  File "/usr/local/lib/python3.8/dist-packages/dbt/clients/registry.py", line 55, in package
    if ('redirectnamespace' in response) or ('redirectname' in response):
TypeError: argument of type 'NoneType' is not iterable

Environment

- dbt: 1.0.1

What database are you using dbt with?

snowflake

Additional Context

This is a regression of #4178.

[emmyoop]

Thanks for the report @BrJan! This issue is a duplicate of #4577 that was recently reported. The log output for each is a bit different but in both we're hitting a bug here:

dbt-core/core/dbt/clients/registry.py

Lines 36 to 39 in 7798f93

	if resp is None:
	raise requests.exceptions.ContentDecodingError(
	'Request error: The response is None', response=resp
	)

I'm going to close this issue as a duplicate. We're targeting the next release for a fix for this.

[leahwicz]

Backport: #4649

[barberscott]

Turns on v1.0.3 did not fix this variant of the failure:

Traceback (most recent call last):
  File "/usr/src/app/sinter/clients/dbt.py", line 1280, in call
    dbt_main.handle(command + extra_args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 152, in handle
    res, success = handle_and_check(args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 192, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 246, in run_from_args
    results = task.run()
  File "/usr/local/lib/python3.8/dist-packages/dbt/task/deps.py", line 56, in run
    final_deps = resolve_packages(packages, self.config)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/resolver.py", line 135, in resolve_packages
    target = final[package].resolved().fetch_metadata(config, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/registry.py", line 150, in resolved
    available = registry.get_available_versions(self.package)
  File "/usr/local/lib/python3.8/dist-packages/dbt/clients/registry.py", line 90, in get_available_versions
    response = package(name)
  File "/usr/local/lib/python3.8/dist-packages/dbt/clients/registry.py", line 67, in package
    if ('redirectnamespace' in response) or ('redirectname' in response):
TypeError: argument of type 'NoneType' is not iterable

[barberscott]

Another variation on the same theme:

Traceback (most recent call last):
  File "/usr/src/app/sinter/clients/dbt.py", line 1284, in call
    dbt_main.handle(command + extra_args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 152, in handle
    res, success = handle_and_check(args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 192, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 246, in run_from_args
    results = task.run()
  File "/usr/local/lib/python3.8/dist-packages/dbt/task/deps.py", line 56, in run
    final_deps = resolve_packages(packages, self.config)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/resolver.py", line 135, in resolve_packages
    target = final[package].resolved().fetch_metadata(config, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/base.py", line 86, in fetch_metadata
    self._cached_metadata = self._fetch_metadata(project, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/registry.py", line 61, in _fetch_metadata
    return RegistryPackageMetadata.from_dict(dct)
  File "<string>", line 38, in from_dict
ValueError: Argument for dbt.contracts.project.RegistryPackageMetadata.from_dict method should be a dict instance

[leahwicz]

Goal: Let's confirm if we are retrying like expected? If not, why not?

[emmyoop]

We are not retrying for these cases.

After looking through some logs, I'm seeing three exceptions thrown that we are not catching.

1. requests.exceptions.ReadTimeout - more rare, all on 0.20.2
2. requests.exceptions.ChunkedEncodingError - once in March on 1.0.0
3. requests.exceptions.HTTPError caused by 500 and 502 server errors. Many times. On all versions

These need to be added to the list of exceptions to catch and retry.

dbt-core/core/dbt/utils.py

Lines 609 to 613 in ecfd77f

	except (
	requests.exceptions.ConnectionError,
	requests.exceptions.Timeout,
	requests.exceptions.ContentDecodingError,
	ReadError,

Alternatively we could catch all exceptions from Requests by catching just requests.exceptions.RequestException since all exceptions that Requests explicitly raises inherit from requests.exceptions.RequestException. We could fire and event to track what exception we're retrying on here as well so that we can track them if needed.

[jtcohen6]

Just to confirm - we're planning to close this one in favor of / by means of #4849?

[emmyoop]

Since that task mentions this one and has all the details to hopefully fix this issue, yes.