[CT-784] When using --log-format json, exceptions during query execution result in JSON Serialization errors

[OwenKephart]

Describe the bug

When an Exception is raised during query execution, and the log format is set to json, a JSON serialization exception will be raised instead of the original Exception.

Steps To Reproduce

Create a dbt project with the following model:

select * from foo.bar

and run dbt --log-format json run

Expected behavior

The error message should be something along the lines of:
404 Not found: Table my-dataset:foo.bar was not found in location US"

Screenshots and log output

The actual error message (once parsed out of the "msg" field of the json logs: dbt --log-format-json run | jq .msg):

"Completed with 1 error and 0 warnings:"
"Compilation Error in macro statement (macros/etc/statement.sql)"
"  Object of type NotFound is not JSON serializable"
"  "
"  > in macro materialization_table_bigquery (macros/materializations/table.sql)"
"  > called by macro statement (macros/etc/statement.sql)"

System information

The output of dbt --version:

Core:
  - installed: 1.1.1
  - latest:    1.1.1 - Up to date!

Plugins:
  - bigquery: 1.1.0 - Up to date!

The operating system you're using:
macOS Catalina

The output of python --version:
Python 3.8.7

Additional context

This is happening because of this line: https://github.com/dbt-labs/dbt-bigquery/blob/main/dbt/adapters/bigquery/connections.py#L217

With the default log formatter, this Exception will be automatically stringified. The json log formatter instead just takes the raw exception and attempts to convert it to JSON.

[jtcohen6]

@OwenKephart Thanks for opening, and for the clear explanation!

This is happening because of this line: https://github.com/dbt-labs/dbt-bigquery/blob/main/dbt/adapters/bigquery/connections.py#L217

This is exactly right.

I believe this is the same underlying issue as the one we caught here: dbt-labs/dbt-spark#371. That is, this line is not appropriately stringifying a complex object that's passed into the AdapterLogger, which results in serialization errors. I'm going to open a dbt-core ticket to track that work accordingly. I'm not strictly opposed to the change in #207 as a temporary salve, but the real solution should live in dbt-core.

[VersusFacit]

Hiya people on the thread.

Per this core PR, this JSON serialization bug should be solved across all adapters, although I specifically verified the bug & fix on a bigquery cluster. There's a lot of layers of indirection in the logger call stack, so finding the root cause of this error took us some concerted time. I should have corrected this while addressing similar JSON issues in our core/dbt/events/ package.

I also threw on our PRs backlog tags, so in theory, you should be able to "seamlessly" integrate the fix into your env. Also, it's live on main.

I'd love to close this if (🤞) people report things working here.

[Fleid]

Closing as per @VersusFacit's comment above.
Please open again if need be!