[CT-2693] [Feature] Allow debug logs for dbt-databricks

[susodapop]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Add a mapping to enable DEBUG logs for log events from databricks-sql-connector similar to how snowflake.connector is handled today. This will assist in debuggin dbt-databricks issues.

Describe alternatives you've considered

This must be set at the dbt-core level.

Who will this benefit?

Anyone doing technical debugging of performance and behaviour of dbt-databricks.

Are you interested in contributing this feature?

Yes. PR incoming.

Anything else?

No response

[dbeatty10]

Sounds good @susodapop If you share a branch or open a PR with your proposed changes, then we can give it a look.

[jtcohen6]

The fact that this needs to be set within dbt-core doesn't feel right! Is there any way to put this logic inside the dbt-databricks adapter instead? All we need to do is call logging.getLogger(...).setLevel() — but I understand that this needs to happen at initialization, perhaps before the adapter has been registered/loaded:

dbt-core/core/dbt/logger.py

Lines 500 to 510 in b947b2b

	LOG_LEVEL_GOOGLE = _env_log_level("DBT_GOOGLE_DEBUG_LOGGING")
	LOG_LEVEL_SNOWFLAKE = _env_log_level("DBT_SNOWFLAKE_CONNECTOR_DEBUG_LOGGING")
	LOG_LEVEL_BOTOCORE = _env_log_level("DBT_BOTOCORE_DEBUG_LOGGING")
	LOG_LEVEL_HTTP = _env_log_level("DBT_HTTP_DEBUG_LOGGING")
	LOG_LEVEL_WERKZEUG = _env_log_level("DBT_WERKZEUG_DEBUG_LOGGING")
	logging.getLogger("botocore").setLevel(LOG_LEVEL_BOTOCORE)
	logging.getLogger("requests").setLevel(LOG_LEVEL_HTTP)
	logging.getLogger("urllib3").setLevel(LOG_LEVEL_HTTP)
	logging.getLogger("google").setLevel(LOG_LEVEL_GOOGLE)
	logging.getLogger("snowflake.connector").setLevel(LOG_LEVEL_SNOWFLAKE)

[susodapop]

I agree with your reasoning @jtcohen6. The reason to change it in dbt-core was primarily expediency, since this is where the log level is set for all the other connectors. I'm happy to move this into dbt-databricks assuming it will work that way.

[jtcohen6]

assuming it will work that way

I don't know for sure. If you're able to give it a quick spin and try it out, I'd definitely prefer moving this platform-specific logic into the relevant adapter plugins. If it's too tricky to get working, we already have this (legacy) code in place; I won't stand in the way of adding a teensy bit more for expediency.

[colin-rogers-dbt]

Ran into this same problem for dbt-snowflake so figured we could use a more general solution.