Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove usage of dbt.deprecations in dbt/adapters, enable core & adapter-specific #9051

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changes/unreleased/Under the Hood-20231116-174251.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
kind: Under the Hood
body: Remove usage of dbt.deprecations in dbt/adapters, enable core & adapter-specific
event types and protos
time: 2023-11-16T17:42:51.005023-05:00
custom:
Author: michelleark
Issue: 8927 8918
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Configuration for pre-commit hooks (see https://pre-commit.com/).
# Eventually the hooks described here will be run as tests before merging each PR.

exclude: ^(core/dbt/docs/build/|core/dbt/common/events/types_pb2.py)
exclude: ^(core/dbt/docs/build/|core/dbt/common/events/types_pb2.py|core/dbt/events/core_types_pb2.py|core/dbt/adapters/events/adapter_types_pb2.py)

# Force all unspecified python hooks to run python 3.8
default_language_version:
Expand Down
9 changes: 9 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,15 @@ dev: dev_req ## Installs dbt-* packages in develop mode along with development d
proto_types: ## generates google protobuf python file from types.proto
protoc -I=./core/dbt/common/events --python_out=./core/dbt/common/events ./core/dbt/common/events/types.proto

.PHONY: core_proto_types
core_proto_types: ## generates google protobuf python file from core_types.proto
protoc -I=./core/dbt/events --python_out=./core/dbt/events ./core/dbt/events/core_types.proto

.PHONY: adapter_proto_types
adapter_proto_types: ## generates google protobuf python file from core_types.proto
protoc -I=./core/dbt/adapters/events --python_out=./core/dbt/adapters/events ./core/dbt/adapters/events/adapter_types.proto


.PHONY: mypy
mypy: .env ## Runs mypy against staged changes for static type checking.
@\
Expand Down
2 changes: 1 addition & 1 deletion core/dbt/adapters/base/connections.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
)
from dbt.common.events import AdapterLogger
from dbt.common.events.functions import fire_event
from dbt.common.events.types import (
from dbt.adapters.events.types import (
NewConnection,
ConnectionReused,
ConnectionLeftOpenInCleanup,
Expand Down
7 changes: 4 additions & 3 deletions core/dbt/adapters/base/impl.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@
from dbt.contracts.graph.manifest import Manifest, MacroManifest
from dbt.contracts.graph.nodes import ResultNode
from dbt.common.events.functions import fire_event, warn_or_error
from dbt.common.events.types import (
from dbt.adapters.events.types import (
CacheMiss,
ListRelations,
CodeExecution,
Expand All @@ -82,7 +82,8 @@
from dbt.adapters.base import Column as BaseColumn
from dbt.adapters.base import Credentials
from dbt.adapters.cache import RelationsCache, _make_ref_key_dict
from dbt import deprecations
from dbt.adapters.events.types import CollectFreshnessReturnSignature


GET_CATALOG_MACRO_NAME = "get_catalog"
GET_CATALOG_RELATIONS_MACRO_NAME = "get_catalog_relations"
Expand Down Expand Up @@ -1263,7 +1264,7 @@ def calculate_freshness(
]
result = self.execute_macro(FRESHNESS_MACRO_NAME, kwargs=kwargs, manifest=manifest)
if isinstance(result, agate.Table):
deprecations.warn("collect-freshness-return-signature")
warn_or_error(CollectFreshnessReturnSignature())
adapter_response = None
table = result
else:
Expand Down
10 changes: 5 additions & 5 deletions core/dbt/adapters/base/meta.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
import abc
from functools import wraps
from typing import Callable, Optional, Any, FrozenSet, Dict, Set

from dbt.deprecations import warn, renamed_method

from dbt.common.events.functions import warn_or_error
from dbt.adapters.events.types import AdapterDeprecationWarning

Decorator = Callable[[Any], Callable]

Expand Down Expand Up @@ -62,11 +61,12 @@

def wrapper(func):
func_name = func.__name__
renamed_method(func_name, supported_name)

@wraps(func)
def inner(*args, **kwargs):
warn("adapter:{}".format(func_name))
warn_or_error(

Check warning on line 67 in core/dbt/adapters/base/meta.py

View check run for this annotation

Codecov / codecov/patch

core/dbt/adapters/base/meta.py#L67

Added line #L67 was not covered by tests
AdapterDeprecationWarning(old_name=func_name, new_name=supported_name)
)
return func(*args, **kwargs)

if parse_replacement:
Expand Down
2 changes: 1 addition & 1 deletion core/dbt/adapters/cache.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
NoneRelationFoundError,
)
from dbt.common.events.functions import fire_event, fire_event_if
from dbt.common.events.types import CacheAction, CacheDumpGraph
from dbt.adapters.events.types import CacheAction, CacheDumpGraph
from dbt.utils import lowercase


Expand Down
3 changes: 1 addition & 2 deletions core/dbt/adapters/contracts/connection.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,8 @@
from dbt.common.contracts.util import Replaceable
from dbt.common.utils import md5

# TODO: dbt.common.events dependency
from dbt.common.events.functions import fire_event
from dbt.common.events.types import NewConnectionOpening
from dbt.adapters.events.types import NewConnectionOpening

# TODO: this is a very bad dependency - shared global state
from dbt.common.events.contextvars import get_node_info
Expand Down
54 changes: 54 additions & 0 deletions core/dbt/adapters/events/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# Events Module
The Events module is responsible for communicating internal dbt structures into a consumable interface. Because the "event" classes are based entirely on protobuf definitions, the interface is really clearly defined, whether or not protobufs are used to consume it. We use Betterproto for compiling the protobuf message definitions into Python classes.

# Using the Events Module
The event module provides types that represent what is happening in dbt in `events.types`. These types are intended to represent an exhaustive list of all things happening within dbt that will need to be logged, streamed, or printed. To fire an event, `common.events.functions::fire_event` is the entry point to the module from everywhere in dbt.

# Logging
When events are processed via `fire_event`, nearly everything is logged. Whether or not the user has enabled the debug flag, all debug messages are still logged to the file. However, some events are particularly time consuming to construct because they return a huge amount of data. Today, the only messages in this category are cache events and are only logged if the `--log-cache-events` flag is on. This is important because these messages should not be created unless they are going to be logged, because they cause a noticable performance degredation. These events use a "fire_event_if" functions.

# Adding a New Event
* Add a new message in types.proto, and a second message with the same name + "Msg". The "Msg" message should have two fields, an "info" field of EventInfo, and a "data" field referring to the message name without "Msg"
* run the protoc compiler to update adapter_types_pb2.py: make adapter_proto_types
* Add a wrapping class in core/dbt/adapters/event/types.py with a Level superclass plus code and message methods

We have switched from using betterproto to using google protobuf, because of a lack of support for Struct fields in betterproto.

The google protobuf interface is janky and very much non-Pythonic. The "generated" classes in types_pb2.py do not resemble regular Python classes. They do not have normal constructors; they can only be constructed empty. They can be "filled" by setting fields individually or using a json_format method like ParseDict. We have wrapped the logging events with a class (in types.py) which allows using a constructor -- keywords only, no positional parameters.

## Required for Every Event

- a method `code`, that's unique across events
- assign a log level by using the Level mixin: `DebugLevel`, `InfoLevel`, `WarnLevel`, or `ErrorLevel`
- a message()

Example
```
class PartialParsingDeletedExposure(DebugLevel):
def code(self):
return "I049"

def message(self) -> str:
return f"Partial parsing: deleted exposure {self.unique_id}"

```


# Adapter Maintainers
To integrate existing log messages from adapters, you likely have a line of code like this in your adapter already:
```python
from dbt.logger import GLOBAL_LOGGER as logger
```

Simply change it to these two lines with your adapter's database name, and all your existing call sites will now use the new system for v1.0:
```python
from dbt.common.events import AdapterLogger
logger = AdapterLogger("<database name>")
# e.g. AdapterLogger("Snowflake")
```

## Compiling types.proto

After adding a new message in `adapter_types.proto`, either:
- In the repository root directory: `make adapter_proto_types`
- In the `core/dbt/adapters/events` directory: `protoc -I=. --python_out=. types.proto`
Empty file.
Loading