Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use dbt postgres adapter wrapper for integration tests #614

Merged
merged 4 commits into from
Jun 23, 2023
Merged

Use dbt postgres adapter wrapper for integration tests #614

merged 4 commits into from
Jun 23, 2023

Commits on Jun 23, 2023

  1. Move make_sql_client helper to sql_client_fixtures 

    This helper function was tagged as only being used in tests, but
it was sitting in the main package. This moves it to the test package
to avoid confusion as we restructure around dbt adapter integrations.
    @tlento
    tlento committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    7a46617 View commit details
    Browse the repository at this point in the history

  2. Use dbt postgres adapter wrapper for all integration tests 

    As part of the effort to provide basic metric querying capabilities
for local installations of dbt core, MetricFlow is removing all
engine-specific dependencies and instead delegating warehouse
connection and query management to dbt adapters for integration
tests and command line operations.

This commit is the first step towards moving onto dbt adapters. It
provides a new wrapper class for dbt adapters, conforming to
MetricFlow's SqlClient protocol, and delegating all operations
down to the underlying adapter. It uses this adapter in test cases
only, and sources it from a dummy project in the test fixtures
directory. The dummy project is configured to allow for different
adapters (configured through test runner environment variables), and
in future it will likely be used as the basis for MetricFlow's
core integration test table management.

Note the implementation of the SqlClient wrapper class has some
highly temporary work-arounds to certain rough edges that exist
between dbt's adapter interface and the current SqlClient requirements.
Most of these arise from table and schema management calls, which are
only used in the MetricFlow CLI tutorial and integration test suite.
As such, we expect those methods to be removed from the SqlClient
interface as we move management of these datasets to dbt.

Further, the dbt adapter fetching is a bit suspect, and not to be
copied - we are relying on the dbtRunner()'s state loading to remain
valid as we access certain internal APIs from dbt core to get the
profile we need. All of this works, and we expect it to continue
working, but we will need to maintain this connection and keep it
up to date as the dbt core project updates its external accessors for
the dbt profile and dbt project which ultimately initialize and house
the adapter instance.
    @tlento
    tlento committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    e5946d7 View commit details
    Browse the repository at this point in the history

  3. Fix generate snapshots helper for postgres 

    The switch to dbt adapters breaks the Postgres test run for
generating snapshots. Since Postgres runs so fast we simply
run all tests via the relevant hatch env command.
    @tlento
    tlento committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    6c841ac View commit details
    Browse the repository at this point in the history

  4. Clean up confusing test environment constructs 

    The dbt adapter-backed SqlClient needs to skip some tests and
depends on some custom package installations. This commit updates
the logic for the former to depend only on DuckDB, which is not
currently planned for migration to adapters. We also add a comment
explaining the installation configuration in the pyproject.toml.
    @tlento
    tlento committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    cfab7c7 View commit details
    Browse the repository at this point in the history