upstream dependencies for dbt assets

[brandybobandyistaken]

I'm using dagster to orchestrate airbyte and dbt.

I have connections to multiple airbyte streams that serve as the root level assets, that then feed into downstream dbt models.

Currently, because I'm loading in dbt assets with one dbt_assets decorator, if any of the upstream airbyte assets fail none of the downstream dbt assets will run.

I've had limited success with creating multiple @dbt_assets decorators, which seems to separate the dbt assets from eachother in terms of upstream dependencies.

What I want to happen, is for the dbt models that don't depend on the failed airbyte sync to continue to run.

Is there any way to do this without loading in dbt models with n @dbt_assets decorators?

[jamiedemaria]

Hi @brandybobandyistaken there isn't a way to do this right now. The @dbt_asset is a @multi_asset under the hood and multi assets are processed as a single step. If any of the upstream assets for a multi asset fail, the entire step is deemed unexecutable. In the dagster internals we could potentially do some more introspection on the type of step an if it's a multi asset and it's subsettable and there's a distinct set of assets that could be executed without the failed upstream step, then we could modify the execution plan to execute just the subset of the multi-asset. But this is pretty in-depth work and I dont foresee us having time to prioritize it in the near future

[brandybobandyistaken]

Thanks for the clarification!

What we've done in the meantime, that others might find useful, is created an asset factory that dynamically creates n @dbt_asset decorators for the n dbt models that we wish to load in, thus executing n dbt commands instead of a single dbt command targeting a group models. This seems to have allowed us to allow a dbt asset model to only depend on its direct upstream assets, and not also on the upstream assets of any other models loaded in with the same @dbt_asset decorator.

[tweavers]

@brandybobandyistaken - Would dbt freshness work for you? We are starting to explore dbt + dagster and I imagine we would run into a similar use case. Our current orchestration heavily relies ondbt freshness to determine if source data has been updated

[brandybobandyistaken]

Sorry for the late reply @tweavers . We are currently not using dbt_freshness, but might explore it in the future. Our reasoning is that dagster is the only user that has credentials for and should be writing to our prod schemas. Thus, if a failure occurs in dagster, we can be almost sure that the data is indeed not up to date. This way, we don't have competing sources of truth for whether or not data is up to date between dbt freshness and dagster dependencies.

[mitchbregs]

@brandybobandyistaken would you be able to share how you performed this?

[brandybobandyistaken]

1. read in the manifest.json manually
2. select the nodes key inside the json file (dictionary)
3. loop over the nodes key
4. for each node, if the key starts with 'model', create a new @dbt_assets decorator for it (this is where we created factories depending on the tag we were subsetting for)