How to set MAX_RUNTIME_SECONDS_TAG and tags on auto materializing @graph_asset

[chrishiste]

Hello, we have a job backed by a @graph_asset using define_asset_job(). We are setting tags for job concurrency and MAX_RUNTIME_SECONDS_TAG for timeouts on that job.

I would like to use an auto materializing @graph_asset instead of a job.

It appears I will lose the ability to configure concurrency and timeouts since I cannot set auto_materialize_policy on the job nor tags/MAX_RUNTIME_SECONDS_TAG on a @graph_asset. I feel like there must be a way to do that. Any help appreciated. 😁

[chrishiste]

Adding some context.

I saw you can configure global tags for all auto materializing assets:

But in my case, I want to configure specific tags on a specific @graph_asset because I'm trying to set the concurrency to 1. So that there is always only one partition of my @graph_asset running in parallel. I would also want to set a timeout using MAX_RUNTIME_SECONDS_TAG on a case by case instead of globally since I could have very different kind of auto-materializing assets.

[sryza]

@chrishiste would op/asset-level concurrency work for you? https://docs.dagster.io/guides/limiting-concurrency-in-data-pipelines#limiting-opasset-concurrency-across-runs

For run-level concurrency, here's a relevant issue: #15752

[sryza]

Another relevant issue: #8979

[chrishiste]

@sryza I can achieve what I want using op/asset-level concurrency, I didn't that new feature existed thanks :)

Unfortunately for timeouts, I will have to wait apparently: #3666

Unless that changed too?

[sryza]

That's right for timeouts.

Something that may be helpful in the near future on this: we're working on a change that will allow saying "auto-materialize should always execute these assets in their own run, not combine them with other runs", and you'll also be able to specify tags that apply to that run.

[chrishiste]

Awesome, looking forward to it. Thanks.

[chrishiste]

#17498