-
Notifications
You must be signed in to change notification settings - Fork 3.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Workflow of workflows fails when input parameters are specified on inner workflow #12634
Comments
I've seen some similar problems before on Slack, but unfortunately I'm not sure there's a clean way of solving this. For instance, you can't just exclude At my last job, I was able to workaround this with some indirection by chucking an Argo CD
Also surprised you're on such an old version, would be good to update 😅 |
What sort of escaping? Escaping the
Yea, scheduled 😅 |
Yes, Argo would have to, for instance, convert That's not a feature that exists, which is why I labeled this as a feature rather than a bug. It's possible, but hacky, and Argo templating already has a number of hacks to work within the confines of YAML (e.g. all expressions are strings, but some need to be converted to different types, etc) |
Got it, thanks for clarifying! And thanks for labeling the issue as an enhancement. I followed what you mentioned about template and here's what I did (with Hera): from uuid import uuid4
from hera.workflows.argo import Workflow, Steps, script, Resource, WorkflowTemplate, models as m
@script(image="python:3.7-slim")
def foo(msg):
print(msg)
with WorkflowTemplate(name=str(uuid4()), entrypoint="s") as w1_template:
with Steps(name="s"):
foo(name="foo1", arguments={"msg": "Hello, World!"})
# create w1 as a workflow template since it will have everything necessary "pre-registered", like arguments
w1_template.create()
# then make a workflow that references it
w1 = Workflow(name="foo1", workflow_template_ref=m.WorkflowTemplateRef(name="foo1"))
with WorkflowTemplate(name=str(uuid4()), entrypoint="s") as w2_template:
with Steps(name="s"):
foo(name="foo1", arguments={"msg": "Hello, World!"})
# similarly to w1, create w2 as a workflow template
w2_template.create()
# then make a workflow that references it
w2 = Workflow(name="foo2", workflow_template_ref=m.WorkflowTemplateRef(name="foo2"))
# finally, make a workflow of workflows that launches the other two
with Workflow(generate_name="wf-of-wf-", entrypoint="main") as w:
w1r = Resource(name="w1r", manifest=w1, action="create")
w2r = Resource(name="w2r", manifest=w2, action="create")
with Steps(name="main"):
w1r()
w2r()
w.create() While valid, it comes with the cost of implementing a way to clean up templates. The reason I mention this is because there are many use cases for "experimentation on Argo Workflows". Since parameters between workflow submission can change, so will the templates, so clients end up with a proliferation of templates unless they are deleted periodically and strategically (based on labels, annotations, name patterns, etc). |
Ah right, a plain
You mean like testing out Workflows in a dev environment?
Also, speaking of old versions, Python 3.7 is EoL 😅 |
No, I am referring to experimentation. Think prod environment that supports arbitrary async job execution. I know multiple orgs who use Argo Workflows as their "experimentation platform" so their workflows cannot be turned into templates because they change often as part of the experiment (parameters change, scripts change, etc). This is all from Hera's lens btw, so it might feel a bit strange given AW only supports YAML and "static" templating. However, for "workflows of workflows" users can turn their workflows into templates (like above) prior to execution, but then have to delete them probably because those templates will not be used again
Hera defaults to python:3.8 😅 the above was my mistake for including an EoL image, sorry |
Huh okay. My last job was on an ML Platform, but the experimentation side was nearly all Notebooks (JupyterHub or Kubeflow). There were some advanced use-cases on Kubeflow Pipelines, but they were in the minority and were more typically productionized training jobs (i.e. no longer experimental). Also, all of that environment had periodic pruning/clean up jobs already, even before Argo/KFP was introduced.
Yea I've worked enough with KFP which has even more out-of-Argo code sharing features, so I'm familiar with having a bit of a impedance mismatch (esp with all the spec renames, metadata server, etc etc in KFP) 😅 I was thinking you could further use Python's From the Argo side, a small bit of indirection with a |
Expr now supports raw strings: https://expr-lang.org/docs/language-definition#strings |
Thanks for the note! This does appear to work. I thought that would work in isolation, but apparently Argo needs to escape its own expression syntax Concatenating two strings should be a sufficient workaround for this use-case too, and would avoid needing a
There might be a third (or foruth) workaround here for this specific use-case too: make the entire manifest (or a chunk of it) into a parameter. Then |
Concretely, the current workaround is to use |
No I don't believe that will work (though you can try) due to lack of escaping. I meant the same tactic as used in #12805 (comment) (which I linked above). In this case, that would be:
Might need to do something similar with
That effectively forces no substitutions |
What about adding a helper to wrap in brackets?
|
Mmm I don't think a helper would be optimal -- Argo should probably not try to parse any of the text inside an expression; that seems pretty confusing that it does and I don't think that's intentional (I think it's just running a simple replace when it detects braces; I believe that predates |
Pre-requisites
:latest
What happened/what did you expect to happen?
Context
My organization has a use case for workflows of workflows. However, when we create inner workflows via the resource specification the Argo Server fails to resolve
{{input.parameters.whatever}}
. The outer workflow template does not have any inputs specified while the inner workflows (specified as resources) have inputs + arguments specified (via steps).Expected
I expected the workflow to be submitted successfully since inner workflows submitted as resources should be independent of the outer workflow.
What actually happened
I fail to submit the workflow with an error like
Extra context
The workflow YAML file is created via Hera. I went through Hera's code and could not find anything that generates anything unnecessarily. In addition, I verified the YAML, looked through Argo docs + examples. Since I cannot even submit this workflow I cannot get
kubectl
logs.Hera discussion on the topic: argoproj-labs/hera#803
Hera specification
Version
v3.3.3
Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
Logs from the workflow controller
Logs from in your workflow's wait container
The text was updated successfully, but these errors were encountered: