YAML Pipeline Validation

[ZanSara]

Rationale

Pipelines can be defined as YAML files. However right now it's nearly impossible to understand if the supplied pipeline is actually valid and if not, and if not, what's the reason.

Providing validation can be extremely useful to users, especially less experienced ones, or users with very complex pipeline designs.

Main Goals

• Investigate YAML schema possibilities
• Define how strict the validation should be. For example, shall we allow for almost nonsensical pipelines, as long as the system could technically work with no exceptions? Shall warnings be thrown?
• Validate pipeline syntax: verify from a YAML if it's valid (e.g. params, node classes ...)
• Validate pipeline semantics: verify if combination of nodes makes sense (e.g. no retriever after reader)

[lalitpagaria]

It would be great to build it as a CLI utility. CLI utility can -

• performs pipeline validation
• install required dependencies (Improve dependency management #1979 (comment))
• convert YAMLs to different version if required
• view a YAML file

[ZanSara]

I'm not sure what the benefit of a CLI tool would be here to be honest:

• performs pipeline validation: it could be done with a method call, rather than a CLI tool, but I see that it might be handy for some.
• install required dependencies: a good error message can tell the user what to do without having to maintain code that handles pip from Python (like: This YAML requires the 'faiss' dependency group. Please run 'pip install haystack[faiss]')
• convert YAMLs to different version if required: I'm not sure what you mean here. As a migration tool for future YAML versions?
• view a YAML file: I guess a text editor like VSCode could suffice 🙂

However, if we end up making a CLI utility with a bigger scope in the future, these are nice features to remember about.

[lalitpagaria]

With view, I mean pipeline visualisation.

I agree with your view. Anyway click is part of haystack as transitive dependency hence suggested it. Yes I think haystack as CLI utility can provide good value to the users.

[ZanSara]

I see! I totally misunderstood the last one, sorry. I would say we could open a separate issue for a future CLI tool. As soon as we see there are enough features to justify the cost, we'll plan for it. It would be a shame to forget about these smaller features later on.

I think it's better if you open the issue, you may know about other proposals that came out in the past. If so please gather them so we have a good starting point for the discussion 🙂

[tholor]

I agree, let's put this to a separate issue. One thing that might also be part of such a CLI is "setting up a haystack project / pipeline" based on some command line questionnaire / templates (bit like cookiecutter). This might help you spin up a quick template for a "dense document retrieval pipeline based on elasticsearch" or let you choose the "right reader model" by prioritizing speed or accuracy etc.

[lalitpagaria]

Sure. Let me gather my thoughts, I will create an issue.