Process UUID, updating and versioning

[fmigneault]

Main feature

In order to support process updates for things like metadata (abstract, title, etc.), a new route method should be added:

PATCH /processes/{id} with update payload like {"abstract": "<text>"}

Process versioning issues and considerations

• Only fields that do not impact the integrity of the package can be updated (eg: quick patches to docs).
• Other fields (ie: docker version, new I/O, etc.) will require a new processVersion (the field specified by WPS standard). Internally, we will treat this new version as a totally different process.
• On the API side, requesting the slug-name will return the latest matching one, and requesting the literal UUID will return that specific version (relates to discovery Improve discovery/listing of deployed processes #102).
• Because same slug names can be used, any process iterating/set operation in the code will have to be checked to make sure they don't ignore all but one version.
• Process conflict will have to be validated against slug+version combination instead of slug-only.
• Process deployment will not take a local UUID as input. It is generated automatically and used to manage versions.
• Probably need a get_any_version method as both version and processVersion keys are used to reflect the same information at different locations (ie: process list vs single process description).
• workflow steps will need to support versions (ie: <slug> will return the latest process, an explicit <uuid> will pin to tag version, and for convenience, <slug>:<version> for a more readable slug-pinned version. <slug>:latest resolved as <slug> to match docker style references).
  - [ ] we can consider storing a commit hash if the reference is from a git repository (eg: reference notebook, relates to Process Package generation from Jupyter Notebook definition #63)
• we can provide link items named predecessor-version, successor-version and latest-version that should refer to the corresponding semantically ordered versions when available (as per https://www.iana.org/assignments/link-relations/link-relations.xhtml)
• listing of historical versions should be supported for lookup (could be a query parameter), and referenced by corresponding processes by link item named version-history

Process deployment issues

To make sure that latest versions are executed by ADES, an EMS workflow will need to make sure to validate the process with specific UUID. Up to this point, an existing process-id (slug) on the ADES was assumed to be already deployed. Since the slug name was used instead of an UUID, this is both insecure (could be a conflicting process named the same) but also doesn't help to know exactly which process package was executed. Worst, if process is updated, undeployed, re-deployed, an old job execution doesn't point to the real process definition that was executed. This makes it hard (if not impossible) to audit the results.

Therefore, process deployment should accept in body an optional source dictionary :

{
  "process": {
    "processVersion": "x.y.z",
    "source": {
      "uuid": "<uuid>",
      "href": "<path-to-ems-process-uuid>",
      "title": "<process-id-slug>" 
    }
    [...]
  },
}

And process description should provide it as a link if available:

{
  "process": { [...] },
  "links": [
    {
      "rel": "<uuid>",
      "href": "<path-to-ems-process-uuid>",
      "title": "<process-id-slug>" 
    }
  ]
} 

Optionally, an EMS could update back its process links with successful response of deployed ADES link references for tracking deployed instances (note: they could be undeployed later on, so must test links to be sure).

Support for other implementations

Since we are not the only ADES providers, we cannot assume the source UUID is always available in the process description. For this reason, we need to use the local-EMS UUID as the remote-ADES process-id slug for deployment.

Using that UUID directly as the process-id slug will remove any chance to search corresponding processes by version on an ADES that supports it, but this is acceptable as it is only a working unit. The EMS (user interface) will have versioning, and ADES that support versions will have a link referencing back to the original EMS process.

If the ADES supports versions though (it provides the links[].source), the EMS should validate that the reference refers correctly to itself and the local UUID it self-assigned for that process.