Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Remote UDP extension #515 #538

Merged
merged 4 commits into from
Jul 1, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## Unreleased / Draft

### Added

- **New extensions:**
- [Remote UDP Extension](./extensions/remote-udp/README.md)

### Fixed

- `GET /file_formats`: Base paramater on top of normal JSON Schema, not Process JSON Schema
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ See also the [changelog](CHANGELOG.md) and the [milestones](https://github.com/O
| ---------------------------------------------- | ------- | ------------ | ----------- |
| [Commercial Data](extensions/commercial-data/) | 0.1.0 | experimental | Provides an interface for discovering, ordering and using commercial data. |
| [Federation](extensions/federation/) | 0.1.0 | experimental | Covers federation aspects, i.e. where multiple back-ends are exposed as a single API. |
| [Remote UDP](extensions/remote-udp/) | 0.1.0 | experimental | Load user-defined processes that are hosted externally through the process namespace into process graphs. |

## Repository

Expand Down
69 changes: 69 additions & 0 deletions extensions/remote-udp/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# Remote UDP Extension

The openEO API is a specification for interoperable cloud-based processing of large Earth observation datasets.

This extension enables user to load user-defined processes that are hosted external to the openEO API (e.g., GitHub or cloud storage) through the process namespace into process graphs.

- Version: **0.1.0**
- Stability: **experimental**
- Conformance class: `https://api.openeo.org/extensions/remote-udp/0.1.0`

## Justification

The openEO API defines the `namespace` property in a process node of a process graph as follows:
> The following options are predefined by the openEO API, but additional namespaces may be introduced by back-ends or in a future version of the API.
> * `null` [...]
> * `backend` [...]
> * `user` [...]

This makes it possible for this extension to add additional allowed values to the `namespace` property.

## Specification

This extension extends the `namespace` property of process graph nodes so that it accepts **absolute** URL with the protocols `https` (**recommended**) and `http` (discouraged). The URLs specified MUST return one of the following two options:

1. A single process, compatible\* to the endpoint `GET /process_graphs/{process_graph_id}`.
In this case, the `id` property of the process graph node MUST be equal to the `id` of the process,
otherwise a `ProcessNamespaceInvalid` error is thrown
2. A list of processes, compatible\* to the endpoint `GET /process_graphs`.
In this case, the `id` property of the process graph node is used to identify the process from the list.
If not found a `ProcessNamespaceInvalid` error is thrown

\* Compatible means in this context that the requests and responses must comply to the openEO API specification with the following exceptions:

- User credentials / tokens that are obtained through the openEO API MUST NOT not be sent to URIs that are external to the openEO API.
The requirement to provide an `Authorization` header for the respective endpoints doesn't apply.
- For a list of processes, the full process description MUST be provided for the process with the given ID within the first request.
This means that the recommendation to omit large roperties such as `process_graph` doesn't apply.
It also requires that the requester doesn't need to paginate through additional pages to find the process with the given ID.
Ideally, the list of processes is not paginated as otherwise the process with the given ID may move to other pages over time.

### Client Considerations

If a client is conncected to a specific backend, the client MUST only offer this functionality to users if the conformance class of this extension is listed in the `conformsTo` property of the `GET /` endpoint.

The protocol `http` is discouraged for URLs as web-based clients may not be able to retrieve HTTP URLs from a HTTPS context.
For the same reason it is also RECOMMENDED to enable CORS for all URLs.
m-mohr marked this conversation as resolved.
Show resolved Hide resolved

### Error Handling

The following error SHOULD be reported if the namespace can't be resolved:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if the URL can be resolved, but the document does not fit? e.g. it's not JSON, it does not follow expected schema, it doesn't contain a process_graph property, ...?

Shouldn't this also be a ProcessNamespaceInvalid?
Or would that be for standard errors like ProcessInvalid, ProcessGraphMissing? Note however that these don't have placeholders like for a reason, so that could lead to poor error messages

Copy link
Member Author

@m-mohr m-mohr Jul 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I guess that should fallback to the standard errors. Generally, back-ends can also define own error messages and codes. If specific common cases occur that are not covered and need a reason, we can fine-tune this later. Please give feedback :-)


- Code: `ProcessNamespaceInvalid`
- Message: `The value passed for namespace '{namespace}' in process '{process}' is invalid: {reason}`
- HTTP Status Code: 400

## Example

An exemplary process graph node:

```json
{
"process_id": "echo",
"namespace": "https://openeo.example/processes/echo",
"arguments": {
"message": "Hello World"
},
"result": true
}
```
Loading