add load_stac process

[jdries]

https://processes.openeo.org/draft/#load_stac

This probably has a lot in common with load_result.
There is one important change: we want to use FileLayerProvider, with a STAC client.

Specific case:
STAC API Collection that allows to filter items and to download assets.

[bossie]

Is a v2.x feature: Open-EO/openeo-processes#439

[bossie]

Differences with load_result:

• only accepts URLs, not batch job IDs
• supports static STAC catalogs as well as dynamic STAC APIs
• supportsproperties argument

[bossie]

From the process description:

Batch job results can be specified in two ways:

For Batch job results at the same back-end, a URL pointing to the corresponding batch job results endpoint should be provided. The URL usually ends with /jobs/{id}/results and {id} is the corresponding batch job ID.
For external results, a signed URL must be provided. Not all back-ends support signed URLs, which are provided as a link with the link relation canonical in the batch job result metadata.

@jdries does this imply that the logged-in user should be able load his own batch job results by providing a /jobs/{id}/results URL that is not signed?

[jdries]

Good question, because I thought the canonical link was the only way to retrieve the correct url, but I think that this is indeed implied here. Note that for this ticket, implementing the remote url option is actually the main goal.

[soxofaan]

@jdries does this imply that the logged-in user should be able load his own batch job results by providing a /jobs/{id}/results URL that is not signed?

Indeed, I pushed for that because the signed URLs have an expiry, could be invalidated/rotated, ..., while the /jobs/{id}/results URL is static and easily predictable (but require auth headers). I think both options are valuable to have (for different use cases)

[bossie]

@jdries does this imply that the logged-in user should be able load his own batch job results by providing a /jobs/{id}/results URL that is not signed?

Indeed, I pushed for that because the signed URLs have an expiry, could be invalidated/rotated, ..., while the /jobs/{id}/results URL is static and easily predictable (but require auth headers). I think both options are valuable to have (for different use cases)

@soxofaan ok. I was thinking about the implications of this.

In load_result, we can simply assume that if the id starts with http(s)://, it's a signed URL and we can point our STAC client there; if it doesn't, it's a batch job ID and we load the results directly instead of invoking an HTTP request.

In the case of load_stac, doesn't this mean that we have to detect whether or not the URL is our own and act accordingly? In this case: pass auth headers if it is. But if we can determine that the URL is our own, we might as well parse the job ID from it and avoid HTTP again.

Thoughts?

[soxofaan]

But if we can determine that the URL is our own, we might as well parse the job ID from it and avoid HTTP again.

Indeed, that was also my thought: if we detect a static (non-signed) job result URL, we just have to check that the current user owns the referenced job (which is equivalent to go full HTTP route with auth header stuff)

[bossie]

It looks like that, at some point in the code, we have to determine whether the provided URL represents a static STAC catalog (incl. batch job results) or a dynamic STAC API Collection (which supports search requests).

A possible way to accomplish this seems to consist of:

• fetching the STAC Collection document that the URL points to;
• fetching the Collection's root Catalog by navigating to its "root" link;
• looking at the root Catalog document's "conformsTo" property to determine whether it's a STAC API that supports item search;
• if it does, we can send search requests to the Catalog's "search" URL for this collection; otherwise it's a static Catalog and we browse it.

@m-mohr does this approach seem right to you?

[m-mohr]

Sounds reasonable.

[bossie]

With the temporary catalog workarounds applied, this batch job runs successfully and produces 3 GeoTiffs with 3 bands each:

connection = openeo.connect("openeo-3-1.openeo-vlcc-prod").authenticate_oidc()

data_cube = (connection
             .load_stac(url="https://geoville/resto/collections/BVLPROBA_v1",
                        spatial_extent={"west": 20.579494542018466, "south": 54.31120577537291,
                                        "east": 20.631426035739594, "north": 54.33263361375995},
                        temporal_extent=["2019-01-01", "2021-12-31"],
                        bands=["band1", "band2", "band3"])
             .save_result("GTiff"))

job = data_cube.execute_batch()
job.download_results("/tmp")

# openEO_2019-01-01Z.tif
# openEO_2020-01-01Z.tif
# openEO_2021-01-01Z.tif

[bossie]

As discussed: seeing as client-side filtering on Item properties tends to work better than the STAC API Filter extension (400 "Unknown property in filter" most of the time) it was decided to postpone this TODO.

[m-mohr]

Are you aware of the queryables endpoint, which you can use to retrieve the available properties for the filters?

[bossie]

This particular STAC API returns "additionalProperties": true but still rejects most properties.

[m-mohr]

Is it a public API? If yes, which?

[bossie]

The API is not public.

[jdries]

done!