[WIP] Generate links in api layer

[geospatial-jeff]

Related Issue(s):

• Split backends into independent repos #432
• Related to Improve separation of concerns between core and backend libraries #453

Description:

Moves link generation into the API layer to create better separation of concerns between API/backend libraries. Links are split into two categories:

1. Inferred links (self, root, parent, child etc.) are dynamically generated by the API layer based on the request path and the stac object returned by that request.
2. Backend specific links (next, prev, links to external resources etc.) may be returned by backend implementations and resolved by the API layer - urljoin(base_url, link['href']). The current landing page is a good example of this; links to children collections are generated by the backend (Backend.all_collections) and resolved by the API layer. This allows backends to return their own links, and makes it easy for users of those backends to extend this behavior.

With this PR the starlette request object is no longer passed to the backend client as a key word argument and core.py no longer imports HTTP specific things like urllib.parse.urljoin, fastapi.Request, and stac_fastapi.types.request.get_base_url.

TODOs

• Check the API spec to make sure we are returning recommended links for each endpoint.
• Make sure media types are correct.
• Figure out a good way to differentiate between the two types of links mentioned above. The API layer should override any inferred links returned by the backend and resolve + pass through any other links.
• Write tests.

PR Checklist:

• Code is formatted and linted (run pre-commit run --all-files)
• Tests pass (run make test)
• Documentation has been updated to reflect changes, if applicable, and docs build successfully (run make docs)
• Changes are added to the CHANGELOG.

[geospatial-jeff]

A few ideas for differentiating between links:

• Reserve a handful of link relations to be used for inferred links. Some link relations like self are obviously inferred but there is some gray area. For example the data relation is currently used by the landing page for /collections but data could easily apply to other links too.
• Require that backend implementations return additional link members indicating if the link should be inferred. The API layer would strip this additional link member from the payload before returning the API response to the caller. For example:

{
    "rel": "next",
    "type": "application/geo+json",
    "method": "GET",
    "href": "/search?token=12345",
    "inferred": false,
}

• Define clear precedence for which link takes priority. Any links dynamically generated by the API layer would take precedence over links returned by the backend to ensure routing is correct. Any links returned by the backend but not generated by the API layer are passed through to the response. All links are resolved using the base url.

Third bullet is my preference. Note that I haven't thought too much about how children/browsable conformance plays into this as we haven't implemented it yet.

Link Resolution

Backends need to return absolute links when referencing external resources like licenses or citations. We must distinguish these links from relative links. Using urllib.parse.urlparse(url).scheme seems to work well for this:

from urllib.parse import urlparse

>>> urlparse("/collections")
ParseResult(scheme='', netloc='', path='/collections', params='', query='', fragment='')

>>> urlparse("s3://bucket/key_prefix/object.txt")
ParseResult(scheme='s3', netloc='bucket', path='/key_prefix/object.txt', params='', query='', fragment='')

>>> urlparse("https://science.nasa.gov/earth-science/earth-science-data/data-information-policy")
ParseResult(scheme='https', netloc='science.nasa.gov', path='/earth-science/earth-science-data/data-information-policy', params='', query='', fragment='')

[bitner]

I definitely agree with the third option using precedence.

The other thought I had related to ability to control extensions that add additional links is to use the request.scope['routes'] and request.scope['route'] to introspect the App (same as how the OpenAPI docs are created - and there are some examples there that we could use) and to include any links that are at the next level up as the "self" link. This would mean that when requesting "/" endpoint, that it should auto-discover the "/conformance" endpoint and that it would be able to scan the routes and if the FiltersExtension was enabled, it would also find the "/queryables" endpoint. The mime_type would be able to be found on the response_class.

[geospatial-jeff]

use the request.scope['routes'] and request.scope['route'] to introspect the App

I think this is a really good idea, I can play around with it in another PR.

[gadomski]

Closing as OBE #555.