Search/Sort/Order for jobs, services, files, process_graphs

[m-mohr]

I'm wondering whether it could be useful to introduce searching (adopt OpenSearch parameters?) and pagination to the collections, i.e. for the list of jobs, services, files, process_graphs? File and job lists may get very long, I assume.

[m-mohr]

Idea for pagination: Pagination should not be used by default. If a client wants to use pagination it needs to send a parameter such as "count" (number of elements) or "page" (page number) with the request. In this case the endpoint returns the requested subset of elements and adds a link with next (and prev) rel types to the links array. The clients can follow these links to the other datasets. If a server does not support pagination, it should return an error such as "PaginationNotSupported" and the client can decide whether it tries again without pagination or not. Returning the full list of elements instead of an error is critical as some applications may have problems with big datasets, which may lead to a slow down/crash/... in the worst case.

As client-side searching/sorting/ordering only works with all elements (i.e. it doesn't work with pagination) this needs to be enabled server-side and therefore needs to be part of the API.

[m-mohr]

Related issues:

• standardize requesting a specific page/index in search api radiantearth/stac-spec#381
• Paging improvements opengeospatial/ogcapi-features#23
• startIndex opengeospatial/ogcapi-features#75

As we are already using STAC/WFS3 we can go the same way. Call the parameter limit and use it to request paginated data. Use next links (optionally also prev) to pagination. Allow startIndex as an extension (though we could also skip that for simplicity).

[m-mohr]

Discussed in 3rd year planning:

• We plan to integrate pagination. By default all data is returned, if a limit parameter is set the number of results is returned.
• Idea was also to return the count of pages somehow so that a number of pages can be calculated.
• Pagination by page numbers or next/prev links?

No searching/filtering for now, that includes no OpenSearch (see #68) for now. Searching/filtering (incl. support for OpenSearch), may be added in the future whenever people express interest for it. OpenSearch could be added s an extension as proposed in #120.

Edit: Depends on results in STAC sprint #5 so that we have a consistent interface.

[m-mohr]

The OAFeat-based endpoint /collections doesn't support pagination, only /collections/:id/items supports it. As it's quite useful for several purposes to know all collections and processes anyway, I'll check whether we only add pagination for non-discovery data (jobs, process graphs, services, files).

If we change process graphs to be stored with the processes (depends on #161), we'd need it though for processes and so we could also add it to collections for consistency.

[m-mohr]

Pagination is solved in 1.0, but there's still room for adding sort/order/searching. We should closely follow STAC and OGC's CQL.