configurable redirecting and remapping

[elf-pavlik]

#107 (comment)

@TallTed: Apache, nginx, etc., are configurable to redirect from (or deliver without actual redirection) / to /index.html or /index.php or /arbitrary.handler or whatever. (It's not just about index.*!) This is neither default, nor universal, behavior -- it is the choice of the server (or website, or directory) admin -- but the discussion I'm seeing seems to be trying to make the decision once and for all, and appears to be largely debating between people who would make different choices on their apache/nginx/whatever instance ... which would be OK if it's actually turned into a configuration option, but will remain a problem if we make it hard-and-fast as the way Solid works.

I think defining something like Apache .htaccess as part of LDP container metadata solid/data-interoperability-panel#32 would provide general and explicit mechanism which would help us avoid some arbitrary implicit behavior. I think it should at least allow to:

• defining redirecting with HTTP 3XX codes
• define remapping with HTTP 2XX codes and Content-Location Header

It could possibly have RDF version of Apache mod_rewrite with equivalents of RewriteCond (HTTP_ACCEPT, HTTP_ACCEPT_LANGUAGE etc.) and RewriteRule.

This would allow explicit remapping for discussed in #69 case of serving HTML from URL denoting LDP Container, without arbitrary convention like index.*. Still implementations could opt-in to include index.* as default remapping rule while allowing applications with sufficient access to overwrite it for specific container in solid storage.

Very likely it could also provide configurable way to address discussed in #107 redirection for IRIs with or without trailing /.

[csarven]

It is very early to tell as to what aspects and degree of this proposed functionality should be part of the spec and be testable, as opposed to leaving it as an implementation detail. Before that happens, we need multiple implementation experience (independently developed, different codebase..) in order to understand what's reasonable to implement; common patterns..

[poef]

For those interested, we're working on a specification and server implementation of redirection, for the case of data that has moved or been deleted, at PDS Interop (https://pdsinterop.org/solid-migrator/). We're looking at tying it into the .meta mechanism (auxiliary resources). We're also looking at the timeshift aspect, to support archived copies of data as a fallback.

We have two solid server implementations, the solid-nextcloud bridge and a reference PHP solid server, where we can test the specification.

Any comments or directions on how to continue on working on the specification proposal are much appreciated.

[elf-pavlik]

We're looking at tying it into the .meta mechanism (auxiliary resources).

Just noticing that .meta is just some implementation-specific detail. Defined link relations are the specified way to discover the auxiliary resource of a given type.

https://solidproject.org/TR/protocol#auxiliary-resources

Clients can discover auxiliary resources associated with a subject resource by making an HTTP HEAD or GET request on the target URL, and checking the HTTP Link header with the rel parameter [RFC8288].

[sjoerdvangroning]

Team PDS Interop has done work that could give an option to resolve this issue.

Principal ideas and specification
We are looking for feedback on our discussion and consideratins. Would be good to know if this is a good idea. In the server specification, we focus on the redirecting of URL, in the ontology, it is also possible to redirect all resources, which makes it possible to redirect only a chapter like storageserver.tld/book#chapter.
Discussion & considerations: https://pdsinterop.org/solid-link-metadata/
Ontology: https://pdsinterop.org/solid-link-metadata/links.ttl (containing redirects, archiving, integrity checks)
Specification: https://pdsinterop.org/solid-link-metadata/solid-storage-specification/ (core ideas are in this specification, we hope that it is well perceived and could find some sponsors to get it to a next level)

Tests for Solid Storage server
Test: https://lm-tests.dev.muze.nl/
Source: https://github.com/pdsinterop/solid-link-metadata-tests
Demo video:https://solid-migrator.dev.muze.nl/demo/Test-Suite.mp4

Implementation done in Solid Nextcloud and PHP Solid server
What we would like to do next, is an implemenattion in the Community Solid Server, but that would require some positive feedback to affirm we are on the right path.
Updated SOLID-Nextcloud
Server URL: https://solid-nextcloud.muze.nl/index.php/login
Source: https://github.com/pdsinterop/solid-nextcloud

Main application: POD migrator
We created our Solid Migrator app, which makes it possible to move data between Solid storage servers
Application: https://solid-migrator.dev.muze.nl/
Source: https://github.com/pdsinterop/solid-migrator-app
Demo video: https://solid-migrator.dev.muze.nl/demo/Solid-Migrator.mp4

** Demo application: Zettelkasten**
We created a demo application to make it easy to verify the working of our ideas about redirection.
Demo application: https://zettelkasten.dev.muze.nl/
Demo video: https://solid-migrator.dev.muze.nl/demo/Zettelkasten.mp4
Source and explanation: https://github.com/pdsinterop/zett

[kjetilk]

I think the direction this is going in is pretty good, and aligned with the way that I have thought about this too.

However, we need to think more broadly about it than .meta. First, as @elf-pavlik said, .meta is just an example of an auxiliary resource, but you cannot rely on the name, you have to rely on a Link header to tell you where the auxiliary type exists.

Also, this particular auxiliary resource type, describedBy is removed when the resource is removed, so it can't be used for 410.

I think we should define another auxiliary resource type for this purpose, as part of a more general approach to auxiliary resources. It would, for example, need to live beyond the subject resource, or have its data merged into a resource higher up in the hierarchy. This would still use the basic principles that you've arrived at @sjoerdvangroning , but the details regarding the use of auxiliary resources would change.

[Potherca]

For the PDS Interop implementation we use .meta as a convenience but we understand and support the fact that actual auxiliary resource URI must come from a resources Link header.

Also, this particular auxiliary resource type, describedBy is removed when the resource is removed, so it can't be used for 410.

Because of this, and other reasons, the meta information must set on a parent of a resource, as the resource itself does not have to exist (only the metadata does).

This would allow, for instance, redirecting an entire pod, without any other content besides the metadata being present.

This is reflected in the specification. 👍

[poef]

@Potherca we did think about this, and added it in the 'server implementation' discussion section, I actually forgot to add it to the specification. So @kjetilk thanks for the reminder. I'll add an extra note explaining that you should fetch the auxiliary resource based on the LINK header of the parent folder of the resource you want to redirect or delete.

[kjetilk]

Because of this, and other reasons, the meta information must set on a parent of a resource, as the resource itself does not have to exist (only the metadata does).

Yup, but the parent may also be removed at some point, so that doesn't guarantee anything. I think we need a mechanism for such things to propagate up the tree towards root, but I'm not entirely sure how that should be done to scale well.

[Potherca]

Yup, but the parent may also be removed at some point

I think there are two separate issues. A removed resource can not have metadata (as the resource does not exist) so the metadata can only be set on a parent resource (up to the root).

When a parent is removed that contains metadata, what should happen to the metadata that parent contains?

I don't recall anything currently in the Solid spec addressing this... There might already be something in the LDP spec, I'd have to take another look.

In the PDS Interop specification there is a related expectation that the metadata of a resource needs to be amended, for instance, when a resource is created again. Something similar would also be needed for when a parent container is removed (or renamed, maybe?).

we need a mechanism for such things to propagate up the tree towards root

I agree. I think we need to write out the usage scenarios first such situations and see what options are available, and which makes most sense.

@poef Should we make issues for this in the spec repo to include these scenarios?

[acoburn]

The combination of an external Memento timegate would likely suffice in addressing this case without the need to extend or change the Solid (or LDP) specifications.

Using pattern 2.1, a client can interact with a stable timegate resource. By providing either the "old" resource URI or the "new" resource URI, the client will be redirected to the correct location.

[poef]

@kjetilk @acoburn The position we've taken in the specification is that the last change indicates the user intent. So if you delete a folder where a child has redirect information, that information is no longer relevant. This is the base line. However I could see how a specific implementation might want to warn a user that this redirect information would be deleted as well, and could give the user the option to hoist the information up to the nearest parent that still exists.

As for memento, we have done extensive research on existing options to mitigate linkrot, memento was part of that. This is reflected in the link-metadata ontology, which has both #archive and #archiveDate predicates. We did not add these to the solid storate server specification for two reasons. I would like to keep the specification as small and simple as possible, and I think this is more useful for a dataset that links to an external resource. Here you can create an archive copy of the linked dataset and add the archive link and date to your own data. See https://pdsinterop.org//solid-link-metadata/#archive-and-archivedate for more information.

You can find our research results at https://pdsinterop.org/solid-migrator/