$IPFS_PATH/gateway for user choice of preferred gateway

[markg85]

Rationale

Currently IPFS doesn't expose any way to know if a gateway is running and what it's URL is. One could opt to use the IPFS API to figure this out but that's quickly rather complicated to implement properly. Providing a file indicating that a gateway is running where the content of the file is the gateway URL would be a simple portable way for applications to detect a running IPFS gateway.

Potential usecases

• In my project to support IPFS in FFMpeg i need to know the gateway otherwise the user must always provide it manually. Reading a file would make this one step more user friendly in this case.
• Another potential user for this could be Brave (the browser). It would allow them to detect a running gateway.
• IPFS Companion also could benefit from this approach.
• There's likely many more potential usecases that could benefit from this too.

Implementation

1. When the gateway part of the IPFS daemon starts, a file named gateway is created. This file is created in $IPFS_PATH. The resulting file will be $IPFS_PATH/gateway
2. The content of this file is the full URL to the gateway without a trailing forward slash. In a default setup scenario the content of the gateway file will be http://localhost:8080. The port section of this URL will follow whatever is provided in the IPFS config file under Addresses.Gateway
3. When IPFS stops (assumption, the general graceful shutdown, not a crash) the gateway file is removed.
4. If the Addresses.Gateway setting is empty or missing from the config file then no gateway file is created because no gateway should be started. This and only creating the file when the gateway starts should - more or less - guarantee that the file only exists if there is a gateway running.

How other applications can use this

Assumption, they do need to assume where the IPFS folder is.

1. They should check for SIPFS_PATH and use that if it exists. Otherwise they should assume ~/.ipfs. They can expect the gateway file to be in that folder. If the gateway file is there it means a local gateway is running.
2. If the path exists but no gateway file does then it's safe to assume no gateway is running from that IPFS data folder.
3. If they need to know the gateway address they can rely on the full content of the file being a single URL.

Open and unresolved issues

1. Ideally it should be able to detect a running IPFS instance when it's running as a service under it's own username. This spec does not handle that. mDNS could potentially be used for that though it might not be ideal for all potential uses.
2. In browser IPFS nodes (like what's possible in Brave). How to even figure out where it's gateway runs?
3. More?...

Code and testcases

Right now there is no code nor testcases. I am planning to make both and will post a message in this issue when i'm actually working on this. You can assume i'm not working on this as long as i haven't explicitly said so in this issue.

Thank you and mentions

Thanx goes out to @Stebalien and @TheDiscordian for the fruitful discussion we had about this!

[aschmahmann]

This proposal makes a lot of sense to me, thanks! A couple questions/clarifications about some edge cases:

1. It's currently possible for applications like go-ipfs with it's client <-> daemon architecture to have the daemon running on a different machine than the client and the api file points to where the API is. It seems like users should be able to manually create a gateway file similar to the api one for forwarding gateway requests even if they're not to localhost.
2. It seems like running the daemon should clobber gateway files (e.g. if manually configured or left behind by an abrupt shutdown) with whatever is the latest information from the config

[markg85]

1. It's currently possible for applications like go-ipfs with it's client <-> daemon architecture to have the daemon running on a different machine than the client and the api file points to where the API is. It seems like users should be able to manually create a gateway file similar to the api one for forwarding gateway requests even if they're not to localhost.

That should still be possible. It's a matter of config though. If the user on the client side doesn't define a gateway (aka, removes it from the config) then the file should stay however the user created it.

2. It seems like running the daemon should clobber gateway files (e.g. if manually configured or left behind by an abrupt shutdown) with whatever is the latest information from the config

Yup :) See my reply above for the manual part. As for leaving artifacts files after IPFS crashes.. I don't know how to properly resolve that. The real fix would be to not crash ;) but yeah, there will undoubtedly be cases where the file exists and the gateway isn't running.

[Stebalien]

Yup :) See my reply above for the manual part. As for leaving artifacts files after IPFS crashes.. I don't know how to properly resolve that. The real fix would be to not crash ;) but yeah, there will undoubtedly be cases where the file exists and the gateway isn't running.

I think the answer is: try to contact that gateway and, if it fails, fall back on something else. Basically, the "which gateway" function should return a ranked list of gateways to try.

[markg85]

I have now implemented this logic in my ffmpeg patch.
You can see it here: https://github.com/markg85/FFmpeg/blob/ipfs/libavformat/ipfs.c

The order of precedence:

1. Define a -gateway <url> to the gateway without trailing forward slash.
2. Define $IPFS_GATEWAY with the full http link to the gateway without trailing forward slash.
3. Define $IPFS_PATH and point it to the IPFS data path.
4. Have IPFS running in your local user folder (under $HOME/.ipfs).

The -gateway option takes precedence. If it's set it's used. The rest is ignored.
If it doesn't the next step is executed. It looks at $IPFS_GATEWAY and uses that if it exists.
If that also doesn't exist then it looks at $IPFS_PATH and tries to compose the path to the gateway file. If that file exists and has content then the first line of that file is used as gateway. All this logic is in https://github.com/markg85/FFmpeg/blob/ipfs/libavformat/ipfs.c in the ff_populate_ipfs_gateway function.

If a change in this order is preferred, now is the time to respond :)

While, in strict terms, this doesn't have an effect on IPFS yet. If this is included in ffmpeg it does mean that IPFS now has at least an "unwritten standard" for those environment variables.

[lidel]

@markg85 would you be interested in opening a complimentary PR against go-ipfs to create $IPFS_PATH/gateway every time ipfs daemon starts? Does not need to be fancy, just be aware Addresses.Gateway can be either a string or array of strings (if the latter, pick the first addr from the list).

• $IPFS_PATH/gateway and IPFS_GATEWAY will simplify a bunch of code in ipfs-desktop, and other tools in ecosystem (e.g. ipfs-or-gateway)
• potential place for spec of those conventions may be an appendix to the gateway spec (until then, keeping this issue open).

[markg85]

Hi @lidel,

Yes, definitely! It's bit more challenging to me though as i know 0 about the whole go stack and the language (which looks weird to me). But it feels like a nice small manageable feature to get myself a little more comfortabele in Go.

I'll post an update here when i start on that. In the meantime, i'm definitely not offended if anyone else wants to take it so please do be my guest.

[markg85]

I just started working on this!

Or rather, i'm diving into Go to figure out how that language works and use this small project as my example :)

[lidel]

@markg85 hey, any chance you found time to poke at it, or is it ok for someone else to give it a try?

I think adding it in this function will be a good start, we can refine it later.

[markg85]

Actually, i prefer if someone else could give it a shot.
I'm working on a lot of different things right now (some for that ffmpeg project, some for my other work). Mixing in go is a bit of a low priority for me at the moment.

That being said, i would really really love to have this in the next version of IPFS.
If someone else wants to poke it, that would be awesome!

[lidel]

Cool, I'll see if I can find some way to move it forward.
We had discussions about ipfs:// resolution in curl and ffmpeg and having this file would allow such user agents to:

• leverage local node, when present
• ask for gateway address(es, one per line), when missing, linking to https://github.com/ipfs/public-gateway-checker or similar
  • this would solve the problem of hardcoding specific gateway as "the default", removing the privacy concerns around The Tyranny of the Default

[markg85]

I might be able to handle the curl side of things. I did talk to @autonome about that. Now with those ffmpeg patches i did learn one very hard lesson. It's C and i really don't like C. He's "knocking on some doors" to find someone suitable to add it in curl. If he can't find anyone this month then i'm the backup and will handle it next month.

Question though. You changed the title to include "agency"... Could you elaborate on the meaning? I can't figure out what you mean by that.

[lidel]

Meant user being in control of which gateway is used :)

[SgtPooki]

Yup :) See my reply above for the manual part. As for leaving artifacts files after IPFS crashes.. I don't know how to properly resolve that. The real fix would be to not crash ;) but yeah, there will undoubtedly be cases where the file exists and the gateway isn't running.

I think the answer is: try to contact that gateway and, if it fails, fall back on something else. Basically, the "which gateway" function should return a ranked list of gateways to try.

This is tangentially related to ipfs/public-gateway-checker#192 & ipfs/public-gateway-checker#61. My below thoughts don't follow the above quote's lines of thought, but I wanted to bring up something this discussion made me think about.

I'm not sure it makes sense to keep a list of gateways.json in an npm package(see ipfs/public-gateway-checker#192), and go, and rust, and....etc. Especially when there needs to be multiple gateway-checking dependent features across multiple implementations in multiple languages.

Do folks have thoughts on how we could centralize our gateway list in a sane way that allows importing across languages? Does it make sense to store a curated list of "official" gateways on IPFS and reference with it's CID? That screams of cyclical dependency issues to me.

[NCGThompson]

Perhaps $IPFS_GATEWAY and $IPFS_PATH/gateway should have a "none" option? It will signal to an application to stop searching for gateways and fail instead. For example if $IPFS_GATEWAY is set to none, then $IPFS_PATH/gateway will not be checked.

[NCGThompson]

Since #9156 has been merged, should this issue be closed?

[markg85]

Closing this one.
The original request has long been implemented and is there in IPFS right now.

I am right now turning this into an IPIP spec which you can follow here: ipfs/specs#280

It's possible gonna lead to a new change in IPFS to get things "according to spec" ;)