Webui not correctly linking to local gateway when running over LAN at .local domain #2031
Comments
chrisguida
added
kind/bug
chrisguida
changed the title
chrisguida
changed the title
|
See our discussion on discord, this isn't a setup we support. The best quick win that would solve your problem is webui trying |
|
@Jorropo I don't understand what you're saying. You literally told me to open an issue about this in our Discord conversation:
What do you mean "webui trying |
|
Also, if I can't solve this, then that means embassy-os users will be unable to use IPFS. I am willing to contribute a PR to fix this if needed. Please don't just dismiss this. |
|
I am moving this to ipfs-webui repo, but keeping it closed as this is not supported setup, so not actionable for maintainers. @chrisguida I am not familiar with embassy-os, so apologies if some suggestions are not feasible, or you already know these things, but here are some quick thoughts (jump to 👉 for a potential fix) which may help you self-fix your custom setup:
|
|
@lidel thanks so much for the response! I will work through this next week and get back to you :) |
|
Hey @lidel thanks again for getting back to me. Apologies for the delayed response.
Does this mean that using two different domains for the API and Gateway will not work? I can put them on the same domain if so.
Yes, I understand, this is not a problem since .local is the TLD designed for mDNS, which only works over LAN. Obviously since embassy-os is designed to run on a personal server, and not on the user's local machine, we cannot use
That's ok; as I said, I'm willing to contribute the work necessary to add this functionality to IPFS. Again, I think allowing users to run IPFS nodes from an always-on server in their homes is a uniquely beneficial use case of IPFS. It allows the user to fully contribute to the IPFS network by hosting files for others; it also allows the user to access IPFS in a fully self-sovereign way, as they don't need to rely on public gateways. But perhaps there are other ways to achieve this.
It's more of a soft requirement that will save dozens of hours of dev work and customer support. Embassy-os only supports access to services via .local when connecting via LAN, and over .onion when connecting remotely. We are planning to add support for direct IP access over LAN, but for now .local/mDNS is the only officially supported method of accessing services over LAN. But again, this is a soft requirement; if I can find a way to connect to the gateway using an IP instead of a domain, that is much better than no gateway access from the webui at all. I can simply hack in a workaround in embassy-os for now if IP addresses will work.
Correct, we need to access the .local domain over HTTPS, and the webui is using http; that is one of the problems that we need to fix or work around.
Ok, I can try this and see whether it will be too cumbersome. The goal of embassy-os is to provide a large marketplace of services that are plug-and-play for people who are new to self-hosting. Asking them to create an SSH tunnel in order to use the gateway will most likely result in a lot of support load, as the majority of our users don't have SSH access, but we'll do it if there's no other way.
Excellent, I suspected this, but couldn't find any online resources explaining this behavior. Thanks for confirming :)
This was from a suggestion somewhere (I'll try to find it) which recommended disabling
Ok, thanks for the info. I haven't tried using an array of addresses yet; I've only tried setting it to single strings, which have all failed in one way or another.
I did try your top example (as a string, not as an array with
I doubt there is an IP address which is both listenable and dialable by It sounds like there are 5 potential fixes to this issue:
I will attempt 1-3 and get back to you. It sounds like you guys are reluctant to look into 4-5, which is sad since I think this would be not too much work for a lot of gain. Again, I am willing to take responsibility for these enhancements myself, as long as the IPFS community can help answer my questions. Thanks for helping me out on this one, I really appreciate it :) |
|
I think it would be useful for you to describe your use case. What is the end user experience you are designing for? FWIW it is perfectly fine to run IPFS Node, or even HTTP Gateway in your LAN. What is tricky is to provide secure access to RPC port (and GUI like ipfs-webui). SSH is the best solution IMO, because it removes surface for footguns,
No, I don't think you need same domain or port, technically it should work (even
Ah.. right. Sadly Kubo's HTTP is designed as a backend service: it is listening on cleartext HTTP ports 8080 and 5001: they do not provide TLS or anything fancy like that, it is just HTTP 1.1. Kubo does not support listening on HTTPS, for that you need to put some reverse proxy like Nginx or Caddy in front of it to correctly terminate TLS. Unfortunately, Kubo won't be able to listen on the final TLS address, so ipfs-webui will never be able to use the correct one.
Yes,
Are we talking about changing ipfs-webui or Kubo? I would be open to review a PR (remember to @lidel me) against ipfs-webui that is addressing (4) – it sounds like a generic bug on Files screen. If I also could see how passing static API and Gateway addresses via URL ( If you had other fix in mind, lmk. As for (5), this is unlikely.. Kubo bending backwards just for ipfs-webui. Changing existing behavior is unacceptable as it will lead to silent errors that are annoying to debug, especially in cloud contexts. If anything, we could add a separate setting, just like we have I think what we could add to Kubo is a list of preferred public gateways – we need it for unrelated, but already planned work described in ipfs/kubo#9159 – if that lands, ipfs-webui could opportunistically read this list and use it too. Lmk if you were successful with any of the paths discussed. |
|
Reporting back with some rest results: I've tried using an array of strings instead of a string (potential solution number 3), and as expected that does not work, since Kubo cannot dial the .local address:
So potential solution number 3 is a no-go. I've also tried forwarding port 8080 from the container to the host like so (potential solution number 2):
Then I set
and also simply:
But still got a "cannot listen" error:
This was attempted before and after activating the For the record, the The SSH tunnel (potential solution number 1) works, I can confirm:
This forwards both the API and Gateway to |
Basically embassy-os consists of an OS and a marketplace of open source software that has been packaged for said OS, not unlike apt, but without the need for the user to touch the command line ever, and without non-service packages; that is to say, all our packages are some kind of self-hostable server software, like bitcoin, lightning, mastodon, synapse, gitea, ghost, vaultwarden, filebrowser, etc. Here's what we currently have on offer, though this collection is rapidly expanding. Any software that can be self-hosted and is open source can be packaged as an "app" for embassy-os (including IPFS of course). Our users are fanatics about privacy and digital independence, and love hosting their own data. In addition to this, the services we host are designed to require no upfront configuration on the part of the user. They can simply navigate to the marketplace, click "install", then "configure", hit "save", then "start", and the service will set itself up and a health check will alert the user when the service is ready, for example, to visit in a browser (if it has a ui). Then they hit "launch ui" and they're using the service. No command line is required, and indeed no configuration is required. Of course the user can configure their service in any way that the package is designed to be configured (by the service package dev). Every service is given either a system-generated .local address (for clearnet LAN access) or .onion address (for onion-routed remote access), or both. We have over 1,000 users now (see our telegram channel), and a large number are eager to start running IPFS on their nodes, with 24/7 uptime, to aid the IPFS network, and also to be able to host, pin, and access their files privately over their own local gateways. IMHO, IPFS could benefit a lot from designing with this use case in mind.
I definitely agree!
Yes, we are considering adding a reverse proxy with basic auth to the container. This would work better as a feature in the webui itself, but I'll see if we can get it done. Adding a reverse proxy to IPFS has been somewhat of a challenge for me so far.
It's not ideal for us because we emphasize that our users won't need to touch the command line, and our users are used to services being plug-and-play with no additional setup required.
Ok great, thanks
Yes, we have a reverse proxy in the OS that takes care of TLS, but it seems like we may need to add another reverse proxy inside the IPFS container to handle auth. Though the API/webui will still only be visible over lan, and over .onion as a Tor hidden service, neither of which will be accessible by the public. But adding auth will definitely make things more secure.
Yes, that is what it looks like. Unfortunate indeed.
Whichever you think is more appropriate. It seems as though the webui should be able to respect the client "public gateway" settings, because as I mentioned previously, it already does this for "click to view", but not for "download". This actually just seems like a bug in the webui, and seems like the least invasive fix for this issue. But, if you think the Go code needs to be modified to separate Kubo's listening address from the webui's connect address, then I'll do that. That also seems likely to be doable in the webui code. The fact that Kubo cannot listen on an address that it itself cannot connect to just seems.. wrong. But I'm new to this codebase so I'm not aware of all the reasons why the design was left that way, so perhaps they were good reasons.
Yes, this seems like the place that makes most sense for us to start.
Yes, this seems like it could be useful.
Again, this strikes me as a design flaw in Kubo. This behavior really makes it seem like this reference IPFS node implementation was not designed with running a on server device (separate from the client it is accessed from) in mind, which seems antithetical to IPFS's mission. Doesn't IPFS want people to run nodes in as many configurations as possible, so that there can be as many nodes as possible? Doesn't that make the data stored on IPFS that much more uncensorable? And isn't censorship resistance the whole point of IPFS?
This sounds like it would work! Let me know if this is what I should do.
Also a great suggestion! I'm down to work on whatever solution you think is best, as long as it helps onboard as many of our users to IPFS as we can :)
Posted earlier with my test results, I didn't find any optimal solutions to the problem. I really appreciate your being here, it means a lot! I'm sure we'll come up with something that works for everyone :) |
|
Ok so I just realized that the IPFS Companion browser extension possibly fixes this. I was using it before and then it stopped working. I removed it and re-added it to my browser and now everything works, as long as I set the local gateway and toggle on "use local gateway". Will continue testing and report back! |
|
Ok never mind, enabling IPFS Companion fixes the "download" behavior, but breaks the "click to view" behavior xD Will see if there's some combination of webui and companion settings that fixes this. |
Checklist
Installation method
built from source
Version
Config
# ipfs config show{ "API": { "HTTPHeaders": { "Access-Control-Allow-Methods": [ "PUT", "POST" ], "Access-Control-Allow-Origin": [ "https://djnofp2u5mvjsewk7anxsj4iep3n5uvez362lek4ixjysgeefk5esxyd.local", #api/webui domain "http://localhost:3000", "http://127.0.0.1:5001", "https://webui.ipfs.io", "http://ipfs.embassy:5001", "https://rfu6pj36capximdisyvwxyjnhlwyeovf4gcmz4hvuptaetwduslav3id.local", #gateway domain "https://0.0.0.0:8080" ] } }, "Addresses": { "API": "/ip4/0.0.0.0/tcp/5001", "Announce": [], "AppendAnnounce": [], "Gateway": "/ip4/0.0.0.0/tcp/8080", "NoAnnounce": [], "Swarm": [ "/ip4/0.0.0.0/tcp/4001", "/ip6/::/tcp/4001", "/ip4/0.0.0.0/udp/4001/quic", "/ip6/::/udp/4001/quic" ] }, "AutoNAT": {}, "Bootstrap": [ "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ", "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ", "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN", "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa", "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb", "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt" ], "DNS": { "Resolvers": {} }, "Datastore": { "BloomFilterSize": 0, "GCPeriod": "1h", "HashOnRead": false, "Spec": { "mounts": [ { "child": { "path": "blocks", "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2", "sync": true, "type": "flatfs" }, "mountpoint": "/blocks", "prefix": "flatfs.datastore", "type": "measure" }, { "child": { "compression": "none", "path": "datastore", "type": "levelds" }, "mountpoint": "/", "prefix": "leveldb.datastore", "type": "measure" } ], "type": "mount" }, "StorageGCWatermark": 90, "StorageMax": "10GB" }, "Discovery": { "MDNS": { "Enabled": true, "Interval": 10 } }, "Experimental": { "AcceleratedDHTClient": false, "FilestoreEnabled": false, "GraphsyncEnabled": false, "Libp2pStreamMounting": true, "P2pHttpProxy": false, "StrategicProviding": false, "UrlstoreEnabled": false }, "Gateway": { "APICommands": [], "HTTPHeaders": { "Access-Control-Allow-Headers": [ "X-Requested-With", "Range", "User-Agent" ], "Access-Control-Allow-Methods": [ "GET" ], "Access-Control-Allow-Origin": [ "*" ] }, "NoDNSLink": false, "NoFetch": false, "PathPrefixes": [], "PublicGateways": { "localhost": null, "rfu6pj36capximdisyvwxyjnhlwyeovf4gcmz4hvuptaetwduslav3id.local": { "Paths": [ "/ipfs", "/ipns" ] } }, "RootRedirect": "", "Writable": false }, "Identity": { "PeerID": "12D3KooWPCNH8gBqGuuCxerv3wLEjBEEWHYJXxa8eUcsNSjLS6cm" }, "Internal": {}, "Ipns": { "RecordLifetime": "", "RepublishPeriod": "", "ResolveCacheSize": 128 }, "Migration": { "DownloadSources": [], "Keep": "" }, "Mounts": { "FuseAllowOther": false, "IPFS": "/ipfs", "IPNS": "/ipns" }, "Peering": { "Peers": null }, "Pinning": { "RemoteServices": {} }, "Plugins": { "Plugins": null }, "Provider": { "Strategy": "" }, "Pubsub": { "DisableSigning": false, "Router": "" }, "Reprovider": { "Interval": "12h", "Strategy": "all" }, "Routing": { "Type": "dht" }, "Swarm": { "AddrFilters": null, "ConnMgr": { "GracePeriod": "20s", "HighWater": 900, "LowWater": 600, "Type": "basic" }, "DisableBandwidthMetrics": false, "DisableNatPortMap": false, "RelayClient": { "Enabled": true }, "RelayService": {}, "Transports": { "Multiplexers": {}, "Network": {}, "Security": {} } }, "Transports": { "Network": { "Relay": true } } }Description
I am trying to package IPFS for embassy-os. See this discussion forum post for some background on embassy-os.
kubov0.15.0 in a docker container on a raspberry pi on my home network using https://github.com/chrisguida/ipfs-wrapper. The Dockerfile here is pretty much the same as the Dockerfile included in thekuborepo, with some slight modifications to adapt it to .kuboas in this entrypoint.ipfs.iopublic gateway sometimes displays the file after a long wait. Other times, if I open the browser console I can see this error:(As you can see, the browser does not like that the request to
http://0.0.0.0:8080is cleartext HTTP, while the domain I'm viewing the webui through is HTTPS ("https://djnofp2u5mvjsewk7anxsj4iep3n5uvez362lek4ixjysgeefk5esxyd.local" in the above example config). Actually, I'm not even sure whathttp://0.0.0.0:8080is supposed to me. I do know that if I runkubolocally on my laptop, this request succeeds. How firefox is able to resolve this URL, ever, is beyond me.)But since the point of embassy-os is to self-host, we want to avoid "public" gateways and use the gateway on our own device, which responds much faster anyway. If I go to the webui "Settings" tab and enter in my gateway address ("https://rfu6pj36capximdisyvwxyjnhlwyeovf4gcmz4hvuptaetwduslav3id.local" in the example config) into the "public gateway" field, clicking on files now works, and sends the files through my local gateway properly.
Issue ipfs/kubo#1: I would like to be able to configure this setting programmatically, ie in
docker_entrypoint.sh. Requiring the user to manually find their gateway domain and input it into this settings field adds a lot of friction to the UX. However, that can be worked around for now if it's the only way. The real bug is below:Issue ipfs/kubo#2: When I click "Download" on any file, even after manually setting the gateway in the webui, the browser still requests
http://0.0.0.0:8080, rather than the local gateway I've configured. This of course results in an error. This makes it so that the user cannot download a file through the UI. The bizarre thing is that if I manually alter the download URL to replacehttp://0.0.0.0:8080with the local gateway URL ("https://rfu6pj36capximdisyvwxyjnhlwyeovf4gcmz4hvuptaetwduslav3id.local" in the example config), the file can be viewed through the browser just fine. For some reason, the webui is unable to simply link the user directly to the local gateway, even when it is specifically configured in the webui.It's possible I'm missing a config value somewhere, as I'm fairly new to
kubo, but this definitely seems like at least a documentation issue, and probably a bug in the webui.Please let me know how I can help debug/fix this, as our community is asking for access to IPFS on embassy-os!
Thanks in advance,
--Chris
The text was updated successfully, but these errors were encountered: