Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for DNS #2214

Open
wants to merge 25 commits into
base: py3
Choose a base branch
from
Open

Add support for DNS #2214

wants to merge 25 commits into from

Conversation

filips123
Copy link
Contributor

@filips123 filips123 commented Oct 1, 2019

This adds support for classic DNS. It closes #104 and it is part of #2049.


The plugin uses DNSPython library for getting DNS responses and DNSLink for resolving DNSLink records.

It uses a few DNS-over-HTTPS resolvers with OpenNIC TLD support, but it can also be configured to use other resolvers or system configuration. Results are also cached in files so domains will still work if the computer is offline.

There are already a few (my) domains that use DNS plugin which you can use for testing:

  • py.zeroframe.oss
  • js.zeroframe.oss

Note that because now plugin has additional dependencies, I had to change the configuration for Travis CI, GitLab CI, Docker and Vagrant to also install dependencies from (enabled) plugins. @HelloZeroNet ZeroBundle also needs to be updated to do this.

Also, DNSPython currently doesn't support DNS-over-HTTPS so I created PR rthalley/dnspython#393 with support for it. Until it is not merged and released, we have to use my temporary branch. Then you would need to update it to use a newly created version.

I also think that there this domain plugins aren't currently working if default Zeroname is disabled. I already reported this in #2128, when it was partly fixed (before that it was completely impossible to create domain plugin), but it appears that there are still problems if Zeroname is disabled. I wasn't able to test this with this plugin, but it needs to be investigated.

There are currently also some problems with CI. Tests works, but there is problem with linter (flake8). Because DNSPython needs to be loaded after Tor monkey-patching is completed, I needed to load them from a class method. I implemented that with importing modules inside a function and adding them to globals(). This works normally as modules are now availible at global context. However, flake8 doesn't like that and throws F821 undefined name errors. Is there any way to tell flake8 that dns and dnslink are actually existing names?


@HelloZeroNet @imachug @ValdikSS Can you test and rewiev this?

@filips123 filips123 mentioned this pull request Oct 1, 2019
plugins/DNS/requirements.txt Outdated Show resolved Hide resolved
plugins/DNS/DNSResolver.py Outdated Show resolved Hide resolved
plugins/DNS/SiteManagerPlugin.py Outdated Show resolved Hide resolved
plugins/DNS/DNSResolver.py Outdated Show resolved Hide resolved
plugins/DNS/DNSResolver.py Outdated Show resolved Hide resolved
plugins/DNS/DNSResolver.py Outdated Show resolved Hide resolved
@purplesyringa
Copy link
Contributor

Congrats, the tests are passing now.

@purplesyringa
Copy link
Contributor

@filips123 Should this PR be merged if it doesn't work without Zeroname plugin?

@filips123
Copy link
Contributor Author

filips123 commented Oct 1, 2019

@imachug It should be. Zeroname is still the default domain system and almost no one would disable it. But I think that this issue should be fixed quickly.

But I actually only tested this few months ago with ENS plugin (which uses same structure as this). I will also try to test this again now.

Update: I tested this again and I can confirm that it works only when Zeroname is enabled. When Zeroname is disabled, the site will try to load (it will open wrapper and page title will be set), but the actual page will load forever. See debug log here - the error is (almost) the same as in #2128. When Zeroname is enabled, it will work normally.

@filips123
Copy link
Contributor Author

@HelloZeroNet Do you know what could be a reason for that error?

@HelloZeroNet
Copy link
Owner

HelloZeroNet commented Oct 3, 2019

I think this should be a separate plugin, because depending on the traditional dns system somehow defeat the decentralized and uncensorable goals of the project.

I will check why is it not working without zeroname

@filips123
Copy link
Contributor Author

filips123 commented Oct 4, 2019

@HelloZeroNet I don't think so. I would agree with you if my plugins would be the only way for domains in ZeroNet. But since users can also use Zeroname (or any other domain plugin), it would actually make it more decentralized as users won't be forced to use one particular domain system.

@ghost
Copy link

ghost commented Oct 4, 2019

Exactly! Multiple centralized solutions provides more decentralization than one centralized solution (and I would even say multiple solutions is better than one period). Of course, the best would be multiple decentralized solutions, but we have more work to get to that point.

@filips123
Copy link
Contributor Author

Of course, the best would be multiple decentralized solutions, but we have more work to get to that point.

And it is actually not so easy to implement a really decentralized solution. Almost every third-party domain system for integration with ZeroNet (or any other project) would require either:

  • Complete protocol implementation with locally stored data (which in case of blockchain-based systems means to have the whole blockchain stored locally).
  • Data stored in a custom location (like Zeroname site).
  • Data from a third-party provider (like Infura for Ethereum).

@filips123
Copy link
Contributor Author

@HelloZeroNet @shortcutme Did you find why this plugin doesn't work without Zeroname?

@HelloZeroNet
Copy link
Owner

Yes, I pushed an update (Rev4238) that should make it work without the Zeroname plugin.

It happened because of the missing actionSiteMedia patching without the Zeroname plugin.
I have moved the domain resolving to the parsePath function and also added a short cache (10sec) for isDomain / resolveDomain function, because the parsePath function could be called multiple times for each http request.
43a5742
924a613

@filips123
Copy link
Contributor Author

@HelloZeroNet Should I change anything in this plugin or it will simply work?

From your commits, I saw that you removed actionSiteMedia. Does this mean that the whole UiRequestPlugin is not needed anymore?

@ghost
Copy link

ghost commented Oct 15, 2019

Hi @filips123 @HelloZeroNet
Look #2208 I don't see how _dnslink is better than a reverse proxy to /raw/. Problematic is the forced headers with ZeroNet. At least when proxy and _dnslink detected to "raw" ZeroNet should remove the headers frame etc... Unnecessary. Anyway content.json is exposed to the world which may be a problem if ZeroNet signs the same file two times! If that's the case the private key can be recovered. Right now I stick to reverse proxy. As mentioned on the link above..

@ghost
Copy link

ghost commented Oct 15, 2019

@filips123 With reverse proxy I can use any TLD I want to host on ZeroNet. I don't really think this _dnslink will allow me to use for example:

zeronet.example.org. IN CNAME zeronet.gateway.example.org
_dnslink.zeronet.example.org. IN TXT dnslink=/raw/bitcoinaddress

That would be the goal no? The headers and sandbox in any case where /raw/ requested should be removed!

@ghost ghost mentioned this pull request Oct 15, 2019
@d14na
Copy link
Contributor

d14na commented Oct 15, 2019

@filips123 i've been thinking about how to bridge ENS to Zeronet for a while now. With the new ENS system and 3,4,5-letter domains being auctioned now, I think the system will gain traction within the crypto community. What's the status of #2164? Is this working?

If so, then imo the next step is to bridge back to the clearnet for users NOT running the client. Cloudflare offers this for IPFS websites. Currently in the process of moving almost all of our front-end code over to IPFS. Honestly, it has its issues (mainly caching seems sketchy, but perhaps more seeding is needed), but overall its been working flawlessly for months.

You can read about it here, if you're unfamiliar:
https://www.cloudflare.com/distributed-web-gateway/

The idea is, our proxy at 0net.io would serve (as it does now) the files to Clearnet users, however, they would be able to use https://example.eth instead of https://0net.io/example.bit. Obviously, this would NOT be limited to just .eth, I'm thinking .xyz too (they have a very progressive admin team).

While still making improvements to the proxy's underlying infrastructure, its realatively stable at this point https://status.0net.io. Let me know if you'd be interested in adding that support to your plugin. And, well, how you think we'd go about implementing that into the current proxy's code.

Regardless, good luck with this effort. Don't hesitate to reach for help testing, debugging, whatever. cheers!

@ghost
Copy link

ghost commented Oct 15, 2019

@d14na This has nothing to do with dnslink on Zeronet far as I read your comment. Lastly forget 0net.io because that is a big trash. If you want to proxy clearnet to ZeroNet you will need your own servers for front and back-end!!!!!

@purplesyringa
Copy link
Contributor

A private key can only be recovered if you sign something twice using the same random number, which is next to impossible

@filips123
Copy link
Contributor Author

@HelloZeroNet I bundled all plugin's dependencies (DNSPython and DNSLink) into ZIP file along with plugin's source code. That ZIP file is then added to Python's path (Python can load modules from ZIP) in case if dependencies are not already installed.

I also included support for DNS-reverse-proxy (implemented by @imachug) which enables you to create a site which is hosted on ZeroNet and access it like normal website. To do this, you need to create DNSLink TXT record that points to site address and CNAME/A/AAAA that points to ZeroNet instance. In case of local ZeroNet instance, the site can be accessed at example.com:43110. In case of public ZeroNet proxy, site can be accessed directly at example.com (because most proxies host ZeroNet on port 80). Note that some sites (like ZeroHello, probably also other official sites with dynamic links to other sites) are not yet ready for this as they make incorrect links to other sites. This needs to be fixed in their end.

@filips123
Copy link
Contributor Author

@HelloZeroNet @shortcutme Can you check and merge this?

Dependencies are now bundles with source code (and all licenses are compatible) so there shouldn't be any problem with using this plugin.

This would allow accessing ZeroNet sites from classic DNS domains which is useful in case if site author already has domain (or want to use free OpenNIC domain) and don't want to register another Namecoin domain.

And it would also allow accessing ZeroNet as reverse-proxy. So if the domain has CNAME/A/AAAA record that points to ZeroNet instance and TXT record which contains the address of the site, the site will be directly accessible from that domain.

In case of local instances, this will have to include port 43110. Solution for this would be to use 127.43.11.0:80 as default host of ZeroNet UI. This will also enable viewing ZeroNet sites with local instances from on port 80 while still keeping ZeroNet separated from other services which may run on port 80 (as it would use 127.43.11.0 which is another loopback IP). IPs in 127.*.*.* should resolve to loopback in most cases, but I think that in macOS, the only loopback address is 127.0.0.1. This means that for it, we would need to manually assign new IP to loopback interface, which probably requires root permissions.

But this is actually not so important for this PR so it would be great if you can merge it.

And even if classic DNS is centralized, adding it as additional, but enabled by default, domain system won't defeat decentralized goals of ZeroNet, as users will actually have more choice between domain systems. And with ENS support, it would make it even more decentralized.

@filips123 filips123 mentioned this pull request Dec 13, 2019
23 tasks
@HelloZeroNet
Copy link
Owner

Ok, I'm not against it, I will do some tests later on it.

@filips123
Copy link
Contributor Author

My PR for DNS-over-HTTPS support has been merged, so we don't need to depend on my fork anymore. However, we still need to use Git-based version, as changes still aren't released in new version.

Before this commit, all requests with non-IP `HOST` header were considered as proxy requests. This means that using ZeroNet as a public proxy with own domain would also be considered as proxy request, so all requests would fail.

This now checks if `HOST` is in `ui_host` config and if it is, it is considered as normal request and not as a proxy. This should now allow using ZeroNet on a public domain.
This allows setting custom WebSocket URL for proxy requests. It is useful when ZeroNet is deployed as public proxy in virtual machine or container, where `SERVER_NAME` and `SERVER_PORT` don't contain public server URL.
@filips123
Copy link
Contributor Author

filips123 commented Feb 24, 2020

@imachug I tested DNS reverse proxy with my public ZeroNet proxy and problem is that sometimes cookies are not sent to WebSocket connection which means that connection fails with Multiuser plugin. Do you think it would be possible to send that cookie to WebSocket connection even if it is in a different origin than the website?

Also, zero button links to zero/1HeLLo4uzjaLetFx6NH3PMwFP3qbRbTf3D but this can probably be fixed easily with some condition.. There might also be some other problems with other features that try to connect to wrong URL.

Also, can you check if there can be any possible security problems with accessing WebSocket from a different origin and such?

@purplesyringa
Copy link
Contributor

Do you think it would be possible to send that cookie to WebSocket connection even if it is in a different origin than the website?

AFAIK there's no way to add a cookie to another site. The only solution I've managed to come up with is sending the cookie via the websocket, say, in the first message. @HelloZeroNet Do you think that might work?

@purplesyringa
Copy link
Contributor

Also, zero button links to zero/1HeLLo4uzjaLetFx6NH3PMwFP3qbRbTf3D but this can probably be fixed easily with some condition..

Where do you want it to redirect to? Probably the root of the site?

@purplesyringa
Copy link
Contributor

Also, can you check if there can be any possible security problems with accessing WebSocket from a different origin and such?

I think that's safe. We're using wrapper_key as a security measure so there's no problem as long as the key verification is in action.

@filips123
Copy link
Contributor Author

AFAIK there's no way to add a cookie to another site.

In non-web-based WebSocket clients (in Node.js and Python) you can set cookies. However, I'm not sure if browsers also allow this.

So yes, if sending cookies is not possible, it should be sent either in URL or in first message.

Where do you want it to redirect to? Probably the root of the site?

Yes.

@purplesyringa
Copy link
Contributor

In non-web-based WebSocket clients (in Node.js and Python) you can set cookies. However, I'm not sure if browsers also allow this.

You can set cookies in browsers too but only if they are same-origin.

@filips123
Copy link
Contributor Author

There is one other thing. Is there any reason why WebSocket requests are made to different origin that is specified in server_url instead of accessing from same origin? I think it may be with some issues with old Chrome proxy extension with WebSockets but I'm not sure?

@purplesyringa
Copy link
Contributor

I think there was a reason but I can't remember now. @HelloZeroNet?

@HelloZeroNet
Copy link
Owner

HelloZeroNet commented Feb 26, 2020

Yeah PAC-based proxy configs/extensions does not work with websocket. So when you access http://zeronet.bit you still have to connect http://127.0.0.1:43110/Websocket

@purplesyringa
Copy link
Contributor

What is the state of this PR?

@filips123
Copy link
Contributor Author

Just accessing DNS domains works.

However, DNS reverse proxy with Multiuser plugin sometimes doesn't because of cookies and CORS.

@filips123
Copy link
Contributor Author

I have idea how this cross-origin cookie/multiuser handling could be implemented:

  1. Wrapper/client normally sets cookie and tries to use it on WebSocket connection, like it does now.

  2. When WebSocket server (Multiuser plugin) receives request, it checks for cookie. If it exists, it uses it to get account/address, also like it does now.

  3. However, if cookie isn't set (probably because of CORS), instead of failing with error, it immediately sends some message/request/action/command to wrapper/client for user address.

  4. Wrapper/client listens for that message, and if it receives it, it sends user address as response. Then it can start sending commands normally.

  5. Once server receives response, it uses it as account. If it receives any other commands before response (and response was requested), it fails with error that user isn't authenticated (similarly to what currently happens).

Advantage of this is that if cookies are sent in first place, there isn't any need for any additional message, which means that it shouldn't affect speed. Additionally, it stays compatible with all current clients.

@purplesyringa
Copy link
Contributor

Wrapper/client listens for that message, and if it receives it, it sends user address as response. Then it can start sending commands normally.

A small fix: user masterseed, not address.

@filips123
Copy link
Contributor Author

Master seed or master address? Cookie currently contains master address, so that response should probably also use it. And given that it will probably be used in somewhere in browser's storage (probably cookie just like now, but client just reads it from cookie and sends it to server if needed), it probably won't be good to store sensitive information such as master seed.

@purplesyringa
Copy link
Contributor

Daaaaaamn, and that's what we're calling "secure"... /cc @HelloZeroNet

Both SecureDNS and Dnswarden were shut down recently so I replaced them with LibreDNS. I hope current providers won't also shut down and that there will be some alternatives.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add DNS Support
5 participants