Allow setting source interface in open_tcp_stream

[njsmith]

Right now open_tcp_stream has no option to control which outgoing interface the connections are made from. We should make this possible.

Subtleties:

• It needs to be possible to set this separately for AF_INET and AF_INET6
• We can't support setting the actual outgoing port, because we might need to make multiple outgoing connections in parallel

On Linux 4.2+, there's sock.setsockopt(IPPROTO_IP, IP_BIND_ADDRESS_NO_PORT, 1); sock.bind((address, 0)) which means "use this host, but delay picking the port until I call connect". We should use this if available. We should not use SO_REUSEADDR, because then it's possible to bind to a port that will fail when we actually call connect (because there might be a TIME_WAIT with the same 4-tuple).

So maybe something like: source_host={AF_INET: "...", AF_INET6: "..."}?

Should we also allow source_host="1.2.3.4" and auto-expand it into {AF_INET: "1.2.3.4"}?

If one is not specified, what happens? Maybe we should treat INADDR_ANY as meaning "you can use this protocol, and I don't care what host you originate from", and missing as meaning "don't use this protocol"? That could be nice since it would also give a way to say "I just want you to use IPv6", albeit not an obvious one.

[njsmith]

Question: how does IP_BIND_ADDRESS_NO_PORT work with IPv6? it's only documented as an IPPROTO_IP option, but the patch cover letter says it works on IPv6; maybe you can just use IPPROTO_IP on IPv6 sockets, at least in this case? Should check this.

#39 has some discussion of SO_REUSEPORT details, for reference.

[njsmith]

Here's a possible API that might be a bit simpler to use: allow specifying a list of source addresses, and each socket walks down the list and uses the first one that it can.

So ["1.2.3.4", "1:2:3::4"] would use 1.2.3.4 for outgoing IPv4 connections, and 1:2:3::4 for outgoing IPv6 connections, with random ports. (Using IP_BIND_ADDRESS_NO_PORT if available, otherwise falling back on port 0.) Specifying ["1.2.3.4"] would force the use of IPv4 for this connection.

Do people need the ability to control the outgoing port? We could also allow [("1.2.3.4", 5678)] to specify the source port, and this would even allow folks to write silly things like [("1.2.3.4", 5678), ("1.2.3.4", 9012)] if they wanted to specify a set of allowable source ports (so the first attempt would use 5678, then the second one would try to bind that and fail if the first one was still using it, and then move on to try binding 9012 instead). But possibly this is something we should defer until someone actually needs it.

[miracle2k]

To determine if an IP address can be used with an entry returned from getaddrinfo, would we try to detect the type (v4 or v6) by looking only the string itself (say if it contains a dot or a colon)?

If so, more than two entries in that list doesn't really make sense, correct?

[njsmith]

@miracle2k I guess I was imagining that we might write a loop like:

for source in preferred_sources:
    try:
        await sock.bind(source)
    except OSError:
        # Try the next source
        continue
    else:
        # The bind worked, we can use this socket
        break
else:
    # no sources worked, give up on this socket

It's extremely not-smart, which is in some ways a benefit, because it arguably makes it smarter :-). For example, it could handle the case where someone lists multiple fully-specified (host, port) pairs – if one port is taken, the bind will fail, and we'll try the next.

I haven't through it through enough to come to any firm conclusion on which way is best, but I do like the idea of not having to deal with parsing IP addresses. (Though the ipaddress module helps.)

[Tronic]

Somewhat related consideration: setting UDP packet source address. I'm running a DNS server on localhost, and following the convention of systemd-resolved, lookups are done on 127.0.0.53 rather than 127.0.0.1. Apparently there is no easy way to answer these packets with the correct source address with Python sockets (even at low level), and packets are always sent with source 127.0.0.1 (interface's main address) and then rejected by recipient because of wrong source address.

[njsmith]

@Tronic Huh, works for me? But let's keep this issue focused on how to expose this feature in open_tcp_stream's API, and discuss that somewhere else, maybe chat?

[njsmith]

The httpcore project is asking us for this feature – see encode/httpcore#88, encode/httpcore#100, #1642

I poked at it a bit more recently. This is a frustrating API to design, because the use cases are not at all clear. For example, the httpcore PR allows setting a source port – is that important? I don't know. It definitely creates awkwardness for happy eyeballs though, since if you naively try to make multiple sockets that are all bound to the same source port then it's not going to work. (Maybe some kind of SO_REUSEADDR thing could help? I'm not at all confident I understand the subtleties.)

Things I do know:

• Most networking libraries do support this in some form. I hoped that by digging into their issue trackers etc. I'd find discussion of use cases that prompted them to add it, but so far no luck. Probably its universality means that it's actually useful, and we're not just all implementing it because other libraries implemented it? But it's real hard to find

  • Most libraries have a pretty simple connect method that basically just creates a socket → calls bind → calls connect, and they expose knobs to control the bind arguments, done. As noted above, that doesn't translate super-easily to happy-eyeballs
  • The exception is Twisted: their HostnameEndpoint (= happy eyeballs code) only lets you specify a source host, not a source port

• Real use case: forcing the use of ipv4 vs ipv6 to work around firewall issues (example: @hynek has this problem here Advanced connection options. encode/httpcore#88 (comment)). Less of an issue if you have happy eyeballs of course, and really this is a misconfigured network, but still, if your network is misconfigured in a way that normally adds 250 ms of latency on every outgoing connection, then it's reasonable to want a knob to work around that.

• Real use case: hosts that are connected to multiple different networks, with not-quite-solid routing rules. Example: socket.connect does not use specified localAddress for DNS lookup nodejs/node#14617. In that issue they're asking about DNS stuff, which is out-of-scope for this issue, but their basic configuration where they're forcing the use of specific uplinks for specific connections seems like a good example of the kind of situation where this feature is useful.

• Real use case: Did you know that IPv4 has multiple loopback addresses? Usually we use 127.0.0.1, but in fact the entire 127.*.*.* subnet is reserved for loopback. And on Linux you can even bind to all of these addresses by default! (And on other systems it might be possible, but take more configuration.) Sometimes people use this as a kind of poor-man's virtual LAN: spin up some servers bound to 127.0.0.2, 127.0.0.3, etc., and they can all connect to each other, all use distinct source addresses, use the same port numbers without conflicting, etc. Of course these days "real" virtualization and container tools have become so popular and easy-to-use that probably most folks are using those instead of loopback-based hacks, but still, wanting to bind to 127.0.0.2 is not a ridiculous thing to do.

So overall, my impression is that this is mostly useful to enable slightly-janky-but-understandable hacks and workarounds for weird routing configurations. Fair enough.

Implementing it is pretty straightforward. I think the questions to answer are:

• do you want to allow specifying ports, or not
  • and if you do, then do you want to mess with SO_REUSEADDR to try to keep happy eyeballs working? how do the various platforms we support even behave if you try to do this?
• do you want to allow specifying multiple local addresses with some kind of fallback sequence (and in particular, IPv4 and IPv6 sources?), or not. (AFAICT no other library does this)
• how do you handle when the local host is given using a DNS name, which may resolve to multiple IPs?
• for the "force ipv4"/"force ipv6" use case, should you do that using a separate flag, or by specifying a wildcard source address? (0.0.0.0 as a source would mean "any ipv4, but not ipv6", and :: as a source would mean "any ipv6, but not ipv4". At least Windows and Linux both handle this properly.)
• If you're restricted to a specific address family, do you propagate that into the destination host lookup, as an optimization? (i.e., skip fetching AAAA records if we're going to use the local host is ipv4). OTOH, a downside of this optimization is that if the connection fails, you can give a better error message if you have the information "you could have tried connecting to this IPv6 address if you hadn't specified an IPv4 source address".

It's not actually that hard to implement even the most complicated all-the-bells-and-whistles version... but testing it is super annoying, because there are so many different tricky combinations of cases. And to make testing even more fun, on most test machines you can't even assume that there are multiple addresses available to attempt binding to.

So, given all this, my inclination is to start by implementing the simplest possible version: let the user pass in a single source host, no port, no fallback sequence, no clever optimizations. That's sufficient to handle all the actual known use cases, can be extended later if necessary, and seems like an appropriate amount of effort to put into a feature that's only used for rare workarounds.

One wrinkle is that in their draft PR, the httpx folks do want to support setting a source port: encode/httpcore#100. Maybe we can convince them to skip that feature? It doesn't look great for Trio if downstream projects have to put stuff in their docs like "NOTE: the XX feature is supported on all backends except Trio.", regardless of whether the XX feature is actually useful or not.

[njsmith]

Oh, here's some more discussion of this on the httpx tracker, with more use cases: encode/httpx#755

AFAICT they're all similar to what I said above, and in particular I don't see any requests to control the source port.

[smurfix]

Setting the source port makes sense on UDP, but with TCP I don't see the usecase. Worse, today you set your fancy source port (because you can, y'know), next month your co-worker tries to run your client on two destinations simultaneously, whcih happen to resolve to the same IP address, which suddenly won't work because the address+port combo is no longer unique. Owch.

Linux can happily assign a specific IP address to multiple interfaces. Worse, binding to a local address doesn't actually force packets to use the interface / any of the interfaces which the address is bound to. So setting that local address is not sufficient if you want to control which interface to use. There's a SO_BINDTODEVICE socket option to do that. Supporting it thus makes sense and is pretty easy to implement (if not exactly easy to test).

[njsmith]

Unfortunately SO_BINDTODEVICE is Linux-only and requires root, so it's not really a general substitute for binding to a local IP. I think if we did want to support it in open_tcp_stream, probably the best way would be to add a generic "prepare the socket" interface that can issue whatever weirdo setsockopt calls it wants? But that would be a separate issue, and we can wait until someone complains :-).

[bwelling]

Probably not too important, but the restriction of SO_BINDTODEVICE to root was removed in the 5.7 version of the Linux kernel. Who knows when distributions will pick this change up, though.

[pquentin]

@bwelling For what it's worth, we're already testing 5.6 with Fedora 32 in CI, and will be able to test a newer version when Fedora 33 comes out in a few months.