Provide local and remote socket addresses from a Connecting

[Demi-Marie]

libp2p needs to know the local and remote socket addresses for every incoming connection as soon as the connection is accepted. Is this a feature that you would be willing to include? If so, I will submit a PR.

[Ralith]

You can get the local address from Endpoint::local_addr, or just hold onto whatever you passed to EndpointBuilder::bind. We don't have plans to support multiple local addresses, so this should be sufficient, but let us know if it's not.

Exposing the remote address from Connecting makes perfect sense to me. I imagine many applications will want this even just for diagnostics. A PR would be very welcome!

[Ralith]

Alternatively, since this doesn't make a lot of sense for outgoing connections, perhaps <Incoming as Stream>::Item could be made into a struct containing Connecting and associated metadata, but my gut feeling is that's less ergonomic for dubious benefit.

[Demi-Marie]

Libp2p needs to know the destination IP address of the UDP datagram that initiated the connection. This allows it to publish new addresses at which the local host can be reached. This address is generally different from what was passed to EndpointBuilder::bind if a wildcard address was used for the latter. On Linux, it can be obtained by setting the IPV6_RECVPKTINFO option for IPv6 sockets, or the IP_PKTINFO option for IPv4 sockets.

[Ralith]

Ah, good point, I forgot about wildcard IPs. Exposing that makes sense, though it'll require a certain amount of piping, and perhaps attention to win32/macos/BSD compatibility.

[Demi-Marie]

Windows is going to require us to handle sockets ourselves, just as we already do on *nix. I don’t have access to a Windows machine, so I am not particularly comfortable writing this code. I am, however, willing to write an implementation for Linux.

[Ralith]

If we're lucky it'll be exactly the same, but obscure cmsg stuff is pretty hit-or-miss portability wise. Linux would be a good start for us to build off of, at least!

[Demi-Marie]

Here is what I have found:

OS	Linux	FreeBSD	OpenBSD	NetBSD	macOS	illumos (and presumably Solaris, AIX, z/OS, etc)	Windows
IPv4 destination address	IP_PKTINFO	IP_RECVDSTADDR	IP_RECVDSTADDR	all are supported	IP_PKTINFO	IP_RECVPKTINFO	IP_PKTINFO
IPv4 ingress interface	IP_PKTINFO	IP_RECVIF	IP_RECVIF	all are supported	IP_PKTINFO	IP_RECVPKTINFO	IP_PKTINFO
IPv6 destination address	IPV6_RECVPKTINFO	IPV6_RECVPKTINFO	IPV6_RECVPKTINFO	IPV6_RECVPKTINFO	IPV6_RECVPKTINFO	IPV6_RECVPKTINFO	IPV6_RECVPKTINFO
IPv6 ingress interface	IPV6_RECVPKTINFO	IPV6_RECVPKTINFO	IPV6_RECVPKTINFO	IPV6_RECVPKTINFO	IPV6_RECVPKTINFO	IPV6_RECVPKTINFO	IPV6_RECVPKTINFO

I don’t have access to Solaris, AIX, or any other big iron UNIX docs, so I assumed that they work the same as illumos, as they are all System V derivatives. ? means that I could not find the documentation ― it does not mean that it is not possible.

I believe that we should add this functionality to libstd, async-std, and Tokio, rather than trying to implement it ourselves.

The IPv6 API is standardized in RFC 3542, so it works on all OSs.

[Ralith]

Nice work! Do you expect to need interfaces too?

[Demi-Marie]

Probably not right away, although they would be nice to have. I think we should have a crate that abstracts these differences, and then port async-std and tokio to use it.

I can test on Linux and OpenBSD as I have both of those installed. I can probably FreeBSD, NetBSD, and illumos without too much trouble. I do not have access to any of the proprietary OSs, so those will need to be tested by someone else.

[Ralith]

Adding support for semi-portable cmsg stuff to foundational libraries is a very tricky engineering problem, particularly when trying to be zero-cost. We're already doing similar things by hand for ECN support and I think continuing in that vein is reasonable.

That said, if you have ideas for how to address this sort of thing more reusably, don't let me stand in your way! I know tokio, at minimum, is receptive to exposing cmsg-related capabilities if a good API can be found.

[Ralith]

Bear in mind also that we expect Quinn to continue to dig deeper into esoteric and bleeding-edge platform-specific APIs as part of ongoing performance work, e.g. #501, so any attempt to create a generic "advanced datagram socket" abstraction will have a moving target.

[Demi-Marie]

@Ralith I think io_uring on Linux will be the biggest performance win of them all, at least for APIs that don’t require special privilege. That very much should be incorporated into Tokio and async-std, as it also supports filesystem I/O.

Going faster than that generally requires bypassing the kernel, which requires special privileges. On most systems, such privileges are only available to root.

[Ralith]

While io_uring is pretty cool, and I'm strongly supportive of its adoption in general-purpose toolkits, it doesn't solve the problems addressed by segmentation offload.

[Demi-Marie]

As far as control messages, I think libraries can provide a thin wrapper around the raw system calls with a reasonable amount of effort. Users would be expected to provide their own buffers.

[Ralith]

I think we need this to handle the case where a single socket accepts connections on multiple addresses on the same subnet. Currently, the source address of outgoing packets is not guaranteed to match what the peer expects in that situation. See https://blog.powerdns.com/2012/10/08/on-binding-datagram-udp-sockets-to-the-any-addresses/ for related discussion.

[Demi-Marie]

This is actually less of an issue for QUIC than it is for DNS, since QUIC supports roaming of clients natively. We do need to get this right for servers, though.

[Ralith]

Migration can be disabled for a given connection by the server, so even clients may need this.

[zz85]

I was looking up whether quinn supports IP_PKTINFO and found this issue. @Demi-Marie, just wondering if you had the chance to work on this issue? It seems that one way IP_PKTINFO can be set on socket would be to extend the send_ext and recv_ext methods

[Ralith]

That would be the correct approach, yeah.

[Matthias247]

I have implemented the Linux version of this in #943. It works fine. It might also work on other platforms - but I didn't get to verify this so far, so I haven't enabled the flag on other platforms.

I also have another change which will allow to reuse that particular incoming IP address for all outgoing transmissions on the same connection. But I will park that for another CR.

[Ralith]

Looks like this was fixed by #943.