Support for TCP fast open / TLS early data

[njsmith]

TCP has a feature now where you can send some data along with the initial SYN packet, called TCP fast open (LWN article, rfc). (At least, if you've made a connection to the same server somewhat recently.) Linux has actually supported this for quite some time (some support since 3.7, complete support since 3.16). This is a bit tricky because the point of the 3-way handshake is to establish a reliable connection, and this lets you send data before that, so it's intrinsically unreliable. In particular a server might randomly seem to get multiple connections and read this data multiple times, even though the client only made one connection, so you can only use this as an opt-in thing with protocols where you know the initial data being sent is idempotent.

TLS 1.3 has a semantically very similar feature, where you can send data along with the initial handshake, called "early data". It has the same problem with requiring idempotency (and the same requirement that you have made a connection to the server before), so it has to be opt-in. It doesn't actually exist as something one can use yet though. (Needs openssl 1.1.1, plus support in wrappers.)

We should support these in our high-level API on both the client and server side. I think that on the client side, this would mean something like an extra early_data= argument to open_tcp_stream or open_ssl_over_tcp_stream. And on the server side, something like an allow_early_data argument to open_tcp_listeners/open_ssl_over_tcp_listeners (defaulting to False)? I guess for open_tcp_listeners this should be a "queue length" (to avoid DoS issues).

For TLS early open, we'll also want to expose some of the richer API that openssl exposes on the SSLStream API. (Or maybe this is the only thing we want to do on the server side?) For TCP fast open, AFAICT the socket API doesn't give any way to distinguish between regular data and fast-open data, so this is unnecessary.

Also, I believe that in general the initial part of a TLS client handshake is always idempotent, so open_ssl_over_tcp_stream should use TCP fast open by default, and likewise for open_ssl_over_tcp_listeners.

However, there is a subtlety when TLS early data is combined with TCP fast open: there is a rule that servers are supposed to reject re-use of TLS early data tokens, so (a) on the server side maybe we need to help implement that somehow?, (b) on the client side, we can't necessarily make one SSLObject and then pass its data to open_tcp_stream, because if happy-eyeballs kicks in then we'll end up sending that same data multiple times. OTOH maybe this is fine because (a) happy eyeballs rarely kicks in, and (b) when it does whichever connection wins is usually going to be the one whose SYN was received first, and (c) even if we do end up using a connection where the token was invalidated by another happy-eyeballs connection, TLS transparently falls back on a regular handshake. I think.

Testing this is going to be super annoying. I'm not sure how to even do it except by like, shelling out to tcpdump or something.

Cross-platform support will also be a bit tricky. On Windows you have to set TCP_FASTOPEN and then use ConnectEx, which means you MUST use IOCP. On MacOS, you have to use the odd connectx, which Python doesn't currently wrap.

This page has some useful notes: https://dnsprivacy.org/wiki/display/DP/TCP+Fast+Open#TCPFastOpen-OSX

[Fuyukai]

Doesn't TLS early data rely on 0-RTT, which is super easy to replay attack?

[njsmith]

TLS early data is indeed subject to replay attacks, which is why (a) servers are supposed to detect and reject when the same session token is reused, (b) it would be something that clients have to explicitly request via a special argument or something (presumably in cases where they know their particular early data is safe), not something that we can enable automatically. But if people want to use it at all with Trio then we need to eventually provide something.

It's true though that this is a somewhat niche case and it's not clear what people will actually use it for (if anything). And in any case we can't do anything about early data now anyway since openssl 1.1.1 isn't even out yet, and then it will be some time after that before we have any way to access the necessary new APIs from Python.

TCP fast open is something we could theoretically implement now and get some small win in the common case of setting up SSL-over-TCP, though it's not really urgent compared to all the other things we have to do.

[njsmith]

Interesting update, though perhaps not relevant to us: torvalds/linux@19f6d3f

[njsmith]

When implementing TLS early data, things to review and think about:

• Non-blocking Socket and Early Data openssl/openssl#7757
• Interactions with the workaround for Deadlocks caused by openssl's handling of TLS 1.3 session tickets #819 (comment)