New subflows send MP_JOIN to port 0

[matttbe]

More info are coming soon but we can see in different setup that MP_JOIN are sent to port 0:

12:02:25.985960 IP 10.0.1.2.8000 > 10.0.1.1.59574: Flags [.], ack 78, win 509, options [nop,nop,TS val 58365988 ecr 260327143,mptcp add-addr[bad opt]>
12:02:25.986311 IP 10.0.2.1.46531 > 10.0.2.2.0: Flags [S], seq 1204877313, win 64240, options [mss 1460,sackOK,TS val 2027756336 ecr 0,nop,wscale 7,mptcp join backup id 0 token 0xc777f628 nonce
0xacb9a776], length 0
12:02:25.986322 IP 10.0.2.2.0 > 10.0.2.1.46531: Flags [R.], seq 0, ack 1204877314, win 0, length 0

Or even with packetdrill:

root@(none):/opt/packetdrill/gtests/net/mptcp/mp_join# tcpdump -i any -n -c 20 tcp &
[1] 260
root@(none):/opt/packetdrill/gtests/net/mptcp/mp_join# tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes

root@(none):/opt/packetdrill/gtests/net/mptcp/mp_join# ../../packetdrill/packetdrill -vvv mp_join_client.pkt 
socket syscall: 1594826109.716840
setsockopt syscall: 1594826109.719996
fcntl syscall: 1594826109.723111
fcntl syscall: 1594826109.725905
connect syscall: 1594826109.729016
outbound sniffed packet:  0.089384 S 3824942547:3824942547(0) win 65535 <mss 1460,sackOK,TS val 978433635 ecr 0,nop,wscale 8,mp_capable v1 flags: |H| >
inbound injected packet:  0.102111 S. 0:0(0) ack 3824942548 win 65535 <mss 1460,sackOK,TS val 4074410674 ecr 978433635,nop,wscale 8,mp_capable v1 flags: |H| sender_key: 2>
outbound sniffed packet:  0.112000 . 3824942548:3824942548(0) ack 1 win 256 <nop,nop,TS val 978433658 ecr 4074410674,mp_capable v1 flags: |H| sender_key: 15724967926438798442 receiver_key: 2>
15:15:09.728921 IP 192.168.228.105.47794 > 192.0.2.1.8080: Flags [S], seq 3824942547, win 65535, options [mss 1460,sackOK,TS val 978433635 ecr 0,nop,wscale 8,mptcp capable[bad opt]>
15:15:09.751397 IP 192.0.2.1.8080 > 192.168.228.105.47794: Flags [S.], seq 0, ack 3824942548, win 65535, options [mss 1460,sackOK,TS val 4074410674 ecr 978433635,nop,wscale 8,mptcp capable Unknown Version (1)], length 0
15:15:09.751537 IP 192.168.228.105.47794 > 192.0.2.1.8080: Flags [.], ack 1, win 256, options [nop,nop,TS val 978433658 ecr 4074410674,mptcp capable Unknown Version (1)], length 0
getsockopt syscall: 1594826109.963115
fcntl syscall: 1594826109.966067
15:15:10.968919 IP 192.168.228.105.47794 > 192.0.2.1.8080: Flags [P.], seq 1:3, ack 1, win 256, options [nop,nop,TS val 978434875 ecr 4074410674,mptcp capable[bad opt]>
write syscall: 1594826110.979351
outbound sniffed packet:  1.329382 P. 3824942548:3824942550(2) ack 1 win 256 <nop,nop,TS val 978434875 ecr 4074410674,mp_capable v1 flags: |H| sender_key: 15724967926438798442 receiver_key: 2 mpcdatalen=2,nop,nop>
inbound injected packet:  1.356829 . 1:1(0) ack 3824942550 win 256 <nop,nop,TS val 4074418293 ecr 978434875,add_address address_id: 1 ipv4: 192.0.2.2 hmac: 18175360766677029581,dss dack8 9168515192191584501 flags: Aa>
15:15:11.009981 IP 192.0.2.1.8080 > 192.168.228.105.47794: Flags [.], ack 3, win 256, options [nop,nop,TS val 4074418293 ecr 978434875,mptcp add-addr[bad opt]>
15:15:11.019680 IP 192.168.228.105.35147 > 192.0.2.2.0: Flags [S], seq 2261540031, win 65535, options [mss 1460,sackOK,TS val 3362971286 ecr 0,nop,wscale 8,mptcp join backup id 0 token 0xd86e8112 nonce 0x91218654], length 0
15:15:12.059316 IP 192.168.228.105.35147 > 192.0.2.2.0: Flags [S], seq 2261540031, win 65535, options [mss 1460,sackOK,TS val 3362972326 ecr 0,nop,wscale 8,mptcp join backup id 0 token 0xd86e8112 nonce 0x91218654], length 0
15:15:14.107395 IP 192.168.228.105.35147 > 192.0.2.2.0: Flags [S], seq 2261540031, win 65535, options [mss 1460,sackOK,TS val 3362974374 ecr 0,nop,wscale 8,mptcp join backup id 0 token 0xd86e8112 nonce 0x91218654], length 0
15:15:18.139331 IP 192.168.228.105.35147 > 192.0.2.2.0: Flags [S], seq 2261540031, win 65535, options [mss 1460,sackOK,TS val 3362978406 ecr 0,nop,wscale 8,mptcp join backup id 0 token 0xd86e8112 nonce 0x91218654], length 0

(Note: tcpdump version is old, not support MPTCPv1)

[nrybowski]

Here is a minimal setup to reproduce this bug (mptcp-tools is supposed to be in the same folder than this script and the use_mptcp tool compiled) :

#! /bin/bash

setup_iface() {
    ns="$4"
    ns_exec="ip netns exec $ns"

    ip l set "veth$1" netns "$ns"
    $ns_exec ip l set dev veth"$1" up  
    $ns_exec ip a add dev veth"$1" 10.0."$2"."$3"/24
}

cgroups=("client" "server")
use_mptcp="./mptcp-tools/use_mptcp/use_mptcp.sh"

ip l add veth1 type veth peer name veth2
ip l add veth3 type veth peer name veth4

i=1
for cgroup in ${cgroups[@]}
do
    ns_name=ns_$cgroup
    ns_exec="ip netns exec $ns_name"
    ip netns list | grep $ns_name > /dev/null
    if [ $? -eq 1 ]
    then
        ip netns add $ns_name 

        setup_iface "$i" "1" "$i" "$ns_name" 
        setup_iface "$((i+2))" "2" "$i" "$ns_name"
        $ns_exec ip mptcp endpoint flush
        $ns_exec ip mptcp limits set add_addr_accepted 2 subflows 2
    fi

    if [ "${cgroup}" = "server" ]
    then
            addrs=$($ns_exec ip a | grep inet | sed -e 's/inet[6]*//g' -e 's/fe80.*$//g' -e 's/\/24.*$//g')
            echo "${addrs[@]}"
     	    #for addr in ${addrs[@]}
            #do
            #   ${ns_exec} ip mptcp endpoint add ${addr} signal
            #done
            ${ns_exec} ip mptcp endpoint add ${addrs[0]} signal
            ${ns_exec} ${use_mptcp} python3 -m http.server &
            #${ns_exec} tc qdisc add dev veth$i root netem delay 1000ms 
    fi

    ((i++))

done

In shell 1 : ip netns exec ns_server tcpdump -ni any tcp and in shell 2 : ip netns exec ns_client mptcp-tools/use_mptcp/use_mptcp.sh curl 10.0.1.2:8000 -o /dev/null.

From the tcpdump :

[...]
14:08:32.077470 IP 10.0.1.2.8000 > 10.0.1.1.41760: Flags [.], ack 78, win 509, options [nop,nop,TS val 1413864222 ecr 3687490168,mptcp add-addr[bad opt]>
14:08:32.077841 IP 10.0.2.1.43719 > 10.0.2.2.0: Flags [S], seq 2048241466, win 64240, options [mss 1460,sackOK,TS val 2245203070 ecr 0,nop,wscale 7,mptcp join backup id 0 token 0xe131e65a nonce
0x763548b7], length 0
14:08:32.077850 IP 10.0.2.2.0 > 10.0.2.1.43719: Flags [R.], seq 0, ack 2048241467, win 0, length 0
[...]

I'm not sure if add signal has to be called on both the addresses of ns_server but when I tried (the for loop in the above script) I got the same bug but on the first interface :

[...]
14:11:47.055974 IP 10.0.1.2.8000 > 10.0.1.1.41762: Flags [.], ack 78, win 509, options [nop,nop,TS val 1414059201 ecr 3687685147,mptcp add-addr[bad opt]>
14:11:47.056372 IP 10.0.1.1.41485 > 10.0.1.2.0: Flags [S], seq 3336345692, win 64240, options [mss 1460,sackOK,TS val 3687685147 ecr 0,nop,wscale 7,mptcp join backup id 0 token 0x27143ab nonce 0xf3a22c5c], length 0
14:11:47.056381 IP 10.0.1.2.0 > 10.0.1.1.41485: Flags [R.], seq 0, ack 3336345693, win 0, length 0
[...]

Tested on commit eeb8340.

[matttbe]

Arf, I forgot to add "Closes #63" in the commit message of my last patch.

This is fixed in the export branch. It has been sent to netdev for -net branch.