Consider adding routing capabilities to tcp_proxy

[enricoschiattarella]

Currently the tcp_proxy network filter can only open connections to hosts belonging to a single cluster specified by name in the configuration.

We propose the addition of an optional list of L3/L4 routing rules pointing to specific clusters by name.
The rules would be examined in-order and the first rule to match would be applied.

A rule would consist of a cluster name and 1 or more match criteria from this set:

• src ip address (with optional subnet mask)
• original destination ip address (with optional subnet mask). This is useful if the connection has been redirected with iptables
• src TCP port
• original destination port (might be different from listener port due to redirect)

If all specified match criteria are satisfied (src/dst ports are equal to the provided number, src/dest IP addresses belong to the range), the connection is opened towards the cluster named in the rule.

Later on we can add probabilistic fall-through similar to http_connection_manager routes.

The "cluster" parameter currently defined at the filter level becomes the default cluster (used if no rules match). If it is not defined (assuming it will become optional) the connection is terminated.

If the feature looks useful and the proposal reasonable, I will start filling in the details of the configuration model and the implementation.

[mattklein123]

@enricoschiattarella this sounds reasonable and useful. As you suggest, I agree a good next step would be to do a short proposal of the route config schema (here or in doc) and we can all review.

[mattklein123]

P.S., longer term it would also be pretty cool/useful to put rate limit stuff directly into the TCP route configuration much like @ccaraman is now working on for HTTP. As long as we have a generic TCP route table config it should be easy to extend in the future.

[enricoschiattarella]

tcp_proxy configuration example

{
  "port": 8082,
  "filters": [
    {
      "type": "read",
      "name": "tcp_proxy",
      "config": {
        // Config-level "cluster" parameter becomes mutually-exclusive 
        // with route_config block. Exactly one of the two must be specified.
        // "cluster": "service2",
        "stat_prefix": "tcp"
        "route_config": {
          "routes": [
            // Routes are examined in order. The first match is used. 
            // If no rule matches the connection is dropped.
            {  
             // Any combination of source_ip, destination_ip, 
             //source_port, destination_port can be specified  

              // matches addresses in the range 192.168.0.0 - 192.168.0.255
              "source_ip": "192.168.0.0/24",

              // exact match, equivalent to "10.10.10.10/32"
              "destination_ip": "10.10.10.10",

              // destination port must be in range 1-1024 OR 2048-4096 OR be equal to 12345
              "destination_port": "1-1024,2048-4096,12345", 

              // if the match criteria for source_ip AND destination_ip AND destination_port
              // are satisfied, open the connection to cluster "service1".
              // source_port is ignored because it is not specified
              "cluster": "service1" // required
            },
            { 
              // this route has no criteria to satisfy, so it always matches
              // typically it is the last route, providing a default (as in "last resort") cluster
              "cluster": "service2"
            }
        ]      
      }     
    }                                                                                        
  ]   
}

[mattklein123]

@enricoschiattarella personally I would vote to just get rid of the config-level cluster param and make everyone specify the route table config. It's more verbose and a breaking change but IMO it's better to have one config type if possible. I don't have a strong opinion on this if others feel differently though.

re: destination IP/port, we will obviously have to expose this out of the connection interface since we don't do this currently. This should't be difficult.

Otherwise this all seems reasonable to me.

[enricoschiattarella]

@mattklein123 Thanks, Matt.
I will start working on it and continue this thread if other non-straightforward design choices come up.

[mattklein123]

implemented

[abhinav1998agarwal]

Hey! I am looking for the api reference corresponding to this TCP routing on https://www.envoyproxy.io/docs/envoy/v1.12.1/configuration/listeners/network_filters/tcp_proxy_filter#config-network-filters-tcp-proxy , but unable to find. @PiotrSikora , @kyessenov can you guide me to the api reference for this TCP routing.

[YvesZHI]

So have these functions been merged into Envoy? I'm working with Envoy V1.22.0 but there doesn't exist a route_config in the TcpProxy...

[abhinav1998agarwal]

@YvesZHI Please check https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/listener/v3/listener_components.proto#config-listener-v3-filterchainmatch to perform layer 4 routing.