--allow-unknown-fields has no effect; unknown fields are rejected

[howardjohn]

Bug Template

Title: --allow-unknown-fields has no effect; unknown fields are rejected

Description:
When setting --allow-unknown-fields, unknown fields still seem to be rejected

When NOT setting --allow-unknown-fields, unknown fields seem to be accepted

Repro steps:
Download bootstrap.yaml and lds.yaml:
repro.zip

Run docker run -v ${PWD}:/tmp/envoy envoyproxy/envoy-dev:882a30677619856446f7e1b9d28c6ab319b21d1b -c /tmp/envoy/bootstrap.yaml

Output:

[2019-08-15 17:06:28.348][1][info][config] [source/server/configuration_impl.cc:62] loading 0 static secret(s)
[2019-08-15 17:06:28.348][1][info][config] [source/server/configuration_impl.cc:68] loading 0 cluster(s)
[2019-08-15 17:06:28.349][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:147] cm init: all clusters initialized
[2019-08-15 17:06:28.349][1][info][config] [source/server/configuration_impl.cc:72] loading 0 listener(s)
[2019-08-15 17:06:28.349][1][info][config] [source/server/configuration_impl.cc:97] loading tracing configuration
[2019-08-15 17:06:28.349][1][info][config] [source/server/configuration_impl.cc:117] loading stats sink configuration
[2019-08-15 17:06:28.349][1][info][main] [source/server/server.cc:506] all clusters initialized. initializing init manager
[2019-08-15 17:06:28.350][1][info][upstream] [source/server/lds_api.cc:60] lds: add/update listener 'virtualOutbound'
[2019-08-15 17:06:28.350][1][info][config] [source/server/listener_manager_impl.cc:775] all dependencies initialized. starting workers
[2019-08-15 17:06:28.351][1][info][main] [source/server/server.cc:523] starting main dispatch loop

Run docker run -v ${PWD}:/tmp/envoy envoyproxy/envoy-dev:882a30677619856446f7e1b9d28c6ab319b21d1b -c /tmp/envoy/bootstrap.yaml --allow-unknown-fields

Output:

[2019-08-15 17:06:55.556][1][info][config] [source/server/configuration_impl.cc:62] loading 0 static secret(s)
[2019-08-15 17:06:55.556][1][info][config] [source/server/configuration_impl.cc:68] loading 0 cluster(s)
[2019-08-15 17:06:55.556][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:147] cm init: all clusters initialized
[2019-08-15 17:06:55.556][1][info][config] [source/server/configuration_impl.cc:72] loading 0 listener(s)
[2019-08-15 17:06:55.556][1][info][config] [source/server/configuration_impl.cc:97] loading tracing configuration
[2019-08-15 17:06:55.556][1][info][config] [source/server/configuration_impl.cc:117] loading stats sink configuration
[2019-08-15 17:06:55.556][1][info][main] [source/server/server.cc:506] all clusters initialized. initializing init manager
[2019-08-15 17:06:55.557][1][warning][config] [source/common/config/filesystem_subscription_impl.cc:54] Filesystem config update failure: Protobuf message (type envoy.api.v2.DiscoveryResponse reason INVALID_ARGUMENT:fake_field: Cannot find field.) has unknown fields
[2019-08-15 17:06:55.557][1][info][config] [source/server/listener_manager_impl.cc:775] all dependencies initialized. starting workers
[2019-08-15 17:06:55.558][1][info][main] [source/server/server.cc:523] starting main dispatch loop

[mattklein123]

I think this is fixed by #7857

[mattklein123]

cc @htuch

[howardjohn]

Great, thanks!

Just a note -- it seems that while with the latest the flag is inverted, with an older version envoyproxy/envoy-dev:829b905ca0fdc85233c3969247e53a62a52ac627, it is rejected with or without the flag

[htuch]

@howardjohn if it's working now, but inverted, I think this should be solved by #7857. Hope to merge this in the next day or so.

[howardjohn]

@htuch it seems that with #7857 it is no longer inverted, but rather all flags allow-unknown-fields, allow-unknown-static-fields, and reject-unknown-dynamic-fields, and no flag, will now result in the config being rejected.

This should be reproducible with docker run -v ${PWD}:/tmp/envoy envoyproxy/envoy-dev:e1cd4cca46179562cf80de7f81774f4dc6354085 -c /tmp/envoy/bootstrap.yaml, where $PWD is the repro.zip from above.

Any ideas what is happening here?

[mandarjog]

In 1.3 we are using allow-unknown-fields in istio 1.3 is that not working ?

[howardjohn]

@mandarjog it appears to have been broken for a while

[htuch]

@howardjohn I'll look into this later today.

[htuch]

@howardjohn this works for me, see:

<master> htuch@htuch:~/src/envoy ⊧ bazel-bin/source/exe/envoy-static -c ./bootstrap.yaml
[2019-08-27 17:29:53.733][118238][info][main] [source/server/server.cc:242] initializing epoch 0 (hot restart version=11.104)
[2019-08-27 17:29:53.734][118238][info][main] [source/server/server.cc:244] statically linked extensions:
[2019-08-27 17:29:53.734][118238][info][main] [source/server/server.cc:246]   access_loggers: envoy.file_access_log,envoy.http_grpc_access_log,envoy.tcp_grpc_access_log                                                                                                                                                       
[2019-08-27 17:29:53.734][118238][info][main] [source/server/server.cc:249]   filters.http: envoy.buffer,envoy.cors,envoy.csrf,envoy.ext_authz,envoy.fault,envoy.filters.http.dynamic_forward_proxy,envoy.filters.http.grpc_http1_reverse_bridge,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.original_src,envoy.filters.http.rbac,envoy.filters.http.tap,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash                                                              
[2019-08-27 17:29:53.734][118238][info][main] [source/server/server.cc:252]   filters.listener: envoy.listener.http_inspector,envoy.listener.original_dst,envoy.listener.original_src,envoy.listener.proxy_protocol,envoy.listener.tls_inspector                                                                               
[2019-08-27 17:29:53.734][118238][info][main] [source/server/server.cc:255]   filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.dubbo_proxy,envoy.filters.network.mysql_proxy,envoy.filters.network.rbac,envoy.filters.network.sni_cluster,envoy.filters.network.thrift_proxy,envoy.filters.network.zookeeper_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
[2019-08-27 17:29:53.734][118238][info][main] [source/server/server.cc:257]   stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd                                                                                                                                                         
[2019-08-27 17:29:53.734][118238][info][main] [source/server/server.cc:259]   tracers: envoy.dynamic.ot,envoy.lightstep,envoy.tracers.datadog,envoy.tracers.opencensus,envoy.zipkin                                                                                                                                            
[2019-08-27 17:29:53.734][118238][info][main] [source/server/server.cc:262]   transport_sockets.downstream: envoy.transport_sockets.alts,envoy.transport_sockets.tap,raw_buffer,tls                                                                                                                                            
[2019-08-27 17:29:53.734][118238][info][main] [source/server/server.cc:265]   transport_sockets.upstream: envoy.transport_sockets.alts,envoy.transport_sockets.tap,raw_buffer,tls                                                                                                                                              
[2019-08-27 17:29:53.734][118238][info][main] [source/server/server.cc:271] buffer implementation: new
[2019-08-27 17:29:53.745][118238][info][main] [source/server/server.cc:331] admin address: 0.0.0.0:15000
[2019-08-27 17:29:53.828][118238][info][main] [source/server/server.cc:445] runtime: layers:
  - name: base
    static_layer:
      {}
  - name: admin
    admin_layer:
      {}
[2019-08-27 17:29:53.829][118238][info][config] [source/server/configuration_impl.cc:62] loading 0 static secret(s)
[2019-08-27 17:29:53.829][118238][info][config] [source/server/configuration_impl.cc:68] loading 0 cluster(s)
[2019-08-27 17:29:53.830][118238][info][upstream] [source/common/upstream/cluster_manager_impl.cc:161] cm init: all clusters initialized
[2019-08-27 17:29:53.830][118238][info][config] [source/server/configuration_impl.cc:72] loading 0 listener(s)
[2019-08-27 17:29:53.830][118238][info][config] [source/server/configuration_impl.cc:97] loading tracing configuration
[2019-08-27 17:29:53.830][118238][info][config] [source/server/configuration_impl.cc:117] loading stats sink configuration
[2019-08-27 17:29:53.831][118238][info][main] [source/server/server.cc:513] all clusters initialized. initializing init manager
[2019-08-27 17:29:53.838][118238][warning][config] [source/common/protobuf/message_validator_impl.cc:28] Unknown field: type envoy.api.v2.DiscoveryResponse reason INVALID_ARGUMENT:fake_field: Cannot find field.                                                                                                             
[2019-08-27 17:29:53.840][118238][info][upstream] [source/server/lds_api.cc:59] lds: add/update listener 'virtualOutbound'
[2019-08-27 17:29:53.840][118238][info][config] [source/server/listener_manager_impl.cc:789] all dependencies initialized. starting workers
[2019-08-27 17:29:53.859][118238][info][main] [source/server/server.cc:530] starting main dispatch loop
[2019-08-27T21:30:16.264Z] "GET / HTTP/1.1" 200 - 0 5072 10 - "100.118.30.65" "curl/7.64.0" "-" "localhost:15000" "-"
[2019-08-27T21:30:21.161Z] "GET /listeners HTTP/1.1" 200 - 0 31 0 - "100.118.30.65" "curl/7.64.0" "-" "localhost:15000" "-"

and

<master> htuch@htuch:~/src/envoy ⊧ curl localhost:15000/listeners
virtualOutbound::0.0.0.0:15001

It's detecting the unknown field and warning (as expected for the dynamic config), but it accepts it still.

[howardjohn]

@htuch thanks! I didn't realize it was just warning. Looks good now, I think this can be closed. Thanks for the fix