Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using a HTTP based external auth service with GRPC proxy entries crashes envoy on 401s #6902

Closed
thorkildcognite opened this issue May 10, 2019 · 1 comment · Fixed by #6913
Closed

Using a HTTP based external auth service with GRPC proxy entries crashes envoy on 401s #6902

thorkildcognite opened this issue May 10, 2019 · 1 comment · Fixed by #6913
Labels
bug
Milestone
1.11.0

Comments

@thorkildcognite
Copy link

Title: HTTP Auth combined with grpc endpoints lead to auth body being copied into grpc header and potential assert() crashes when auth service returns 401

Description:
Combining a HTTP based external auth service with GRPC crashes envoy if the HTTP body returned from the external auth service has \r, \n or \0 in it and envoy decides to return it to the caller. This happens only when the request to the external auth service ends up denying access (401, but probably also on 403s). The root cause is that Utility::sendLocalReply() copies the external auth body text into a GrpMessage header without encoding the content (there is, in 1.10, even a TODO that this needs to be percent encoded). In our case it copies a pretty printed 401 JSON response into the header, and when the code in Filter::onComplete(...) calls HeaderString::setCopy, the latter has an an assert that verifies that the headers do not include \0, \r or \n (code in envoy/http/header_map.h's validHeaderString method).

(Caveat: We are using envoy through ambassador, but we checked with a plain upstream 1.10 release and could reproduce the issue. We also tested that removing the offending code did make it stop crashing)

Before going through this with code references, it is interesting to contemplate whether it makes sense to add the body as an header in the first place? Or is it actually added as a trailer but the code is tricking me?

All line numbers are relative to source corresponding to the v1.10 tag:

  1. A request comes into envoy, which is configured with ext_authz.
  2. this calls out to a configured external auth service, which in this case returns this as body:
{
  "error": {
    "code": 401,
    "message": "Unauthorized"
  }
}
  1. Since we are running a http based external auth, and we end up in Filter::onComplete in source/extensions/filters/http/ext_authz/ext_authz.cc, line 157. response->status == CheckStatus::Denied
    is true, so we enter the if-clause and callbacks_->sendLocalReply (line 160) is called.This ends up in source/common/http/async_client_impl.h's sendLocalReply which calls Utility::sendLocalReply (source/common/http/utility.cc) with the first value as true since it is a grpc request. This agains ends up in this code:
if (is_grpc) {
    HeaderMapPtr response_headers{new HeaderMapImpl{
        {Headers::get().Status, std::to_string(enumToInt(Code::OK))},
        {Headers::get().ContentType, Headers::get().ContentTypeValues.Grpc},
        {Headers::get().GrpcStatus,
         std::to_string(
             enumToInt(grpc_status ? grpc_status.value()
                                   : Grpc::Utility::httpToGrpcStatus(enumToInt(response_code))))}}};
    if (!body_text.empty() && !is_head_request) {
      // TODO: GrpcMessage should be percent-encoded
      response_headers->insertGrpcMessage().value(body_text);
    }
    encode_headers(std::move(response_headers), true); // Trailers only response
    return;
  }

Notice the TODO. Here we copy the body into a HeaderMapImpl (source/common/http/header_map_impl.cc), which then goes through it and is all happy until ti reaches ASSERT(valid()); on line 23, which verifies headers:

     bool HeaderString::valid() const { return validHeaderString(getStringView()); }

which again leads to:

// Used by ASSERTs to validate internal consistency. E.g. valid HTTP header keys/values should
// never contain embedded NULLs.
static inline bool validHeaderString(absl::string_view s) {
  for (const char c : {'\0', '\r', '\n'}) {
    if (s.find(c) != absl::string_view::npos) {
      return false;
    }
  }
  return true;
}

which returns false due to the newlines in the body, and then the assert hits and envoy shuts down.

Repro steps:

  1. Setup envoy with an external HTTP based auth service (we do this with ambassador), in our case the generated envoy config for the filter looks like this:
                                    "http_filters": [
                                        {
                                            "config": {
                                                "http_service": {
                                                    "authorization_request": {
                                                        "allowed_headers": {
                                                            "patterns": [ ] # removed for brevity
                                                        }
                                                    },
                                                    "authorization_response": {
                                                        "allowed_client_headers": {
                                                            "patterns": [] # removed for brevity
                                                    },
                                                    "authorization_response": {
                                                        "allowed_client_headers": {
                                                            "patterns": [] # removed for brevity
                                                        },
                                                        "allowed_upstream_headers": {
                                                            "patterns": [] # removed for brevity
                                                        }
                                                    },
                                                    "path_prefix": "/auth/ext",
                                                    "server_uri": {
                                                        "cluster": "cluster_extauth_lts_auth_9999",
                                                        "timeout": "5.000s",
                                                        "uri": "http://authservice/"
                                                    }
                                                }
                                            },
                                            "name": "envoy.ext_authz"
                                        },
  1. Define a grpc service, such as:
{
  "connect_timeout": "4.000s",
  "dns_lookup_family": "V4_ONLY",
  "http2_protocol_options": {},
  "lb_policy": "ROUND_ROBIN",
  "load_assignment": {
    "cluster_name": "cluster_grpc_example_50051",
    "endpoints": [
      {
        "lb_endpoints": [
          {
            "endpoint": {
              "address": {
                "socket_address": {
                  "address": "grpc-example",
                  "port_value": 50051,
                  "protocol": "TCP"
                }
              }
            }
          }
        ]
      }
    ]
  },
  "name": "cluster_grpc_example_50051",
  "type": "STRICT_DNS"
}

Here we use the example hello world service from grpc: https://github.com/grpc/grpc/tree/master/examples/python/helloworld

Logs and call stack:

[2019-05-10 22:42:54.560][67][debug][http] [source/common/http/conn_manager_impl.cc:580] [C22][S758412733360187786] request headers complete (end_stream=false):
':method', 'POST'
':scheme', 'http'
':path', '/helloworld.Greeter/SayHello'
':authority', 'localhost:8080'
'content-type', 'application/grpc'
'user-agent', 'grpc-go/1.20.0-dev'
'te', 'trailers'

[2019-05-10 22:42:54.560][67][trace][filter] [source/extensions/filters/http/ext_authz/ext_authz.cc:61] [C22][S758412733360187786] ext_authz filter calling authorization server
[2019-05-10 22:42:54.560][67][debug][router] [source/common/router/router.cc:320] [C0][S12701450807089142569] cluster 'cluster_extauth_lts_auth_9999' match for URL '/auth/ext/helloworld.Greeter/SayHello'
[2019-05-10 22:42:54.561][67][debug][router] [source/common/router/router.cc:381] [C0][S12701450807089142569] router decoding headers:
':method', 'POST'
':path', '/auth/ext/helloworld.Greeter/SayHello'
':authority', 'localhost:8080'
':scheme', 'http'
'content-length', '0'
'x-request-id', '97666d59-fc77-4edb-8f7f-3ff8343e6f76'
'x-forwarded-for', '172.19.20.1,172.19.20.5'
'x-forwarded-proto', 'http'
'user-agent', 'grpc-go/1.20.0-dev'
'x-envoy-internal', 'true'
'x-envoy-expected-rq-timeout-ms', '5000'

[2019-05-10 22:42:54.561][67][debug][pool] [source/common/http/http1/conn_pool.cc:88] creating a new connection
[2019-05-10 22:42:54.561][67][debug][client] [source/common/http/codec_client.cc:26] [C23] connecting
[2019-05-10 22:42:54.561][67][debug][connection] [source/common/network/connection_impl.cc:644] [C23] connecting to 172.19.99.8:9999
[2019-05-10 22:42:54.561][67][debug][connection] [source/common/network/connection_impl.cc:653] [C23] connection in progress
[2019-05-10 22:42:54.561][67][debug][pool] [source/common/http/conn_pool_base.cc:20] queueing request due to no available connections
[2019-05-10 22:42:54.561][67][trace][http] [source/common/http/conn_manager_impl.cc:814] [C22][S758412733360187786] decode headers called: filter=0x560236421540 status=1
[2019-05-10 22:42:54.561][67][trace][http2] [source/common/http/http2/codec_impl.cc:49] nghttp2: recv: [IB_READ_HEAD]
[2019-05-10 22:42:54.561][67][trace][http2] [source/common/http/http2/codec_impl.cc:49] nghttp2: recv: payloadlen=13, type=0, flags=0x01, stream_id=1
[2019-05-10 22:42:54.561][67][trace][http2] [source/common/http/http2/codec_impl.cc:49] nghttp2: recv: DATA
[2019-05-10 22:42:54.561][67][trace][http2] [source/common/http/http2/codec_impl.cc:49] nghttp2: recv: no padding in payload
[2019-05-10 22:42:54.561][67][trace][http2] [source/common/http/http2/codec_impl.cc:49] nghttp2: recv: [IB_READ_DATA]
[2019-05-10 22:42:54.561][67][trace][http2] [source/common/http/http2/codec_impl.cc:49] nghttp2: recv: readlen=13, payloadleft=0
[2019-05-10 22:42:54.561][67][trace][http2] [source/common/http/http2/codec_impl.cc:49] nghttp2: recv: data_readlen=13
[2019-05-10 22:42:54.561][67][trace][http2] [source/common/http/http2/codec_impl.cc:411] [C22] recv frame type=0
[2019-05-10 22:42:54.561][67][debug][http] [source/common/http/conn_manager_impl.cc:1037] [C22][S758412733360187786] request end stream
[2019-05-10 22:42:54.562][67][trace][http] [source/common/http/conn_manager_impl.cc:931] [C22][S758412733360187786] decode data called: filter=0x560236421540 status=2
[2019-05-10 22:42:54.562][67][trace][http2] [source/common/http/http2/codec_impl.cc:368] [C22] dispatched 107 bytes
[2019-05-10 22:42:54.562][67][trace][http2] [source/common/http/http2/codec_impl.cc:49] nghttp2: stream: adjusting kept idle streams num_idle_streams=0, max=100
[2019-05-10 22:42:54.562][67][trace][connection] [source/common/network/connection_impl.cc:440] [C23] socket event: 2
[2019-05-10 22:42:54.562][67][trace][connection] [source/common/network/connection_impl.cc:508] [C23] write ready
[2019-05-10 22:42:54.562][67][debug][connection] [source/common/network/connection_impl.cc:517] [C23] connected
[2019-05-10 22:42:54.562][67][debug][client] [source/common/http/codec_client.cc:64] [C23] connected
[2019-05-10 22:42:54.562][67][debug][pool] [source/common/http/http1/conn_pool.cc:245] [C23] attaching to next request
[2019-05-10 22:42:54.562][67][debug][router] [source/common/router/router.cc:1165] [C0][S12701450807089142569] pool ready
[2019-05-10 22:42:54.562][67][trace][connection] [source/common/network/connection_impl.cc:376] [C23] writing 311 bytes, end_stream false
[2019-05-10 22:42:54.562][67][trace][connection] [source/common/network/connection_impl.cc:508] [C23] write ready
[2019-05-10 22:42:54.562][67][trace][connection] [source/common/network/raw_buffer_socket.cc:66] [C23] write returns: 311
[2019-05-10 22:42:54.562][67][trace][connection] [source/common/network/connection_impl.cc:440] [C23] socket event: 2
[2019-05-10 22:42:54.562][67][trace][connection] [source/common/network/connection_impl.cc:508] [C23] write ready
[2019-05-10 22:42:54.662][67][trace][connection] [source/common/network/connection_impl.cc:440] [C23] socket event: 3
[2019-05-10 22:42:54.662][67][trace][connection] [source/common/network/connection_impl.cc:508] [C23] write ready
[2019-05-10 22:42:54.662][67][trace][connection] [source/common/network/connection_impl.cc:478] [C23] read ready
[2019-05-10 22:42:54.663][67][trace][connection] [source/common/network/raw_buffer_socket.cc:23] [C23] read returns: 260
[2019-05-10 22:42:54.663][67][trace][connection] [source/common/network/raw_buffer_socket.cc:37] [C23] read error: Resource temporarily unavailable
[2019-05-10 22:42:54.663][67][trace][http] [source/common/http/http1/codec_impl.cc:363] [C23] parsing 260 bytes
[2019-05-10 22:42:54.663][67][trace][http] [source/common/http/http1/codec_impl.cc:476] [C23] message begin
[2019-05-10 22:42:54.663][67][trace][http] [source/common/http/http1/codec_impl.cc:331] [C23] completed header: key=Date value=Fri, 10 May 2019 22:42:54 GMT
[2019-05-10 22:42:54.663][67][trace][http] [source/common/http/http1/codec_impl.cc:331] [C23] completed header: key=Content-Type value=application/json
[2019-05-10 22:42:54.663][67][trace][http] [source/common/http/http1/codec_impl.cc:331] [C23] completed header: key=X-Request-Id value=97666d59-fc77-4edb-8f7f-3ff8343e6f76
[2019-05-10 22:42:54.663][67][trace][http] [source/common/http/http1/codec_impl.cc:331] [C23] completed header: key=Vary value=Accept-Encoding
[2019-05-10 22:42:54.663][67][trace][http] [source/common/http/http1/codec_impl.cc:442] [C23] headers complete
[2019-05-10 22:42:54.663][67][trace][http] [source/common/http/http1/codec_impl.cc:331] [C23] completed header: key=Content-Length value=67
[2019-05-10 22:42:54.663][67][debug][router] [source/common/router/router.cc:717] [C0][S12701450807089142569] upstream headers complete: end_stream=false
[2019-05-10 22:42:54.664][67][debug][http] [source/common/http/async_client_impl.cc:94] async http request response headers (end_stream=false):
':status', '401'
'date', 'Fri, 10 May 2019 22:42:54 GMT'
'content-type', 'application/json'
'x-request-id', '97666d59-fc77-4edb-8f7f-3ff8343e6f76'
'vary', 'Accept-Encoding'
'content-length', '67'
'x-envoy-upstream-service-time', '101'

[2019-05-10 22:42:54.664][67][trace][http] [source/common/http/async_client_impl.cc:102] async http request response data (length=67 end_stream=false)
[2019-05-10 22:42:54.664][67][trace][http] [source/common/http/http1/codec_impl.cc:463] [C23] message complete
[2019-05-10 22:42:54.664][67][trace][http] [source/common/http/http1/codec_impl.cc:734] [C23] message complete
[2019-05-10 22:42:54.664][67][debug][client] [source/common/http/codec_client.cc:95] [C23] response complete
[2019-05-10 22:42:54.664][67][trace][main] [source/common/event/dispatcher_impl.cc:133] item added to deferred deletion list (size=1)
[2019-05-10 22:42:54.664][67][trace][http] [source/common/http/async_client_impl.cc:102] async http request response data (length=0 end_stream=true)
[2019-05-10 22:42:54.665][67][trace][filter] [source/extensions/filters/http/ext_authz/ext_authz.cc:153] [C22][S758412733360187786] ext_authz filter received status code 401
[2019-05-10 22:42:54.665][67][debug][filter] [source/extensions/filters/http/ext_authz/ext_authz.cc:159] [C22][S758412733360187786] ext_authz filter rejected the request
[2019-05-10 22:42:54.665][67][critical][assert] [source/common/http/header_map_impl.cc:213] assert failure: valid().
[2019-05-10 22:42:54.665][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:81] Caught Aborted, suspect faulting address 0x35
[2019-05-10 22:42:54.665][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:69] Backtrace (use tools/stack_decode.py to get line numbers):
[2019-05-10 22:42:54.693][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #0: Envoy::SignalAction::sigHandler() [0x56023416b2c8] ??:0
[2019-05-10 22:42:54.693][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #1: [0x7fdeeae1e4b0]
[2019-05-10 22:42:54.711][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #2: Envoy::Http::HeaderMapImpl::HeaderEntryImpl::value() [0x560233f770c2] ??:0
[2019-05-10 22:42:54.735][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #3: Envoy::Http::HeaderMapImpl::HeaderEntryImpl::value() [0x560233f77121] ??:0
[2019-05-10 22:42:54.756][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #4: Envoy::Http::Utility::sendLocalReply() [0x560233e86cbf] ??:0
[2019-05-10 22:42:54.778][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #5: Envoy::Http::ConnectionManagerImpl::ActiveStream::sendLocalReply() [0x560233befa90] ??:0
[2019-05-10 22:42:54.800][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #6: Envoy::Http::ConnectionManagerImpl::ActiveStreamDecoderFilter::sendLocalReply() [0x560233bf8bb5] ??:0
[2019-05-10 22:42:54.820][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #7: Envoy::Extensions::HttpFilters::ExtAuthz::Filter::onComplete() [0x560233194c39] ??:0
[2019-05-10 22:42:54.836][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #8: Envoy::Extensions::Filters::Common::ExtAuthz::RawHttpClientImpl::onSuccess() [0x56023319aae0] ??:0
[2019-05-10 22:42:54.854][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #9: Envoy::Http::AsyncRequestImpl::onComplete() [0x560233b6be3d] ??:0
[2019-05-10 22:42:54.871][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #10: Envoy::Http::AsyncRequestImpl::onData() [0x560233b6bfc8] ??:0
[2019-05-10 22:42:54.888][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #11: Envoy::Http::AsyncStreamImpl::encodeData() [0x560233b6b223] ??:0
[2019-05-10 22:42:54.904][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #12: Envoy::Router::Filter::onUpstreamData() [0x560233b7a018] ??:0
[2019-05-10 22:42:54.922][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #13: Envoy::Router::Filter::UpstreamRequest::decodeData() [0x560233b7c0e2] ??:0
[2019-05-10 22:42:54.938][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #14: Envoy::Http::StreamDecoderWrapper::decodeData() [0x560233ab6f21] ??:0
[2019-05-10 22:42:54.938][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #15: Envoy::Http::StreamDecoderWrapper::decodeData() [0x560233ab6f21] ??:0
[2019-05-10 22:42:54.956][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #16: Envoy::Http::Http1::ClientConnectionImpl::onMessageComplete() [0x560233c0f12f] ??:0
[2019-05-10 22:42:54.972][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #17: Envoy::Http::Http1::ConnectionImpl::onMessageCompleteBase() [0x560233c0cbf3] ??:0
[2019-05-10 22:42:54.990][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #18: Envoy::Http::Http1::ConnectionImpl::{lambda()#7}::operator()() [0x560233c0b578] ??:0
[2019-05-10 22:42:55.006][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #19: Envoy::Http::Http1::ConnectionImpl::{lambda()#7}::_FUN() [0x560233c0b59c] ??:0
[2019-05-10 22:42:55.023][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #20: http_parser_execute [0x560233ecf30a] ??:0
[2019-05-10 22:42:55.039][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #21: Envoy::Http::Http1::ConnectionImpl::dispatchSlice() [0x560233c0c46b] ??:0
[2019-05-10 22:42:55.056][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #22: Envoy::Http::Http1::ConnectionImpl::dispatch() [0x560233c0c30b] ??:0
[2019-05-10 22:42:55.080][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #23: Envoy::Http::CodecClient::onData() [0x560233b50830] ??:0
[2019-05-10 22:42:55.096][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #24: Envoy::Http::CodecClient::CodecReadFilter::onData() [0x560233b512da] ??:0
[2019-05-10 22:42:55.114][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #25: Envoy::Network::FilterManagerImpl::onContinueReading() [0x560233893a2d] ??:0
[2019-05-10 22:42:55.131][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #26: Envoy::Network::FilterManagerImpl::onRead() [0x560233893b44] ??:0
[2019-05-10 22:42:55.148][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #27: Envoy::Network::ConnectionImpl::onRead() [0x56023388a6f4] ??:0
[2019-05-10 22:42:55.164][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #28: Envoy::Network::ConnectionImpl::onReadReady() [0x56023388bebb] ??:0
[2019-05-10 22:42:55.181][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #29: Envoy::Network::ConnectionImpl::onFileEvent() [0x56023388bc59] ??:0
[2019-05-10 22:42:55.200][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #30: Envoy::Network::ConnectionImpl::ConnectionImpl()::{lambda()#3}::operator()() [0x56023388843f] ??:0
[2019-05-10 22:42:55.217][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #31: std::_Function_handler<>::_M_invoke() [0x56023388dda3] ??:0
[2019-05-10 22:42:55.235][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #32: std::function<>::operator()() [0x560233881f42] ??:0
[2019-05-10 22:42:55.251][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #33: Envoy::Event::FileEventImpl::assignEvents()::{lambda()#1}::operator()() [0x560233881b9e] ??:0
[2019-05-10 22:42:55.269][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #34: Envoy::Event::FileEventImpl::assignEvents()::{lambda()#1}::_FUN() [0x560233881c1c] ??:0
[2019-05-10 22:42:55.285][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #35: event_process_active_single_queue.isra.31 [0x560233ea7203] ??:0
[2019-05-10 22:42:55.302][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #36: event_process_active [0x560233ea775f] ??:0
[2019-05-10 22:42:55.318][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #37: event_base_loop [0x560233eabe10] ??:0
[2019-05-10 22:42:55.336][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #38: Envoy::Event::LibeventScheduler::run() [0x5602338a0f0a] ??:0
[2019-05-10 22:42:55.352][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #39: Envoy::Event::DispatcherImpl::run() [0x56023387cfb8] ??:0
[2019-05-10 22:42:55.370][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #40: Envoy::Server::WorkerImpl::threadRoutine() [0x56023386a84b] ??:0
[2019-05-10 22:42:55.387][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #41: Envoy::Server::WorkerImpl::start()::{lambda()#1}::operator()() [0x56023386a11a] ??:0
[2019-05-10 22:42:55.404][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #42: std::_Function_handler<>::_M_invoke() [0x56023386b159] ??:0
[2019-05-10 22:42:55.421][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #43: std::function<>::operator()() [0x5602331c3878] ??:0
[2019-05-10 22:42:55.438][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #44: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::operator()() [0x56023416e344] ??:0
[2019-05-10 22:42:55.455][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:73] #45: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::{lambda()#1}::_FUN() [0x56023416e368] ??:0
[2019-05-10 22:42:55.456][67][critical][backtrace] [bazel-out/k8-fastbuild/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #46: [0x7fdeeae13cf8]

Possible fixes:

  1. Fix the TODO and encode the body
  2. Stop putting the body into the header, but somewhere else (trailer?)? If so, make sure that doesn't need encoding.

I am not (yet) that familiar with GRPC, but 2) sounds pretty good for me.
@dio dio added the bug label May 12, 2019
@mattklein123
Copy link
Member

cc @gsagula

@mattklein123 mattklein123 added this to the 1.11.0 milestone May 13, 2019
@dio dio mentioned this issue May 13, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
1.11.0
Development

Successfully merging a pull request may close this issue.

http: Change sendLocalReply to send percent-encoded GrpcMessage
3 participants
@dio @mattklein123 @thorkildcognite