Multiple layers of server/client spans in proxied service tracing

[objectiser]

Although #86 discussed the issue of tracing when using proxies (sidecars), and that the proxied service/app may handle distributed context propagation directly or through use of a tracer, it didn't touch on how the resulting trace data should be produced in respect of client/server spans.

If we take the first scenario, where the service is propagating the context - the inbound proxy would create a server span and pass the request to the service, which may then forward the trace context to outbound requests, with the outbound proxy creating a client span.

However if the service now wants to participate in tracing (i.e. creating internal spans), or we want to leverage a tracer to handle the context propagation on behalf of the app, then we may want to use framework instrumentations that can do this in a non-intrusive way (i.e. ot-contrib java-spring-cloud). The issue with this approach is that the framework instrumentations would create their own server and client spans on behalf of the service - effectively creating a double layer of service and client spans.

Its also possible that there may be internal spans between the two server spans, or two client spans, representing internal activities being performed by the proxy on behalf of the service.

Having the additional spans from inside the service is useful, as it shows the latency between the proxy and service dealing with the request - however both spans being classified with the same 'span.kind' may be misleading or affect post processing in some tracing systems.

Should multiple layers of the same span kind be allowed?

If yes, then does this scenario need to be explained in the spec so that tracer implementations handle it correctly.

If no, we possibly need a standard way to inform framework instrumentations they are operating in a proxied service, so they can suitably adjust the way they report spans.

[objectiser]

@tedsuo @bhs @yurishkuro @adriancole @pavolloffay thoughts?

[yurishkuro]

we possibly need a standard way to inform framework instrumentations they are operating in a proxied service

even if we did that, there is still a problem of in-process propagation. If the application is only using OT API, then the only way to propagate the context is to extract SpanContext, create a child span, and pass that span using the Context or Scope mechanisms. So even if the application wants to rely purely on the tracing done by the proxies, it still needs to create a new span, there's no other way it can pass the span context to outbound calls (of course it can just propagate the SpanContext object in-process but that's not supported by any OT APIs today).

[objectiser]

@yurishkuro By the 'no' option, I meant that the spans would be created still by the framework instrumentations, but possibly don't specify the span.kind tag, so would just be treated as internal instead.

Had also thought about having SpanContext propagation aswell, but as you say its not supported and would add extra complexity.

[objectiser]

Closing, as does not appear to be an issue with Jaeger or Zipkin in Istio, when spans generated both in the proxy and service. It has the benefit of providing a clear indication of any latency added by the proxy.