diff --git a/docs/core/diagnostics/distributed-tracing-collection-walkthroughs.md b/docs/core/diagnostics/distributed-tracing-collection-walkthroughs.md new file mode 100644 index 0000000000000..fcb5001b63814 --- /dev/null +++ b/docs/core/diagnostics/distributed-tracing-collection-walkthroughs.md @@ -0,0 +1,368 @@ +--- +title: Collect a distributed trace - .NET +description: Tutorials to collect distributed traces in .NET applications using OpenTelemetry, Application Insights, or ActivityListener +ms.topic: tutorial +ms.date: 03/14/2021 +--- + +# Collect a distributed trace + +**This article applies to: ✔️** .NET Core 2.1 and later versions **and** .NET Framework 4.5 and later versions + +Instrumented code can create objects as part of a distributed trace, but the information +in these objects needs to be collected into centralized storage so that the entire trace can be +reviewed later. In this tutorial you will collect the distributed trace telemetry in different ways so that it is +available to diagnose application issues when needed. See +[the instrumentation tutorial](distributed-tracing-instrumentation-walkthroughs.md) if you need to add new instrumentation. + +## Collect traces using OpenTelemetry + +### Prerequisites + +- [.NET Core 2.1 SDK](https://dotnet.microsoft.com/download/dotnet) or a later version + +### Create an example application + +Before any distributed trace telemetry can be collected we need to produce it. Often this instrumentation might be +in libraries but for simplicity you will create a small app that has some example instrumentation using +. At this point no collection is happening yet, +StartActivity() has no side-effect and returns null. See +[the instrumentation tutorial](distributed-tracing-instrumentation-walkthroughs.md) for more details. + +```dotnetcli +dotnet new console +``` + +Applications that target .NET 5 and later already have the necessary distributed tracing APIs included. For apps targeting older +.NET versions add the [System.Diagnostics.DiagnosticSource NuGet package](https://www.nuget.org/packages/System.Diagnostics.DiagnosticSource/) +version 5 or greater. + +```dotnetcli +dotnet add package System.Diagnostics.DiagnosticSource +``` + +Replace the contents of the generated Program.cs with this example source: + +```C# +using System; +using System.Diagnostics; +using System.Threading.Tasks; + +namespace Sample.DistributedTracing +{ + class Program + { + static ActivitySource s_source = new ActivitySource("Sample.DistributedTracing"); + + static async Task Main(string[] args) + { + await DoSomeWork(); + Console.WriteLine("Example work done"); + } + + static async Task DoSomeWork() + { + using (Activity a = s_source.StartActivity("SomeWork")) + { + await StepOne(); + await StepTwo(); + } + } + + static async Task StepOne() + { + using (Activity a = s_source.StartActivity("StepOne")) + { + await Task.Delay(500); + } + } + + static async Task StepTwo() + { + using (Activity a = s_source.StartActivity("StepTwo")) + { + await Task.Delay(1000); + } + } + } +} +``` + +Running the app does not collect any trace data yet: + +```dotnetcli +> dotnet run +Example work done +``` + +### Collect using OpenTelemetry + +[OpenTelemetry](https://opentelemetry.io/) is a vendor neutral open source project supported by the +[Cloud Native Computing Foundation](https://www.cncf.io/) that aims to standardize generating and collecting telemetry for +cloud-native software. In this example you will collect and display distributed trace information on the console though +OpenTelemetry can be reconfigured to send it elsewhere. See the +[OpenTelemetry getting started guide](https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/docs/trace/getting-started/README.md) +for more information. + +Add the [OpenTelemetry.Exporter.Console](https://www.nuget.org/packages/OpenTelemetry.Exporter.Console/) NuGet package. + +```dotnetcli +dotnet add package OpenTelemetry.Exporter.Console +``` + +Update Program.cs with additional OpenTelemetry using statments: + +```C# +using OpenTelemetry; +using OpenTelemetry.Resources; +using OpenTelemetry.Trace; +using System; +using System.Diagnostics; +using System.Threading.Tasks; +``` + +Update Main() to create the OpenTelemetry TracerProvider: + +```C# + public static async Task Main() + { + using var tracerProvider = Sdk.CreateTracerProviderBuilder() + .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("MySample")) + .AddSource("Sample.DistributedTracing") + .AddConsoleExporter() + .Build(); + + await DoSomeWork(); + Console.WriteLine("Example work done"); + } +``` + +Now the app collects distributed trace information and displays it to the console: + +```dotnetcli +> dotnet run +Activity.Id: 00-7759221f2c5599489d455b84fa0f90f4-6081a9b8041cd840-01 +Activity.ParentId: 00-7759221f2c5599489d455b84fa0f90f4-9a52f72c08a9d447-01 +Activity.DisplayName: StepOne +Activity.Kind: Internal +Activity.StartTime: 2021-03-18T10:46:46.8649754Z +Activity.Duration: 00:00:00.5069226 +Resource associated with Activity: + service.name: MySample + service.instance.id: 909a4624-3b2e-40e4-a86b-4a2c8003219e + +Activity.Id: 00-7759221f2c5599489d455b84fa0f90f4-d2b283db91cf774c-01 +Activity.ParentId: 00-7759221f2c5599489d455b84fa0f90f4-9a52f72c08a9d447-01 +Activity.DisplayName: StepTwo +Activity.Kind: Internal +Activity.StartTime: 2021-03-18T10:46:47.3838737Z +Activity.Duration: 00:00:01.0142278 +Resource associated with Activity: + service.name: MySample + service.instance.id: 909a4624-3b2e-40e4-a86b-4a2c8003219e + +Activity.Id: 00-7759221f2c5599489d455b84fa0f90f4-9a52f72c08a9d447-01 +Activity.DisplayName: SomeWork +Activity.Kind: Internal +Activity.StartTime: 2021-03-18T10:46:46.8634510Z +Activity.Duration: 00:00:01.5402045 +Resource associated with Activity: + service.name: MySample + service.instance.id: 909a4624-3b2e-40e4-a86b-4a2c8003219e + +Example work done +``` + +#### Sources + +In the example code you invoked `AddSource("Sample.DistributedTracing")` so that OpenTelemetry would +capture the Activities produced by the ActivitySource that was already present in the code: + +```csharp + static ActivitySource s_source = new ActivitySource("Sample.DistributedTracing"); +``` + +Telemetry from any ActivitySource can captured by calling AddSource() with the source's name. + +#### Exporters + +The console exporter is helpful for quick examples or local development but in a production deployment +you will probably want to send traces to a centralized store. OpenTelemetry supports a variety +of destinations using different +[exporters](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/glossary.md#exporter-library). +See the [OpenTelemetry getting started guide](https://github.com/open-telemetry/opentelemetry-dotnet#getting-started) +for more information on configuring OpenTelemetry. + +## Collect traces using Application Insights + +Distributed tracing telemetry is automatically captured after configuring the Application Insights SDK +([ASP.NET](https://docs.microsoft.com/azure/azure-monitor/app/asp-net), [ASP.NET Core](https://docs.microsoft.com/azure/azure-monitor/app/asp-net-core)) +or by enabling [code-less instrumentation](https://docs.microsoft.com/azure/azure-monitor/app/codeless-overview). + +See the [Application Insights distributed tracing documentation](https://docs.microsoft.com/azure/azure-monitor/app/distributed-tracing) for more +information. + +> [!NOTE] +> Currently Application Insights only supports collecting specific well-known Activity instrumentation and will ignore new user added Activities. Application +> Insights offers [TrackDependency](https://docs.microsoft.com/azure/azure-monitor/app/api-custom-events-metrics#trackdependency) as a vendor +> specific API for adding custom distributed tracing information. + +## Collect traces using custom logic + +Developers are free to create their own customized collection logic for Activity trace data. This example collects the +telemetry using the API provided by .NET and prints +it to the console. + +### Prerequisites + +- [.NET Core 2.1 SDK](https://dotnet.microsoft.com/download/dotnet) or a later version + +### Create an example application + +First you will create an example application that has some distributed trace instrumentation but no trace data is being collected. + +```dotnetcli +dotnet new console +``` + +Applications that target .NET 5 and later already have the necessary distributed tracing APIs included. For apps targeting older +.NET versions add the [System.Diagnostics.DiagnosticSource NuGet package](https://www.nuget.org/packages/System.Diagnostics.DiagnosticSource/) +version 5 or greater. + +```dotnetcli +dotnet add package System.Diagnostics.DiagnosticSource +``` + +Replace the contents of the generated Program.cs with this example source: + +```C# +using System; +using System.Diagnostics; +using System.Threading.Tasks; + +namespace Sample.DistributedTracing +{ + class Program + { + static ActivitySource s_source = new ActivitySource("Sample.DistributedTracing"); + + static async Task Main(string[] args) + { + await DoSomeWork(); + Console.WriteLine("Example work done"); + } + + static async Task DoSomeWork() + { + using (Activity a = s_source.StartActivity("SomeWork")) + { + await StepOne(); + await StepTwo(); + } + } + + static async Task StepOne() + { + using (Activity a = s_source.StartActivity("StepOne")) + { + await Task.Delay(500); + } + } + + static async Task StepTwo() + { + using (Activity a = s_source.StartActivity("StepTwo")) + { + await Task.Delay(1000); + } + } + } +} +``` + +Running the app does not collect any trace data yet: + +```dotnetcli +> dotnet run +Example work done +``` + +### Add code to collect the traces + +Update Main() with this code: + +```C# + static async Task Main(string[] args) + { + Activity.DefaultIdFormat = ActivityIdFormat.W3C; + Activity.ForceDefaultIdFormat = true; + + Console.WriteLine(" {0,-15} {1,-60} {2,-15}", "OperationName", "Id", "Duration"); + ActivitySource.AddActivityListener(new ActivityListener() + { + ShouldListenTo = (source) => true, + Sample = (ref ActivityCreationOptions options) => ActivitySamplingResult.AllDataAndRecorded, + ActivityStarted = activity => Console.WriteLine("Started: {0,-15} {1,-60}", activity.OperationName, activity.Id), + ActivityStopped = activity => Console.WriteLine("Stopped: {0,-15} {1,-60} {2,-15}", activity.OperationName, activity.Id, activity.Duration) + }); + + await DoSomeWork(); + Console.WriteLine("Example work done"); + } +``` + +The output now includes logging: + +```dotnetcli +> dotnet run + OperationName Id Duration +Started: SomeWork 00-bdb5faffc2fc1548b6ba49a31c4a0ae0-c447fb302059784f-01 +Started: StepOne 00-bdb5faffc2fc1548b6ba49a31c4a0ae0-a7c77a4e9a02dc4a-01 +Stopped: StepOne 00-bdb5faffc2fc1548b6ba49a31c4a0ae0-a7c77a4e9a02dc4a-01 00:00:00.5093849 +Started: StepTwo 00-bdb5faffc2fc1548b6ba49a31c4a0ae0-9210ad536cae9e4e-01 +Stopped: StepTwo 00-bdb5faffc2fc1548b6ba49a31c4a0ae0-9210ad536cae9e4e-01 00:00:01.0111847 +Stopped: SomeWork 00-bdb5faffc2fc1548b6ba49a31c4a0ae0-c447fb302059784f-01 00:00:01.5236391 +Example work done +``` + +Setting and + is optional +but helps ensure the sample produces similar output on different .NET runtime versions. .NET 5 uses +the W3C TraceContext ID format by default but earlier .NET versions default to using + ID format. See +[Activity IDs](distributed-tracing-concepts.md#activity-ids) for more details. + + is used to receive callbacks +during the lifetime of an Activity. + +- - Each +Activity is associated with an ActivitySource which acts as its namespace and producer. +This callback is invoked once for each ActivitySource in the process. Return true +if you are interested in performing sampling or being notified about start/stop events +for Activities produced by this source. +- - By default + does not +create an Activity object unless some ActivityListener indicates it should be sampled. Returning + +indicates that the Activity should be created, + should be set +to true, and +will have the +flag set. IsAllDataRequested can be observed by the instrumented code as a hint that a listener +wants to ensure that auxilliary Activity information such as Tags and Events are populated. +The Recorded flag is encoded in the W3C TraceContext ID and is a hint to other processes +involved in the distributed trace that this trace should be sampled. +- and + are +called when an Activity is started and stopped respectively. These callbacks provide an +oportunity to record relevant information about the Activity or potentially to modify it. +When an Activity has just started much of the data may still be incomplete and it will +be populated before the Activity stops. + +Once an ActivityListener has been created and the callbacks are populated, calling + +initiates invoking the callbacks. Call + to +stop the flow of callbacks. Beware that in multi-threaded code callback notifications in +progress could be received while Dispose() is running or even very shortly after it has +returned. diff --git a/docs/core/diagnostics/distributed-tracing-concepts.md b/docs/core/diagnostics/distributed-tracing-concepts.md new file mode 100644 index 0000000000000..99cdc65383d5a --- /dev/null +++ b/docs/core/diagnostics/distributed-tracing-concepts.md @@ -0,0 +1,138 @@ +--- +title: Distributed tracing concepts - .NET +description: .NET distributed tracing concepts +ms.date: 03/14/2021 +--- + +# .NET Distributed Tracing Concepts + +Distributed tracing is a diagnostic technique that helps engineers localize failures and +performance issues within applications, especially those that may be distributed across +multiple machines or processes. See the [Distributed Tracing Overview](distributed-tracing.md) +for general information about where distributed tracing is useful and example code to get +started. + +### Traces and Activities + +Each time a new request is received by an application it can be associated with a trace. In +application components written in .NET, units of work in a trace are represented by instances of + and the trace as a whole forms +a tree of these Activities, potentially spanning across many distinct processes. The first +Activity created for a new request forms the root of the trace tree and it tracks the overall +duration and success/failure handling the request. Child activities can be optionally created +to sub-divide the work into different steps that can be tracked individually. +For example given an Activity that tracked a specific inbound HTTP request in a web server, +child activites could be created to track each of the database queries that were necessary to +complete the request. This allows the duration and success for each query to be recorded independently. +Activities can record other information for each unit of work such as +, name-value pairs +called , and . The +name identifies the type of work being performed, tags can record descriptive parameters of the work, +and events are a simple logging mechanism to record timestamped diagnostic messages. + +> [!NOTE] +> Another common industry name for units of work in a distributed trace are 'Spans'. +> .NET adopted the term 'Activity' many years ago, before the name 'Span' was well +> established for this concept. + +### Activity IDs + +Parent-Child relationships between Activities in the distributed trace tree are established +using unique IDs. .NET's implementation of distributed tracing supports two ID schemes, the W3C +standard [TraceContext](https://www.w3.org/TR/trace-context/) which is the default in .NET 5 and +an older .NET convention called 'Hierarchical' that is available for backwards compatibility. + controls which +ID scheme is used. In the W3C TraceContext standard every trace is assigned a globally unique 16 +byte trace-id () and +every Activity within the trace is assigned a unique 8 byte span-id +(). Each Activity +records the trace-id, its own span-id, and the span-id of its parent +(). Because +distributed traces can track work across process boundaries parent and child Activities may +not be in the same process. The combination of a trace-id and parent span-id can uniquely +identify the parent Activity globally, regardless of what process it resides in. + + controls which +ID format is used for starting new traces, but by default adding a new Activity to an existing +trace uses whatever format the parent Activity is using. +Setting +to true overrides this behavior and creates all new Activities with the DefaultIdFormat, even +when the parent uses a different ID format. + +### Starting and stopping Activities + +Each thread in a process may have a corresponding Activity object that is tracking the work +occuring on that thread, accessible via +. The current activity +automatically flows along all synchronous calls on a thread as well as following async calls +that are processed on different threads. If Activity A is the current activity on a thread and +code starts a new Activity B then B becomes the new current activity on that thread. By default +activity B will also treat Activity A as its parent. When Activity B is later stopped activity +A will be restored as the current Activity on the thread. When an Activity is started it +captures the current time as the +. When it +stops is calculated +as the difference between the current time and the start time. + +### Coordinating across process boundaries + +In order to track work across process boundaries Activity parent IDs need to be transmitted across +the network so that the receiving process can create Activities that refer to them. When using +W3C TraceContext ID format .NET will also use the HTTP headers recommended by +[the standard](https://www.w3.org/TR/trace-context/) to transmit this information. When using the + ID format +.NET uses a custom request-id HTTP header to transmit the ID. Unlike many other language runtimes +.NET in-box libraries such as the ASP.NET web server and System.Net.Http natively understand how to +decode and encode Activity IDs on HTTP messages. The runtime also understands how to flow the ID +through sychronous and asynchronous calls. This means that .NET applications that receive and +emit HTTP messages participate in flowing distributed trace IDs automatically, with no special +coding by the app developer nor 3rd party library dependencies. 3rd party libraries may add +support for transmitting IDs over non-HTTP message protocols or supporting custom encoding +conventions for HTTP. + +### Collecting traces + +Instrumented code can create objects +as part of a distributed trace, but the information in these objects needs to be transmitted +and serialized in a centralized persistant store so that the entire trace can be usefully reviewed +later. There are several telemetry collection libraries that can do this task such as +[Application Insights](https://docs.microsoft.com/azure/azure-monitor/app/distributed-tracing), +[OpenTelemetry](https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/docs/trace/getting-started/README.md), +or a library provided by a 3rd party telemetry or APM vendor. Alternately developers can author +their own custom Activity telemetry collection by using + or +. ActivityListener +supports observing any Activity regardless whether the developer has any a-priori knowledge about it. +This makes ActivityListener a simple and flexible general purpose solution. By contrast using +DiagnosticListener is a more complex scenario that requires the instrumented code to opt-in by +invoking and +the collection library needs to know the exact naming information that the instrumented code +used when starting it. Using DiagnosticSource and DiagnosticListener allows the creator +and listener to exchange arbitrary .NET objects and establish customized information passing +conventions. + +### Sampling + +For improved performance in high throughput applications, distributed tracing on .NET supports +sampling only a subset of traces rather than recording all of them. For activites created with +the recommended +API, telemetry collection libraries can control sampling with the + callback. +The logging library can elect not to create the Activity at all, to create it with minimal +information necessary to propagate distributing tracing IDs, or to populate it with complete +diagnostic information. These choices trade-off increasing performance overhead for +increasing diagnostic utility. Activities that are started using the older pattern of invoking + and + may +also support DiagnosticListener sampling by first calling +. +Even when capturing full diagnostic information the .NET +implementation is designed to be fast - coupled with an efficient collector an Activity can be +created, populated, and transmitted in about a microsecond on modern hardware. Sampling +can reduce the instrumentation cost to less than 100 nanoseconds for each Activity that isn't +recorded. + +## Next steps + +See the [Distributed Tracing Overview](distributed-tracing.md) for example code to get started +using distributed tracing in .NET applications. diff --git a/docs/core/diagnostics/distributed-tracing-instrumentation-walkthroughs.md b/docs/core/diagnostics/distributed-tracing-instrumentation-walkthroughs.md new file mode 100644 index 0000000000000..f40f548b21bd4 --- /dev/null +++ b/docs/core/diagnostics/distributed-tracing-instrumentation-walkthroughs.md @@ -0,0 +1,459 @@ +--- +title: Add distributed tracing instrumentation - .NET +description: A tutorial to instrument distributed traces in .NET applications +ms.topic: tutorial +ms.date: 03/14/2021 +--- + +# Adding distributed tracing instrumentation + +**This article applies to: ✔️** .NET Core 2.1 and later versions **and** .NET Framework 4.5 and later versions + +.NET applications can be instrumented using the API to produce +distributed tracing telemetry. Some instrumentation is built-in to standard .NET libraries but you may want to add more to make +your code more easily diagnosable. In this tutorial you will add new custom distributed tracing instrumentation. See +[the collection tutorial](distributed-tracing-instrumentation-walkthroughs.md) to learn more about recording the telemetry +produced by this instrumentation. + +## Prerequisites + +- [.NET Core 2.1 SDK](https://dotnet.microsoft.com/download/dotnet) or a later version + +## An initial app + +First you will create a sample app that collects telemetry using OpenTelemetry, but doesn't yet have any instrumentation. + +```dotnetcli +dotnet new console +``` + +Applications that target .NET 5 and later already have the necessary distributed tracing APIs included. For apps targeting older +.NET versions add the [System.Diagnostics.DiagnosticSource NuGet package](https://www.nuget.org/packages/System.Diagnostics.DiagnosticSource/) +version 5 or greater. + +```dotnetcli +dotnet add package System.Diagnostics.DiagnosticSource +``` + +Add the [OpenTelemetry](https://www.nuget.org/packages/OpenTelemetry/) and +[OpenTelemetry.Exporter.Console](https://www.nuget.org/packages/OpenTelemetry.Exporter.Console/) NuGet packages +which will be used to collect the telemetry. + +```dotnetcli +dotnet add package OpenTelemetry +dotnet add package OpenTelemetry.Exporter.Console +``` + +Replace the contents of the generated Program.cs with this example source: + +```C# +using OpenTelemetry; +using OpenTelemetry.Resources; +using OpenTelemetry.Trace; +using System; +using System.Threading.Tasks; + +namespace Sample.DistributedTracing +{ + class Program + { + static async Task Main(string[] args) + { + using var tracerProvider = Sdk.CreateTracerProviderBuilder() + .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("MySample")) + .AddSource("Sample.DistributedTracing") + .AddConsoleExporter() + .Build(); + + await DoSomeWork("banana", 8); + Console.WriteLine("Example work done"); + } + + // All the functions below simulate doing some arbitrary work + static async Task DoSomeWork(string foo, int bar) + { + await StepOne(); + await StepTwo(); + } + + static async Task StepOne() + { + await Task.Delay(500); + } + + static async Task StepTwo() + { + await Task.Delay(1000); + } + } +} +``` + +The app has no instrumentation yet so there is no trace information to display: + +```dotnetcli +> dotnet run +Example work done +``` + +#### Best Practices + +Only app developers need to reference an optional 3rd party library for collecting the +distributed trace telemetry, such as OpenTelemetry in this example. .NET library +authors can exclusively rely on APIs in System.Diagnostics.DiagnosticSource which is part +of .NET runtime. This ensures that libraries will run in a wide range of .NET apps, regardless +of the app developer's preferences about which library or vendor to use for collecting +telemetry. + +## Adding Basic Instrumentation + +Applications and libraries add distributed tracing instrumentation using the + and + classes. + +### ActivitySource + +First create an instance of ActivitySource. ActivitySource provides APIs to create and +start Activity objects. Add the static ActivitySource variable above Main() and +`using System.Diagnostics;` to the using statements. + +```csharp +using OpenTelemetry; +using OpenTelemetry.Resources; +using OpenTelemetry.Trace; +using System; +using System.Diagnostics; +using System.Threading.Tasks; + +namespace Sample.DistributedTracing +{ + class Program + { + private static ActivitySource source = new ActivitySource("Sample.DistributedTracing", "1.0.0"); + + static async Task Main(string[] args) + { + ... +``` + +#### Best Practices + +- Create the ActivitySource once, store it in a static variable and use that instance as long as needed. +Each library or library sub-component can (and often should) create its own source. Consider creating a new +source rather than re-using an existing one if you anticipate app developers would appreciate being able to +enable and disable the Activity telemetry in the sources independently. + +- The source name passed to the constructor has to be unique to avoid the conflicts with any other sources. +It is recommended to use a hierarchical name that contains the assembly name and optionally a component name if +there are multiple sources within the same assembly. For example, `Microsoft.AspNetCore.Hosting`. If an assembly +is adding instrumentation for code in a 2nd independent assembly, the name should be based on the +assembly that defines the ActivitySource, not the assembly whose code is being instrumented. + +- The version parameter is optional. It is recommended to provide the version in case you release multiple +versions of the library and make changes to the instrumented telemetry. + +> [!NOTE] +> OpenTelemetry uses alternate terms 'Tracer' and 'Span'. In .NET 'ActivitySource' is the implementation +> of Tracer and Activity is the implementation of 'Span'. .NET's Activity type long pre-dates +> the OpenTelemetry specification and the original .NET naming has been preserved for +> consistency within the .NET ecosystem and .NET application compatibility. + +### Activity Creation + +Use the ActivitySource object to Start and Stop Activity objects around meaningful units of work. Update +DoSomeWork() with the code shown here: + +```csharp + static async Task DoSomeWork(string foo, int bar) + { + using (Activity activity = source.StartActivity("SomeWork")) + { + await StepOne(); + await StepTwo(); + } + } +``` + +Running the app now shows the new Activity being logged: + +```dotnetcli +> dotnet run +Activity.Id: 00-f443e487a4998c41a6fd6fe88bae644e-5b7253de08ed474f-01 +Activity.DisplayName: SomeWork +Activity.Kind: Internal +Activity.StartTime: 2021-03-18T10:36:51.4720202Z +Activity.Duration: 00:00:01.5025842 +Resource associated with Activity: + service.name: MySample + service.instance.id: 067f4bb5-a5a8-4898-a288-dec569d6dbef +``` + +#### Notes + +- creates and starts +the activity at the same time. The listed code pattern is using the `using` block which automatically disposes +the created Activity object after executing the block. Disposing the Activity object will stop it so the code +doesn't need to explicitly call . +That simplifies the coding pattern. + +- internally determines if +there are any listeners recording the Activity. If there are no registered listeners or there are listeners which +are not interested, StartActivity() will return `null` and avoid creating the Activity object. This +is a performance optimization so that the code pattern can still be used in functions that are called very +frequently. + +## Optional: Populate tags + +Activities support key-value data called Tags, commonly used to store any parameters of the work that +may be useful for diagnostics. Update DoSomeWork() to include them: + +```csharp + static async Task DoSomeWork(string foo, int bar) + { + using (Activity activity = source.StartActivity("SomeWork")) + { + activity?.SetTag("foo", foo); + activity?.SetTag("bar", bar); + await StepOne(); + await StepTwo(); + } + } +``` + +```dotnetcli +> dotnet run +Activity.Id: 00-2b56072db8cb5a4496a4bfb69f46aa06-7bc4acda3b9cce4d-01 +Activity.DisplayName: SomeWork +Activity.Kind: Internal +Activity.StartTime: 2021-03-18T10:37:31.4949570Z +Activity.Duration: 00:00:01.5417719 +Activity.TagObjects: + foo: banana + bar: 8 +Resource associated with Activity: + service.name: MySample + service.instance.id: 25bbc1c3-2de5-48d9-9333-062377fea49c + +Example work done +``` + +#### Best Practices + +- As mentioned above, `activity` returned by +may be null. The null-coallescing operator ?. in C# is a convenient short-hand to only invoke + if activity is not null. The behavior is identical to +writing: + +```csharp +if(activity != null) +{ + activity.SetTag("foo", foo); +} +``` + +- OpenTelemetry provides a set of recommended +[conventions](https://github.com/open-telemetry/opentelemetry-specification/tree/main/specification/trace/semantic_conventions) +for setting Tags on Activities that represent common types of application work. + +- If you are instrumenting functions with high performance requirements, + is a hint that indicates whether any +of the code listening to Activities intends to read auxilliary information such as Tags. If no listener will read it then there +is no need for the instrumented code to spend CPU cycles populating it. For simplicity this sample doesn't apply that +optimization. + +## Optional: Adding Events + +Events are timestamped messages that can attach an arbitrary stream of additional diagnostic data to Activities. Add +some events to the Activity: + +```csharp + static async Task DoSomeWork(string foo, int bar) + { + using (Activity activity = source.StartActivity("SomeWork")) + { + activity?.SetTag("foo", foo); + activity?.SetTag("bar", bar); + await StepOne(); + activity?.AddEvent(new ActivityEvent("Part way there")); + await StepTwo(); + activity?.AddEvent(new ActivityEvent("Done now")); + } + } +``` + +```dotnetcli +> dotnet run +Activity.Id: 00-82cf6ea92661b84d9fd881731741d04e-33fff2835a03c041-01 +Activity.DisplayName: SomeWork +Activity.Kind: Internal +Activity.StartTime: 2021-03-18T10:39:10.6902609Z +Activity.Duration: 00:00:01.5147582 +Activity.TagObjects: + foo: banana + bar: 8 +Activity.Events: + Part way there [3/18/2021 10:39:11 AM +00:00] + Done now [3/18/2021 10:39:12 AM +00:00] +Resource associated with Activity: + service.name: MySample + service.instance.id: ea7f0fcb-3673-48e0-b6ce-e4af5a86ce4f + +Example work done +``` + +#### Best Practices + +- Events are stored in an in-memory list until they can be transmitted which makes this mechanism only suitable for +recording a modest number of events. For a large or unbounded volume of events using a logging API focused on this task +such as [ILogger](https://docs.microsoft.com/aspnet/core/fundamentals/logging/) is a better choice. ILogger also ensures +that the logging information will be available regardless whether the app developer opts to use distributed tracing. +ILogger supports automatically capturing the active Activity IDs so messages logged via that API can still be correlated +with the distributed trace. + +## Optional: Adding Status + +OpenTelemetry allows each Activity to report a +[Status](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#set-status) +that represents the pass/fail result of the work. .NET does not currently have a strongly typed API for this purpose but +there is an established convention using Tags: + +- `otel.status_code` is the Tag name used to store `StatusCode`. Values for the StatusCode tag must be one of the +strings "UNSET", "OK", or "ERROR", which correspond respectively to the enums `Unset`, `Ok`, and `Error` from StatusCode. +- `otel.status_description` is the Tag name used to store the optional `Description` + +Update DoSomeWork() to set status: + +```csharp + static async Task DoSomeWork(string foo, int bar) + { + using (Activity activity = source.StartActivity("SomeWork")) + { + activity?.SetTag("foo", foo); + activity?.SetTag("bar", bar); + await StepOne(); + activity?.AddEvent(new ActivityEvent("Part way there")); + await StepTwo(); + activity?.AddEvent(new ActivityEvent("Done now")); + + // Pretend something went wrong + activity?.SetTag("otel.status_code", "ERROR"); + activity?.SetTag("otel.status_description", "Use this text give more information about the error"); + } + } +``` + +## Optional: Add Additional Activities + +Activities can be nested to describe portions of a larger unit of work. This can be particularly valuable around +portions of code that might not execute quickly or to better localize failures that come from specific external +dependencies. Although this sample uses an Activity in every method, that is solely because extra code has been +minimized. In a larger and more realistic project using an Activity in every method would produce extremely +verbose traces so it is not recommended. + +Update StepOne and StepTwo to add more tracing around these separate steps: + +```csharp + static async Task StepOne() + { + using (Activity activity = source.StartActivity("StepOne")) + { + await Task.Delay(500); + } + } + + static async Task StepTwo() + { + using (Activity activity = source.StartActivity("StepTwo")) + { + await Task.Delay(1000); + } + } +``` + +```dotnetcli +> dotnet run +Activity.Id: 00-9d5aa439e0df7e49b4abff8d2d5329a9-39cac574e8fda44b-01 +Activity.ParentId: 00-9d5aa439e0df7e49b4abff8d2d5329a9-f16529d0b7c49e44-01 +Activity.DisplayName: StepOne +Activity.Kind: Internal +Activity.StartTime: 2021-03-18T10:40:51.4278822Z +Activity.Duration: 00:00:00.5051364 +Resource associated with Activity: + service.name: MySample + service.instance.id: e0a8c12c-249d-4bdd-8180-8931b9b6e8d0 + +Activity.Id: 00-9d5aa439e0df7e49b4abff8d2d5329a9-4ccccb6efdc59546-01 +Activity.ParentId: 00-9d5aa439e0df7e49b4abff8d2d5329a9-f16529d0b7c49e44-01 +Activity.DisplayName: StepTwo +Activity.Kind: Internal +Activity.StartTime: 2021-03-18T10:40:51.9441095Z +Activity.Duration: 00:00:01.0052729 +Resource associated with Activity: + service.name: MySample + service.instance.id: e0a8c12c-249d-4bdd-8180-8931b9b6e8d0 + +Activity.Id: 00-9d5aa439e0df7e49b4abff8d2d5329a9-f16529d0b7c49e44-01 +Activity.DisplayName: SomeWork +Activity.Kind: Internal +Activity.StartTime: 2021-03-18T10:40:51.4256627Z +Activity.Duration: 00:00:01.5286408 +Activity.TagObjects: + foo: banana + bar: 8 + otel.status_code: ERROR + otel.status_description: Use this text give more information about the error +Activity.Events: + Part way there [3/18/2021 10:40:51 AM +00:00] + Done now [3/18/2021 10:40:52 AM +00:00] +Resource associated with Activity: + service.name: MySample + service.instance.id: e0a8c12c-249d-4bdd-8180-8931b9b6e8d0 + +Example work done +``` + +Notice that both StepOne and StepTwo include a ParentId that refers to SomeWork. The console is +not a great visualization of nested trees of work, but many GUI viewers such as +[Zipkin](https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry.Exporter.Zipkin/README.md) +can show this as a Gantt chart: + +[![Zipkin Gantt chart](media/zipkin-nested-activities.jpg)](media/zipkin-nested-activities.jpg) + +### Optional: ActivityKind + +Activities have an property which +describes the relationship between the Activity, its parent and its children. By default all new Activities are +set to which is appropriate for Activities that are an +internal operation within an application with no remote parent or children. Other kinds can be set using the +kind parameter on +. See + for other options. + +### Optional: Links + +When work occurs in batch processing systems a single Activity might represent work on behalf of many +different requests simultaneously, each of which has its own trace-id. Although Activity is restricted +to have a single parent, it can link to additional trace-ids using +. Each ActivityLink is +populated with an that +stores ID information about the Activity being linked to. ActivityContext can be retrieved from in-process +Activity objects using or +it can be parsed from serialized id information using +. + +```csharp +void DoBatchWork(ActivityContext[] requestContexts) +{ + // Assume each context in requestContexts encodes the trace-id that was sent with a request + using(Activity activity = s_source.StartActivity(name: "BigBatchOfWork", + kind: ActivityKind.Internal, + parentContext: null, + links: requestContexts.Select(ctx => new ActivityLink(ctx)) + { + // do the batch of work here + } +} +``` + +Unlike events and Tags that can be added on-demand, links must be added during StartActivity() and +are immutable afterwards. diff --git a/docs/core/diagnostics/distributed-tracing.md b/docs/core/diagnostics/distributed-tracing.md index 13590c6680d3a..bfc825a180af4 100644 --- a/docs/core/diagnostics/distributed-tracing.md +++ b/docs/core/diagnostics/distributed-tracing.md @@ -1,229 +1,48 @@ --- title: Distributed tracing - .NET description: An introduction to .NET distributed tracing. -ms.date: 02/02/2021 +ms.date: 03/15/2021 --- # .NET Distributed Tracing -Distributed tracing is the way to publish and observe tracing data in a distributed system. -.NET Framework and .NET Core has been supporting tracing through the APIs. - -- class which allows storing and accessing diagnostics context and consuming it with logging system. -- that allows code to be instrumented for production-time logging of rich data payloads for consumption within the process that was instrumented. - -Here is an example that shows how to publish tracing data from the HTTP incoming requests: - -```csharp - public void OnIncomingRequest(DiagnosticListener httpListener, HttpContext context) - { - if (httpListener.IsEnabled("Http_In")) - { - var activity = new Activity("Http_In"); - - //add tags, baggage, etc. - activity.SetParentId(context.Request.headers["Request-id"]) - foreach (var pair in context.Request.Headers["Correlation-Context"]) - { - var baggageItem = NameValueHeaderValue.Parse(pair); - activity.AddBaggage(baggageItem.Key, baggageItem.Value); - } - httpListener.StartActivity(activity, new { context }); - try - { - //process request ... - } - finally - { - //stop activity - httpListener.StopActivity(activity, new {context} ); - } - } - } -``` - -Here is example for how to listen to the Activity events: - -```csharp - DiagnosticListener.AllListeners.Subscribe(delegate (DiagnosticListener listener) - { - if (listener.Name == "MyActivitySource") - { - listener.Subscribe(delegate (KeyValuePair value) - { - if (value.Key.EndsWith("Start", StringComparison.Ordinal)) - LogActivityStart(); - else if (value.Key.EndsWith("Stop", StringComparison.Ordinal)) - LogActivityStop(); - }); - } - } -``` - -.NET 5.0 has extended the capability of the distributed tracing to allow the [OpenTelemetry](https://opentelemetry.io/) tracing scenarios, added Sampling capabilities, simplified the tracing coding pattern, and made listening to the Activity events easier and flexible. - -> [!NOTE] -> To access all added .NET 5.0 capabilities, ensure your project references the [System.Diagnostics.DiagnosticSource](https://www.nuget.org/packages/System.Diagnostics.DiagnosticSource/) NuGet package version 5.0 or later. This package can be used in libraries and apps targeting any supported version of the .NET Framework, .NET Core, and .NET Standard. If targeting .NET 5.0 or later, there is no need to manually reference the package as it is included in the shared library installed with the .NET Runtime. - -## Getting Started With Tracing - -Applications and libraries can easily publish tracing data by simply using the and the classes. - -### ActivitySource - -The first step to publish tracing data is to create an instance of the ActivitySource class. The ActivitySource is the class that provides APIs to create and start Activity objects and to register ActivityListener objects to listen to the Activity events. - -```csharp - private static ActivitySource source = new ActivitySource("MyCompany.MyComponent.SourceName", "v1"); -``` - -#### Best Practices - -- Create the ActivitySource once and store it in a static variable and use that instance as long as needed. - -- The source name passed to the constructor has to be unique to avoid the conflicts with any other sources. It is recommended to use a hierarchical name contains the company name, the component name, and the source name. For example, `Microsoft.System.HttpClient.HttpInOutRequests`. - -- The version parameter is optional. It is recommended to provide the version in case you plan to release multiple versions of the library or the application and want to distinguish between the sources of different versions. - -### Activity Creation - -Now the created ActivitySource object can be used to create and start Activity objects which are used to log any tracing data in any desired places in the code. - -```csharp - using (Activity activity = source.StartActivity("OperationName")) - { - // Do something - - activity?.AddTag("key", "value"); // log the tracing - } -``` - -This sample code tries to create the Activity object and then logs some tracing tag `key` and `value`. - -#### Notes - -- `ActivitySource.StartActivity` tries to create and start the activity at the same time. The listed code pattern is using the `using` block which automatically disposes the created Activity object after executing the block. Disposing the Activity object will stop this started activity and the code doesn't have to explicitly stop the activity. That simplifies the coding pattern. - -- `ActivitySource.StartActivity` internally figures out if there are any listeners to such events. If there are no registered listeners or there are listeners which are not interested in such an event, `ActivitySource.StartActivity` simply will return `null` and avoid creating the Activity object. That is why the code used the nullable operator `?` in the statement `activity?.AddTag`. In general, inside the `using` block, always use the nullable operator `?` after the activity object name. - -## Listening to the Activity Events - -.NET provides the class which can be used to listen to the Activity events triggered from one or more ActivitySources. -The listener can be used to collect tracing data, sample, or force creating the Activity object. - -The `ActivityListener` class provides a different callbacks to handle different events. - -```csharp -var listener = new ActivityListener -{ - ShouldListenTo = (activitySource) => object.ReferenceEquals(source, activitySource), - ActivityStarted = activity => /* Handle the Activity start event here */ DoSomething(), - ActivityStopped = activity => /* Handle the Activity stop event here */ DoSomething(), - SampleUsingParentId = (ref ActivityCreationOptions activityOptions) => ActivitySamplingResult.AllData, - Sample = (ref ActivityCreationOptions activityOptions) => ActivitySamplingResult.AllData -}; - -// Enable the listener -ActivitySource.AddActivityListener(listener); -``` - -- `ShouldListenTo` enables listening to specific `ActivitySource` objects. In the example, it enables listening to the `ActivitySource` object we have previously created. There is more flexibility to listen to any other `ActivitySource` objects by checking the `Name` and `Version` of the input `ActivitySource`. - -- `ActivityStarted` and `ActivityStopped` enable getting the `Activity` Start and Stop events for all `Activity` objects created by the `ActivitySource` objects which were enabled by the `ShouldListenTo` callback. - -- `Sample` and `SampleUsingParentId` are the main callbacks which intended for sampling. These two callbacks return the `ActivitySamplingResult` enum value which can tell either to sample in or out the current `Activity` creation request. If the callback returns `ActivitySamplingResult.None` and no other enabled listeners return different value, then the Activity will not get created and `ActivitySource.StartActivity` will return `null`. Otherwise, the `Activity` object will get created. - -## .NET 5.0 New Features - -For awhile the `Activity` class has been supporting tracing scenarios. It allowed adding tags which are key-value pairs of tracing data. It has been supporting Baggage which is are key-value pairs intended to be propagated across the wire. - -.NET 5.0 supports more features mainly to enable OpenTelemetry scenarios. - -### ActivityContext - - is the struct carrying the context of the tracing operations (e.g. the trace Id, Span Id, trace flags, and trace state). Now it is possible to create a new `Activity` providing the parent tracing context. Also, it is easy to extract the tracing context from any `Activity` object by calling `Activity.Context` property. - -### ActivityLink - - is the struct containing the tracing context instance which can be linked to casually related `Activity` objects. Links can be added to the `Activity` object by passing the links list to `ActivitySource.StartActivity` method when creating the `Activity`. The `Activity` object attached links can be retrieved using the property `Activity.Links`. - -### ActivityEvent - - represents an event containing a name and a timestamp, as well as an optional list of tags. Events can be added to the `Activity` object by calling `Activity.AddEvent` method. The whole list of the `Activity` object Events can be retrieved using the property `Activity.Events`. - -### ActivityKind - - describes the relationship between the activity, its parents and its children in a trace. Kind can be set to the `Activity` object by passing the kind value to `ActivitySource.StartActivity` method when creating the `Activity`. The `Activity` object kind can be retrieved using the property `Activity.Kind`. - -### IsAllDataRequested - - indicates whether this activity should be populated with all the propagation information, as well as all the other properties, such as links, tags, and events. The value of `IsAllDataRequested` is determined from the result returned from all listeners `Sample` and `SampleUsingParentId` callbacks. The value can be retrieved using `Activity.IsAllDataRequested` property. - -### Activity.Source - - gets the activity source associated with this activity. - -### Activity.DisplayName - - allows getting or setting a display name for the activity. - -### Activity.TagObjects - -`Activity` class has the property `Activity.Tags` which return the a key-value pair list of the tags of type string for the key and value. Such Tags can be added to the `Activity` using the method `AddTag(string, string)`. `Activity` has extended the support of tags by providing the overloaded method `AddTag(string, object)` allowing values of any type. The complete list of such tags can be retrieved using . - -## Sampling - -Sampling is a mechanism to control the noise and overhead by reducing the number of samples of traces collected and sent to the backend. Sampling is an important OpenTelemetry scenario. In .NET 5.0 it is easy to allow sampling. A good practice is to create the new `Activity` objects using `ActivitySource.StartActivity` and try to provide all possible available data (e.g. tags, kind, links, ...etc.) when calling this method. Providing the data will allow the samplers implemented using the `ActivityListener` to have a proper sampling decision. If the performance is critical to avoid creating the data before creating the `Activity` object, the property `ActivitySource.HasListeners` comes in handy to check if there are any listeners before creating the needed data. - -## OpenTelemetry - -OpenTelemetry SDK comes with many features that support end-to-end distributed tracing scenarios. It provides multiple samplers and exporters which you can choose from. It allows creating a custom samplers and exporters too. - -OpenTelemetry supports exporting the collected tracing data to different backends (e.g. Jaeger, Zipkin, New Relic,...etc.). Refer to [OpenTelemetry-dotnet](https://github.com/open-telemetry/opentelemetry-dotnet/) for more details and search Nuget.org for packages starting with `OpenTelemetry.Exporter.` to get the exporter packages list. - -Here is sample code ported from [OpenTelemetry-dotnet getting started](https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/docs/trace/getting-started) showing how easy it is to sample and export tracing data to the console. - -```csharp -// -// Copyright The OpenTelemetry Authors -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. -// - -using System.Diagnostics; -using OpenTelemetry; -using OpenTelemetry.Trace; - -public class Program -{ - private static readonly ActivitySource MyActivitySource = new ActivitySource( - "MyCompany.MyProduct.MyLibrary"); - - public static void Main() - { - using var tracerProvider = Sdk.CreateTracerProviderBuilder() - .SetSampler(new AlwaysOnSampler()) - .AddSource("MyCompany.MyProduct.MyLibrary") - .AddConsoleExporter() - .Build(); - - using (var activity = MyActivitySource.StartActivity("SayHello")) - { - activity?.SetTag("foo", 1); - activity?.SetTag("bar", "Hello, World!"); - activity?.SetTag("baz", new int[] { 1, 2, 3 }); - } - } -} -``` - -The sample needs to reference the package [OpenTelemetry.Exporter.Console](https://www.nuget.org/packages/OpenTelemetry.Exporter.Console/1.0.0-rc2). +Distributed tracing is a diagnostic technique that helps engineers localize failures and +performance issues within applications, especially those that may be distributed across +multiple machines or processes. This technique tracks requests through an application +correlating together work done by different application components and separating it from +other work the application may be doing for concurrent requests. For example a request to a +typical web service might be first received by a load balancer, then forwarded to a web server +process, which then makes several queries to a database. Using distributed tracing allows +engineers to distinguish if any of those steps failed, how long each step took, and potentially +logging messages produced by each step as it ran. + +## Getting started for .NET app developers + +Key .NET libraries are instrumented to produce distributed tracing information automatically +but this information needs to be collected and stored so that it will be available for review later. +Typically app developers will select a telemetry service that stores this trace information for them and +then use a corresponding library to transmit the distributed tracing telemetry to their chosen +service. [OpenTelemetry](https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/docs/trace/getting-started/README.md) +is a vendor neutral library that supports several services, +[Application Insights](https://docs.microsoft.com/azure/azure-monitor/app/distributed-tracing) +is a full featured service provided by Microsoft, and there are many high quality 3rd party APM vendors +that offer integrated .NET solutions. + +- [Understand distributed tracing concepts](distributed-tracing-concepts.md) +- Guides + - [Collect distributed traces with Application Insights](distributed-tracing-collection-walkthroughs.md#collect-traces-using-application-insights) + - [Collect distributed traces with OpenTelemetry](distributed-tracing-collection-walkthroughs.md#collect-traces-using-opentelemetry) + - [Collect distributed traces with custom logic](distributed-tracing-collection-walkthroughs.md#collect-traces-using-custom-logic) + - [Adding custom distributed trace instrumentation](distributed-tracing-instrumentation-walkthroughs.md) + +For 3rd party telemetry collection services follow the setup instructions provided by the vendor. + +## Getting started for .NET library developers + +.NET libraries don't need to be concerned with how telemetry is ultimately collected, only +with how it is produced. If you believe .NET app developers that use your library would +appreciate seeing the work that it does detailed in a distributed trace then you should add +distributed tracing instrumentation to support it. + +- [Understand distributed tracing concepts](distributed-tracing-concepts.md) +- Guides + - [Adding custom distributed trace instrumentation](distributed-tracing-instrumentation-walkthroughs.md) diff --git a/docs/core/diagnostics/logging-tracing.md b/docs/core/diagnostics/logging-tracing.md index bb4ed7494ef2b..c5fa1f5b8edc2 100644 --- a/docs/core/diagnostics/logging-tracing.md +++ b/docs/core/diagnostics/logging-tracing.md @@ -66,7 +66,11 @@ The following APIs are more event oriented. Rather than logging simple strings t ## Distributed Tracing -[Distributed Tracing](./distributed-tracing.md) is the way to publish and observe the tracing data in a distributed system. +[Distributed Tracing](./distributed-tracing.md) is a diagnostic technique that helps engineers +localize failures and performance issues within applications, especially those that may be +distributed across multiple machines or processes. This technique tracks requests through an +application correlating together work done by different application components and separating +it from other work the application may be doing for concurrent requests. ## ILogger and logging frameworks diff --git a/docs/core/diagnostics/media/zipkin-nested-activities.jpg b/docs/core/diagnostics/media/zipkin-nested-activities.jpg new file mode 100644 index 0000000000000..a51180242db44 Binary files /dev/null and b/docs/core/diagnostics/media/zipkin-nested-activities.jpg differ diff --git a/docs/fundamentals/toc.yml b/docs/fundamentals/toc.yml index e57ea5ae5709c..0e813e2b837fe 100644 --- a/docs/fundamentals/toc.yml +++ b/docs/fundamentals/toc.yml @@ -427,7 +427,15 @@ items: - name: Well-known event providers href: ../core/diagnostics/well-known-event-providers.md - name: Distributed Tracing - href: ../core/diagnostics/distributed-tracing.md + items: + - name: Overview + href: ../core/diagnostics/distributed-tracing.md + - name: Concepts + href: ../core/diagnostics/distributed-tracing-concepts.md + - name: Instrumentation + href: ../core/diagnostics/distributed-tracing-instrumentation-walkthroughs.md + - name: Collection + href: ../core/diagnostics/distributed-tracing-collection-walkthroughs.md - name: Symbols href: ../core/diagnostics/symbols.md - name: Runtime events