Update distributed tracing docs

[noahfalk]

We are about to announce support for OpenTelemetry .NET and there is
likely to be an influx of developers reading these docs. They are
already the #1 link for "distributed tracing" on bing.com so they are
probably getting above average traffic/scrutiny too.

• Reorganized the content into a landing page, conceptual docs,
  instrumentation walkthroughs and collection walkthroughs
• Added some more intro and conceptual information assuming that many
  developers are completely new to distributed tracing and have no
  frame of reference.
• Removed the example of doing distributed tracing with DiagnosticSource
  We might want to restore this in the future but hopefully it represents
  a niche usage scenario that we are migrating away from. Without context
  indicating when to use it it is likely to confuse new developers about
  recommended patterns.
• Updated the examples to ensure they start with copy-pastable code
  that is fully runnable.
• Added links for collecting distributed tracing instrumentation with
  Application Insights

There are many oportunities for further improvement that we may
want to triage:

• Add examples and pictures of more realistic collected distributed
  trace information to help developers internalize why it might be useful
  to them.
• Add a diagnostic guide showing how distributed tracing assists in
  resolving a realistic production issue.
• Ensure the getting started guides for collection transfer smoothly to
  remote pages in the AppInsights and OpenTelemetry docs
  • There was a spot I wanted to link to supported OpenTelemetry
    exporters but there is no anchor that goes solely to that
    information.
• Flesh out the instrumentation guidance to answer more questions
  developers may encounter such as:
  • naming conventions for Activity
  • when to use DisplayName
  • how to log exceptions
  • how to use links for batch processing systems
  • how to propagate IDs to and from network protocols
  • how to migrate pre-existing Activity instrumentation to use
    ActivitySource
• Add performance analysis and guidance to reassure developers that
  Activity instrumentation + OpenTelemetry are serious about high
  performance workloads.
• Discuss more conceptual topics and/or go into greater depth about
  some areas:
  • Hierarchical vs. W3C ID.
  • Format of a W3C ID
  • Describe sampling options in ActivityListener
  • Explain why some activities are uncapturable with ActivityListener
    and require workarounds with DiagnosticListener.
• Roadmap for handling issues around custom propagators, supporting
  protocol updates, and suppressing undesired automatic Activity creation

[noahfalk]

Please take a look @sywhang @shirhatti @tarekgh @cijothomas @reyang

[noahfalk]

Is there a way to see a preview of what this change would look like on docs.microsoft.com without having merged it?

[tarekgh]

@carlossanlop do you know the answer of @noahfalk question Is there a way to see a preview of what this change would look like on docs.microsoft.com without having merged it??

[shirhatti]

Is there a way to see a preview of what this change would look like on docs.microsoft.com without having merged it?

https://review.docs.microsoft.com/en-us/dotnet/core/diagnostics/distributed-tracing?branch=pr-en-us-23313

[sywhang]

@noahfalk Can you also add the new pages you added as links in docs/core/diagnostics/logging-tracing.md, and docs/fundamentals/toc.yml?

[cijothomas]

Will be good if track deleting the following files after merging this change
https://github.com/dotnet/runtime/blob/main/src/libraries/System.Diagnostics.DiagnosticSource/src/ActivityUserGuide.md
https://github.com/dotnet/runtime/blob/main/src/libraries/System.Diagnostics.DiagnosticSource/src/DiagnosticSourceUsersGuide.md

I'll still keep them both and add ta section at the beginning to point to new guidance.
I find the DiagnosticSourceUserGuide to be relevant (to publish strong objects as payloads) even without Activity.

[carlossanlop]

@carlossanlop Carlos Sanchez Lopez FTE do you know the answer of @noahfalk Noah Falk FTE question Is there a way to see a preview of what this change would look like on docs.microsoft.com without having merged it??

@tarekgh Seems to be similar to the one in dotnet-api-docs:

In the CI section, click on "Show all checks" to expand the CI legs. There's one called OpenPublishing.Build. On the right side, click on "Details", which will open a page with some details about the build.

The "Validated files" table (first one) is the one you care about: click on Preview for each link. Or, click on just one, and look at the preview of your new pages, and navigate to the other pages from that one (the links should work).

By the way, the OpenPublishing.Build leg has a warning, which you need to fix: there is a cross reference to a DocId that is not correct (@sywhang). The full list of warnings can be consulted in the details page, but you only care about those coming from files you added in this PR. All others can be ignored (The Docs team should take care of those).

Hope that helped!

[noahfalk]

@noahfalk Can you also add the new pages you added as links in docs/core/diagnostics/logging-tracing.md, and docs/fundamentals/toc.yml?

@sywhang I've added entries to the toc and updated the text in logging-tracing.md but stopped short of adding all the links to logging-tracing.md. It wasn't clear that there was much value in duplicating all the links from distributed-tracing.md into logging-tracing.md if that means we now have to keep both pages in sync and they are replicas of each other. I am guessing that anyone interested in distributed tracing will click through from logging-tracing.md -> distributed-tracing.md -> the specific page they wanted. If you see any sign that people aren't finding the info they want because links aren't there I'm glad to change it.

[noahfalk]

Will be good if track deleting the following files after merging this change
https://github.com/dotnet/runtime/blob/main/src/libraries/System.Diagnostics.DiagnosticSource/src/ActivityUserGuide.md
https://github.com/dotnet/runtime/blob/main/src/libraries/System.Diagnostics.DiagnosticSource/src/DiagnosticSourceUsersGuide.md

I'll still keep them both and add ta section at the beginning to point to new guidance.
I find the DiagnosticSourceUserGuide to be relevant (to publish strong objects as payloads) even without Activity.

Agreed. DiagnosticSource is mostly not about distributed tracing so I only updated the Activity Guide:
dotnet/runtime#49818

[noahfalk]

Thanks for all the great feedback all! I've addressed almost everything but there are a few that it is unclear how to best address. Let me know if you are happy with the current state of the PR or you think we need to make more changes before checking in. If I don't hear anything today I'll assume it is OK to merge and we can address lingering issues with specific follow up PRs. Thanks!

[sywhang]

LGTM - if there is no other suggestions by EOD I'll go ahead and merge this. Thanks everyone!

[cijothomas]

Excellent write-up!
Left few small comments.
Only strong recommendation is to avoid using the "logging" term in tracing doc.