GA requirements and prioritization

[trask]

Trying to draft GA requirements and prioritization. This is just a starting place. Looking for feedback here. Will also discuss in this Thu meeting. Thanks!

P0 (e.g. cannot GA without these):

• End-to-end tests
  • OTLP, Jaeger and Zipkin
  • Tomcat, Wildfly, Spring Boot
  • All Java LTS versions + latest Java version if non-LTS
• Benchmarking
  • Startup overhead benchmark
  • Runtime overhead benchmark
• Implement all required semantic attributes
  • Non-required semantic attributes are nice to have, but not required for GA
• Basic configuration points
  • Add custom auto-instrumentation
  • Ability to build "custom distro"
• Documentation (tag "documentation" minus tag "contributor experience")
  • All configuration options
    • Standard OpenTelemetry SDK + Exporter configuration options
    • Auto-instrumentation configuration options (e.g. disable/enable, peer.service mapping)
  • For each instrumentation
    • Document all spans that it captures
      • Span names
      • Span attributes (including explanation of any non-semantic attributes)
      • Events
    • Document any instrumentation-specific configuration
    • Document any other effects (e.g. updating SERVER span name with route)
  • Document the basic configuration points (above)
    • How to write your own auto-instrumentation
      • (much of this can be shared with contributor documentation below)
    • How to build your own "custom distro"
  • How to troubleshoot (start documenting common issues somewhere)
• Library (manual) instrumentations corresponding to at least 20 auto-instrumentations, with shared code and tests

P1:

• Contributor experience (tag "contributor experience" plus tag "cleanup" plus tag "sporadic test failure")
  • New contributor documentation
    • How to write new instrumentation (auto, library, tests)
    • How to understand and fix muzzle issues
    • How to submit your first PR (CLA, check for CI failures, note about sporadic failures)
  • Faster builds
  • Fewer sporadic CI failures
  • Publish a debug jar without the classdata obfuscation

P3:

• Auto-collected metrics
  • System / JVM metrics (Standard metric names and semantic conventions opentelemetry-specification#651)
  • Request metrics (Define the metrics that we expect to capture from RPC integrations opentelemetry-specification#522, Add metrics semantic conventions for timed operations opentelemetry-specification#657)

P4:

• Library (manual) instrumentation corresponding to all auto-instrumentation

P5:

• Non-required semantic attributes

[anuraaga]

Tomcat, Wildfly, Spring Boot

This list seems sort of arbitrary. Is it P0 to test 3 servlet implementations? My suggestion based on back-of-hand estimation of popularity nowadays is one servlet, spring-boot, and grpc, while others could be P1 or P2

Startup overhead benchmark

This feels lower than P0, maybe P2.

Document all spans that it captures

I don't think I've ever seen such detailed documentation per-instrumentation in other tracing libraries. More docs is always better than less, but I don't think it's P0, maybe P2, and it needs to come with the commitment that we will keep the docs up to date.

Library (manual) instrumentations corresponding to at least 20 auto-instrumentations

I consider Brave to have very high instrumentation coverage, and it's a total of 25 (including some that are multi-version)
https://github.com/openzipkin/brave/tree/master/instrumentation

20 feels very high for a P0, my suggestion would be making this more use-case oriented.

• Libraries commonly used on Android
• Libraries commonly used with spring-boot

for example.

[trask]

Hey, sorry, I realized opening this as an issue was short-sighted due to lack of threaded conversation and how to make updates. Do you mind re-adding your comments on this PR? #734