Add a separate example project for Logs redaction #3744

utpilla · 2022-10-07T19:41:43Z

It's a common ask from customers to provide examples for log redaction.

The current example for showing logs redaction is incorrect.
It's also convoluted as we trying to showcase a lot of things at once in extending-the-sdk project.

Changes

Create a dedicated example project for logs redaction

…-Logs

codecov · 2022-10-07T19:51:54Z

Codecov Report

Merging #3744 (6213601) into main (0feebf3) will increase coverage by 0.23%.
The diff coverage is n/a.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3744      +/-   ##
==========================================
+ Coverage   87.04%   87.28%   +0.23%     
==========================================
  Files         278      278              
  Lines       10167    10167              
==========================================
+ Hits         8850     8874      +24     
+ Misses       1317     1293      -24

Impacted Files	Coverage Δ
...Telemetry/Internal/SelfDiagnosticsEventListener.cs	`97.65% <0.00%> (+0.78%)`	⬆️
...ter.ZPages/Implementation/ZPagesActivityTracker.cs	`100.00% <0.00%> (+2.85%)`	⬆️
...metryProtocol/Implementation/ActivityExtensions.cs	`95.05% <0.00%> (+3.29%)`	⬆️
...ZPages/Implementation/ZPagesExporterEventSource.cs	`62.50% <0.00%> (+6.25%)`	⬆️
...xporter.OpenTelemetryProtocol/OtlpTraceExporter.cs	`77.27% <0.00%> (+40.90%)`	⬆️
...entation/ExportClient/OtlpGrpcTraceExportClient.cs	`78.57% <0.00%> (+42.85%)`	⬆️

cijothomas · 2022-10-07T20:00:58Z

src/OpenTelemetry.Exporter.Console/ConsoleLogRecordExporter.cs

@@ -84,7 +84,28 @@ public override ExportResult Export(in Batch<LogRecord> batch)

                if (logRecord.State != null)
                {
-                    this.WriteLine($"{"LogRecord.State:",-RightPaddingLength}{logRecord.State}");
+                    if (logRecord.State is IReadOnlyList<KeyValuePair<string, object>> listKvp)


we may want to do what OTLP exporter does.
#3186 (comment)

Since ConsoleExporter is only meant to be used for local testing, I was thinking it's probably a good idea to not impose any option from our end and let the user figure out what works best for them.

CodeBlanch

LGTM

docs/logs/redaction/README.md

reyang · 2022-10-10T18:18:28Z

docs/logs/redaction/MyClassWithRedactionEnumerator.cs

+            {
+                var item = this.state[index];
+                var entryVal = item.Value;
+                if (entryVal != null && entryVal.ToString() != null && entryVal.ToString().Contains("<secret>"))


Consider a real world example: input contains email address david@otel.org, output will change it to ***@otel.org.

This is probably not as straightforward to do without causing allocation.

cijothomas · 2022-10-10T18:24:07Z

docs/logs/redaction/Program.cs

+        using var loggerFactory = LoggerFactory.Create(builder =>
+            builder.AddOpenTelemetry(options =>
+            {
+                options.AddProcessor(new MyRedactionProcessor());


processors are run sync.. so it might affect latency if executing complex regex etc. But this might be okay for now. Unless we can show how to do this in Exporter (which wraps real exporter).

Co-authored-by: Reiley Yang <reyang@microsoft.com>

* [Logs] Fix buffered log scopes being reused (#3731) * Fix buffered log scopes being reused. * CHANGELOG update. * Test fixes. Co-authored-by: Cijo Thomas <cithomas@microsoft.com> * clarify that Prometheus HttpListner is not for production at this moment (#3737) * [HttpClient] Export spans corresponding to retries (#3732) * Minor update to OTLP readme (#3739) * Nit fixes to prometheus readme * Add minor clarification about OTLP logs * Update workflow (#3741) * Minor improvement to log message (#3742) * [SDK + Jaeger] Support loading environment variables from IConfiguration in Traces & Metrics (#3720) * Support retrieval of environment variables through IConfiguration in SDK. * Update Jaeger to load environment variables through IConfiguration. * Warning fix. * CHANGELOG patch. * Bug fixes. * Warning cleanup. * Code review. * [Zipkin] Support loading envvars from IConfiguration (#3759) * Support loading envvars from IConfiguration in Zipkin exporter. * CHANGELOG patch. * SqlClient Instrumentation to leverage native Activity Status. (#3751) * [Metrics] Update default buckets for Explicit Bucket Histogram from spec (#3722) * [Logs] Fix: Respect AttributeValueLengthLimit when building otlp LogRecord data (#3684) * Respect SdkConfiguration.AttributeValueLengthLimit so otlp data points are not rejected by monitoring tools * Respect maxAttributeCount and use OtlpKeyValueTransformer in AddStringAttribute and in AddIntAttribute * Extend CHANGELOG.md Co-authored-by: Mikel Blanchard <mblanchard@macrosssoftware.com> * Bump actions/setup-dotnet from 3.0.1 to 3.0.2 (#3764) Bumps [actions/setup-dotnet](https://github.com/actions/setup-dotnet) from 3.0.1 to 3.0.2. - [Release notes](https://github.com/actions/setup-dotnet/releases) - [Commits](actions/setup-dotnet@v3.0.1...v3.0.2) --- updated-dependencies: - dependency-name: actions/setup-dotnet dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [Repo] Attempting to stabilize the API Compatibility CI job (#3766) * Attempting to stablize the API Compatibility CI. * Tweak. * More logging. * Increase build logging level for api compat ci job. * Tweak. * Tweak. * Revert extra logging in ci job. * Switched some logging back to debug. * Adding MinMax to Histograms (#2735) * Logging state during building of TracerProvider (#3746) * [HttpClient] Add unit tests for `RecordException` case (#3761) * Add a separate example project for Logs redaction (#3744) * Update CHANGELOG for 1.4.0-beta.2 release (#3772) * [SDK + Otlp] Support loading envvars from IConfiguration (#3760) * Updated Otlp Trace & Metrics exporters to load envvars from IConfiguration. * Patch CHANGELOGs. * Fix up otlp log exporter for SdkOptions changes. * Revert SdkOptions public api. * Restore tests. * Fix benchmarks. * SdkLimitOptions IConfiguration test. * OtlpExporterOptions IConfiguration test. * MetricReaderOptions IConfiguration test. * Bug fix. * Nit. * [SqlClient] Add support for Filter expression (#3743) * [SDK, Jaeger, Zipkin, & Otlp] Support loading envvars for BatchExportActivityProcessorOptions from IConfiguration (#3776) * Support loading envvars for ExportActivityProcessorOptions & BatchExportActivityProcessorOptions from IConfiguration. * Update src/OpenTelemetry/CHANGELOG.md Co-authored-by: Piotr Kiełkowicz <pkiekowicz@splunk.com> * Patch CHANGELOG. * Unit test. Co-authored-by: Cijo Thomas <cithomas@microsoft.com> Co-authored-by: Piotr Kiełkowicz <pkiekowicz@splunk.com> * Remove Env from CI `DOTNET_MULTILEVEL_LOOKUP = 1` (#3779) * Link to .NET docs about different ports (#3704) * Update DS to rc2 (#3781) * ConsoleLogExporter to output full exception (#3784) * [ASP.NET Core] Add back netstandard2.0 and 2.1 targets (#3755) * [HttpClient] Add back netstandard2.0 target (#3787) * Add back netstandard2.0 target * changelog * public api files * Bump System.Text.Json version due to CVE-2021-26701 (#3789) * Auto-generated Semantic Conventions (#2069) * [SDK] Support dependency injection in ResourceBuilder and load envvars from IConfiguration (#3782) * Add Vishwesh as an Approver (#3783) Co-authored-by: Cijo Thomas <cijo.thomas@gmail.com> * Move more SDK docs to docs folder (#3794) * [Prometheus AspNetCore] Support named options in pipeline extensions (#3780) * Support named options in Prometheus AspNetCore pipeline extensions. * Patch CHANGELOG. Co-authored-by: Cijo Thomas <cijo.thomas@gmail.com> * [Logs] UnitTest: LogRecord attribute limits (#3758) * Unittest for LogRecord attribute limits * Remove maxValueLength from LogRecordExtensions.AddIntAttribute. Change requested from #3684 (comment) * Pr commits addressed Co-authored-by: Mikel Blanchard <mblanchard@macrosssoftware.com> * Add MinMax to console and doc additions (#3795) * Mark private and internal classes as sealed (#3799) Co-authored-by: Cijo Thomas <cijo.thomas@gmail.com> * Move more docs to docs folder (#3801) * [ASP.NET Core] Update enrich callbacks to use specific type. (#3749) * [Http] Update enrich callbacks for http (#3792) * [SDK] Support dependency injection in the GetDefaultResource API (#3798) * Support dependency injection in the GetDefaultResource API. * CHANGELOG patch. * Test tweaks. Co-authored-by: Cijo Thomas <cijo.thomas@gmail.com> * Fix circular reference issue building up tracer provider. (#3803) * [SDK] Add some missing nullable annotations (#3796) * Added some missing nullable annotations in SDK. * Handle null span names in SamplingParameters. * Warning cleanup. * Warning cleanup. * Added link to issue in comment. Co-authored-by: Alan West <3676547+alanwest@users.noreply.github.com> * Guard.Range now uses invariant culture for error message (#3778) Co-authored-by: Utkarsh Umesan Pillai <utpilla@microsoft.com> * Fix circular reference issue building up meter provider. (#3806) Co-authored-by: Utkarsh Umesan Pillai <utpilla@microsoft.com> * Add missing end of code block backticks (#3807) * More doc tweaks (#3805) * More doc tweaks * remove draft staatus * Update grpc client enrich callbacks (#3804) * Port refactor from main-logs branch. (#3808) Co-authored-by: Cijo Thomas <cijo.thomas@gmail.com> * Merge fixes. * Merge fixes. * Merge fix. Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Cijo Thomas <cithomas@microsoft.com> Co-authored-by: Reiley Yang <reyang@microsoft.com> Co-authored-by: Vishwesh Bankwar <vishweshbankwar@users.noreply.github.com> Co-authored-by: Cijo Thomas <cijo.thomas@gmail.com> Co-authored-by: Yun-Ting Lin <yunl@microsoft.com> Co-authored-by: Sebastian Schoder Moreno <35150382+schoder-moreno@users.noreply.github.com> Co-authored-by: Jonathan Wilhelm <Jonathan.wilhelm@sage.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Michael Maxwell <micmax@microsoft.com> Co-authored-by: ggoel <gaurav.goel111@gmail.com> Co-authored-by: Utkarsh Umesan Pillai <utpilla@microsoft.com> Co-authored-by: Pavel Steinl <pavel.steinl@gmail.com> Co-authored-by: Piotr Kiełkowicz <pkiekowicz@splunk.com> Co-authored-by: aristotelos <arisvd@gmail.com> Co-authored-by: Joao Grassi <joao@joaograssi.com> Co-authored-by: Alan West <3676547+alanwest@users.noreply.github.com> Co-authored-by: benhall_io <3179852+benbhall@users.noreply.github.com>

Add redaction project

7ae01f0

utpilla requested a review from a team October 7, 2022 19:41

utpilla added 2 commits October 7, 2022 12:42

Merge remote-tracking branch 'origin/main' into utpilla/Redaction-For…

f2fc2c4

…-Logs

Fix markdown

08069ae

cijothomas reviewed Oct 7, 2022

View reviewed changes

CodeBlanch approved these changes Oct 10, 2022

View reviewed changes

Merge branch 'main' into utpilla/Redaction-For-Logs

c48919f

reyang reviewed Oct 10, 2022

View reviewed changes

docs/logs/redaction/README.md Outdated Show resolved Hide resolved

reyang approved these changes Oct 10, 2022

View reviewed changes

reyang reviewed Oct 10, 2022

View reviewed changes

cijothomas reviewed Oct 10, 2022

View reviewed changes

utpilla and others added 3 commits October 10, 2022 11:45

Update docs/logs/redaction/README.md

97d01aa

Co-authored-by: Reiley Yang <reyang@microsoft.com>

Merge branch 'main' into utpilla/Redaction-For-Logs

62770f4

Merge branch 'main' into utpilla/Redaction-For-Logs

6213601

utpilla merged commit 16d8e22 into open-telemetry:main Oct 17, 2022

utpilla deleted the utpilla/Redaction-For-Logs branch November 3, 2023 22:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a separate example project for Logs redaction #3744

Add a separate example project for Logs redaction #3744

utpilla commented Oct 7, 2022 •

edited

Loading

codecov bot commented Oct 7, 2022 •

edited

Loading

cijothomas Oct 7, 2022

utpilla Oct 12, 2022

CodeBlanch left a comment

reyang Oct 10, 2022

utpilla Oct 13, 2022

cijothomas Oct 10, 2022

Add a separate example project for Logs redaction #3744

Add a separate example project for Logs redaction #3744

Conversation

utpilla commented Oct 7, 2022 • edited Loading

Changes

codecov bot commented Oct 7, 2022 • edited Loading

Codecov Report

cijothomas Oct 7, 2022

Choose a reason for hiding this comment

utpilla Oct 12, 2022

Choose a reason for hiding this comment

CodeBlanch left a comment

Choose a reason for hiding this comment

reyang Oct 10, 2022

Choose a reason for hiding this comment

utpilla Oct 13, 2022

Choose a reason for hiding this comment

cijothomas Oct 10, 2022

Choose a reason for hiding this comment

utpilla commented Oct 7, 2022 •

edited

Loading

codecov bot commented Oct 7, 2022 •

edited

Loading