Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: blocking detection #2709

Merged
merged 49 commits into from
Mar 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
5a29e10
feat: blocking detection
bruno-garcia Oct 9, 2023
17bd0bf
half assed stuff from the weekend
bruno-garcia Oct 11, 2023
a25993c
Merge branch 'main' into feat/blocking-detector
bruno-garcia Dec 18, 2023
7173711
verify
bruno-garcia Dec 18, 2023
ef951da
Merge branch 'main' into feat/blocking-detector
bruno-garcia Dec 22, 2023
e1173fa
ref
bruno-garcia Dec 22, 2023
579fc98
Merge remote-tracking branch 'origin' into feat/blocking-detector
bruno-garcia Dec 22, 2023
43515be
binding
bruno-garcia Dec 22, 2023
d83362b
context
bruno-garcia Dec 22, 2023
047c0dc
wip
bruno-garcia Dec 22, 2023
fc1d708
wip
bruno-garcia Jan 5, 2024
869168f
Merge branch 'main' into feat/blocking-detector
jamescrosswell Feb 7, 2024
9859b21
Fixed compiler errors
jamescrosswell Feb 7, 2024
3fa8364
Update CHANGELOG.md
jamescrosswell Feb 7, 2024
3cb5bb6
Format code
getsentry-bot Feb 7, 2024
50f5b6d
Merge branch 'feat/blocking-detector' of github.com:getsentry/sentry-…
jamescrosswell Feb 7, 2024
85509fd
Reintroducing Bruno's changes with tests
jamescrosswell Feb 13, 2024
c75eb29
Reintroduced changes to MainSentryEventProcessor
jamescrosswell Feb 13, 2024
0c15b10
Reintroduced the core blocking capability
jamescrosswell Feb 13, 2024
0c855f4
Update SentryStackTraceFactory.cs
jamescrosswell Feb 13, 2024
663658f
Merge branch 'main' into feat/blocking-detector
jamescrosswell Feb 13, 2024
dcd5680
Update HubTests.CaptureEvent_ActiveTransaction_UnhandledExceptionTran…
jamescrosswell Feb 14, 2024
669d3b9
Added blocking detection suppression
jamescrosswell Feb 14, 2024
39fb85c
Merge branch 'feat/blocking-detector' of github.com:getsentry/sentry-…
jamescrosswell Feb 14, 2024
3d74dde
Format code
getsentry-bot Feb 14, 2024
c549b92
Merge branch 'main' into feat/blocking-detector
jamescrosswell Feb 15, 2024
679e099
Reverted unintentional changes to ApiDefinitions
jamescrosswell Feb 15, 2024
48d186b
Merge branch 'main' into feat/blocking-detector
jamescrosswell Feb 15, 2024
a00a4a5
Merge branch 'feat/blocking-detector' of github.com:getsentry/sentry-…
jamescrosswell Feb 15, 2024
e2ac986
Fixed stack traces for blocking calls
jamescrosswell Feb 15, 2024
578532e
Merge branch 'main' into feat/blocking-detector
jamescrosswell Feb 26, 2024
c593f82
Update BlockingMonitor.cs
jamescrosswell Feb 26, 2024
4d5b8e1
Some basic tests for BlockingMonitor
jamescrosswell Feb 26, 2024
fb5602f
Added code to ensure blocking events only get sent once per day
jamescrosswell Feb 26, 2024
b315b1a
Format code
getsentry-bot Feb 26, 2024
39926f4
Added System.Diagnostics.Metrics for blocking calls so users can know…
jamescrosswell Feb 26, 2024
81e1951
removed throttling mechanism
jamescrosswell Feb 26, 2024
4e732f4
Update CHANGELOG.md
jamescrosswell Feb 27, 2024
2e8b05a
Merge branch 'main' into feat/blocking-detector
bruno-garcia Feb 27, 2024
df2a8d6
merge issues
bruno-garcia Feb 27, 2024
42ed4a8
Refactored to improve testability
jamescrosswell Feb 28, 2024
c87472e
Merge branch 'feat/blocking-detector' of github.com:getsentry/sentry-…
jamescrosswell Feb 28, 2024
1cf1d18
Format code
getsentry-bot Feb 28, 2024
45c67b9
Moved IBlockingMonitor to a separate file
jamescrosswell Feb 28, 2024
f8d872a
Merge branch 'feat/blocking-detector' of github.com:getsentry/sentry-…
jamescrosswell Feb 28, 2024
9846d5a
Merge branch 'main' into feat/blocking-detector
jamescrosswell Feb 28, 2024
322afff
Merge branch 'main' into feat/blocking-detector
jamescrosswell Feb 28, 2024
80be4dd
Merge branch 'main' into feat/blocking-detector
jamescrosswell Mar 6, 2024
ffef982
Applied review feedback
jamescrosswell Mar 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@

### Features

- ASP.NET Core: Blocking call detection. An event with the stack trace of the blocking call will be captured as event. ([#2709](https://github.com/getsentry/sentry-dotnet/pull/2709))
- IMPORTANT: Verify this in test/staging before prod! Blocking calls in hot paths could create a lot of events for your Sentry project.
- Opt-in via `options.CaptureBlockingCalls = true`
- Disabled for specific code blocks with `using (new SuppressBlockingDetection())`
- Doesn't detect everything. See original [Caveats described by Ben Adams](https://github.com/benaadams/Ben.BlockingDetector?tab=readme-ov-file#caveats).
- Added Crons support via `SentrySdk.CaptureCheckIn` and an integration with Hangfire ([#3128](https://github.com/getsentry/sentry-dotnet/pull/3128))

### Fixes
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
using System.Data;
using System.Diagnostics;
using Microsoft.AspNetCore.Mvc;
using Sentry.Ben.BlockingDetector;
using Sentry.Samples.AspNetCore.Mvc.Models;

namespace Samples.AspNetCore.Mvc.Controllers;
Expand All @@ -12,6 +13,53 @@ public IActionResult Index()
return View();
}

// GET /home/block/true or /home/block/false to observe events
[HttpGet("[controller]/block/{block?}")]
public async Task<string> Block([FromRoute] bool block)
{
if (block)
{
logger.LogInformation("\ud83d\ude31 Calling a blocking API on an async method \ud83d\ude31");

// This will result in an event in Sentry
Task.Delay(10).Wait(); // This is a blocking call. Same with '.Result'
}
else
{
logger.LogInformation("\ud83d\ude31 No blocking call made \ud83d\ude31");

// Non-blocking, no event captured
await Task.Delay(10);
}

return "Was blocking? " + block;
}

// GET /home/suppress/true or /home/suppress/false to observe events
[HttpGet("[controller]/suppress/{suppress?}")]
public async Task<string> Suppress([FromRoute] bool suppress)
{
if (suppress)
{
logger.LogInformation("Blocking suppression enabled");
using (new SuppressBlockingDetection())
{
Task.Delay(10).Wait(); // This is blocking but won't trigger an event, due to suppression
}
logger.LogInformation("Blocking suppression disabled");
}
else
{
logger.LogInformation("\ud83d\ude31 Unsuppressed blocking call on an async method \ud83d\ude31");
Task.Delay(10).Wait(); // This is blocking but won't trigger an event, due to suppression
}

// Non-blocking, no event captured
await Task.Delay(10);

return "Was suppressed? " + suppress;
}

// Example: An exception that goes unhandled by the app will be captured by Sentry:
[HttpPost]
public async Task PostIndex(string? @params)
Expand Down
3 changes: 3 additions & 0 deletions samples/Sentry.Samples.AspNetCore.Mvc/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@
// Example: Disabling support to compressed responses:
options.DecompressionMethods = DecompressionMethods.None;

// Call GET /home/block/true to see this in action
options.CaptureBlockingCalls = true;

options.MaxQueueItems = 100;
options.ShutdownTimeout = TimeSpan.FromSeconds(5);

Expand Down
12 changes: 12 additions & 0 deletions samples/Sentry.Samples.AspNetCore.Mvc/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
Sample usages of the Sentry SDK for ASP.NET Core on an MVC app

Start by changing the DSN in `appsettings.json`, `Sentry:Dsn` property with your own.
No DSN yet? Get one for free at https://sentry.io/ to give this sample a run.

Blocking detection:
* It's turned on via option `CaptureBlockingCalls`
* In the `HomeController` there's an action that causes a blocking call on an async method.
* You can trigger it with:
* `GET http://localhost:5001/home/block/true`

Results `Was blocking? True` and an event captured in Sentry.
Original file line number Diff line number Diff line change
Expand Up @@ -11,55 +11,4 @@
<ProjectReference Include="..\..\src\Sentry.AspNetCore\Sentry.AspNetCore.csproj" />
</ItemGroup>

<ItemGroup>
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-grid.css" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-grid.css.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-grid.min.css" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-grid.min.css.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-grid.rtl.css" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-grid.rtl.css.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-grid.rtl.min.css" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-grid.rtl.min.css.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-reboot.css" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-reboot.css.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-reboot.min.css" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-reboot.min.css.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-reboot.rtl.css" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-reboot.rtl.css.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-reboot.rtl.min.css" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-reboot.rtl.min.css.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-utilities.css" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-utilities.css.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-utilities.min.css" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-utilities.min.css.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-utilities.rtl.css" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-utilities.rtl.css.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-utilities.rtl.min.css" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap-utilities.rtl.min.css.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap.css" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap.css.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap.min.css" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap.min.css.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap.rtl.css" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap.rtl.css.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap.rtl.min.css" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\css\bootstrap.rtl.min.css.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\js\bootstrap.bundle.js" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\js\bootstrap.bundle.js.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\js\bootstrap.bundle.min.js" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\js\bootstrap.bundle.min.js.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\js\bootstrap.esm.js" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\js\bootstrap.esm.js.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\js\bootstrap.esm.min.js" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\js\bootstrap.esm.min.js.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\js\bootstrap.js" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\js\bootstrap.js.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\js\bootstrap.min.js" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\dist\js\bootstrap.min.js.map" />
<_ContentIncludedByDefault Remove="wwwroot\lib\bootstrap\LICENSE" />
<_ContentIncludedByDefault Remove="wwwroot\lib\jquery-validation-unobtrusive\jquery.validate.unobtrusive.js" />
<_ContentIncludedByDefault Remove="wwwroot\lib\jquery-validation-unobtrusive\jquery.validate.unobtrusive.min.js" />
<_ContentIncludedByDefault Remove="wwwroot\lib\jquery-validation-unobtrusive\LICENSE.txt" />
</ItemGroup>

</Project>
2 changes: 2 additions & 0 deletions src/Sentry.AspNetCore/BindableSentryAspNetCoreOptions.cs
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ internal class BindableSentryAspNetCoreOptions : BindableSentryLoggingOptions
public bool? FlushBeforeRequestCompleted { get; set; }
public bool? AdjustStandardEnvironmentNameCasing { get; set; }
public bool? AutoRegisterTracing { get; set; }
public bool? CaptureBlockingCalls { get; set; }

public void ApplyTo(SentryAspNetCoreOptions options)
{
Expand All @@ -26,5 +27,6 @@ public void ApplyTo(SentryAspNetCoreOptions options)
options.FlushBeforeRequestCompleted = FlushBeforeRequestCompleted ?? options.FlushBeforeRequestCompleted;
options.AdjustStandardEnvironmentNameCasing = AdjustStandardEnvironmentNameCasing ?? options.AdjustStandardEnvironmentNameCasing;
options.AutoRegisterTracing = AutoRegisterTracing ?? options.AutoRegisterTracing;
options.CaptureBlockingCalls = CaptureBlockingCalls ?? options.CaptureBlockingCalls;
}
}
12 changes: 12 additions & 0 deletions src/Sentry.AspNetCore/SentryAspNetCoreOptions.cs
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,18 @@ public class SentryAspNetCoreOptions : SentryLoggingOptions
/// </summary>
internal bool FlushBeforeRequestCompleted { get; set; }

/// <summary>
/// Will capture an event if a blocking call is detected.
/// </summary>
/// <remarks>
/// Get a Sentry event for a sync over async call such as (await SomeTask).Result
/// Note that when ConfigureAwait(false) is used, blocking won't be detected.
/// Additionally, detects blocking calls over CLR waits such as lock, ReaderWriterLock, etc.
/// Blocking calls through system calls are also not captured.
/// </remarks>
/// <seealso href="https://github.com/getsentry/Ben.BlockingDetector/"/>
public bool CaptureBlockingCalls { get; set; }

/// <summary>
/// The strategy to define the name of a transaction based on the <see cref="HttpContext"/>.
/// </summary>
Expand Down
32 changes: 31 additions & 1 deletion src/Sentry.AspNetCore/SentryMiddleware.cs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using Sentry.AspNetCore.Extensions;
using Sentry.Ben.BlockingDetector;
using Sentry.Extensibility;
using Sentry.Internal;
using Sentry.Reflection;
Expand Down Expand Up @@ -32,6 +33,11 @@ internal static readonly SdkVersion NameAndVersion

private static readonly string ProtocolPackageName = "nuget:" + NameAndVersion.Name;

// Ben.BlockingDetector
private readonly BlockingMonitor? _monitor;
private readonly DetectBlockingSynchronizationContext? _detectBlockingSyncCtx;
private readonly TaskBlockingListener? _listener;

/// <summary>
/// Initializes a new instance of the <see cref="SentryMiddleware"/> class.
/// </summary>
Expand Down Expand Up @@ -63,6 +69,13 @@ public SentryMiddleware(
_eventExceptionProcessors = eventExceptionProcessors;
_eventProcessors = eventProcessors;
_transactionProcessors = transactionProcessors;

if (_options.CaptureBlockingCalls)
{
_monitor = new BlockingMonitor(_getHub, _options);
_detectBlockingSyncCtx = new DetectBlockingSynchronizationContext(_monitor);
_listener = new TaskBlockingListener(_monitor);
}
}

/// <summary>
Expand Down Expand Up @@ -132,7 +145,24 @@ public async Task InvokeAsync(HttpContext context, RequestDelegate next)
try
{
var originalMethod = context.Request.Method;
await next(context).ConfigureAwait(false);
if (_options.CaptureBlockingCalls && _monitor is not null)
{
var syncCtx = SynchronizationContext.Current;
SynchronizationContext.SetSynchronizationContext(syncCtx == null ? _detectBlockingSyncCtx : new DetectBlockingSynchronizationContext(_monitor, syncCtx));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bruno-garcia won't the SynchronizationContext in ASP.NET Core always be null? Is this code here just in case SDK users have implemented a custom sync context for some reason?

try
{
// For detection to work we need ConfigureAwait=true
await next(context).ConfigureAwait(true);
}
finally
{
SynchronizationContext.SetSynchronizationContext(syncCtx);
}
}
else
{
await next(context).ConfigureAwait(false);
}
if (_options.Instrumenter == Instrumenter.OpenTelemetry && Activity.Current is { } activity)
{
// The middleware pipeline finishes up before the Otel Activity.OnEnd callback is invoked so we need
Expand Down
107 changes: 107 additions & 0 deletions src/Sentry/Ben.BlockingDetector/BlockingMonitor.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
using Sentry.Internal;
using Sentry.Protocol;

// Namespace starting with Sentry makes sure the SDK cuts frames off before reporting
namespace Sentry.Ben.BlockingDetector
{
internal class BlockingMonitor : IBlockingMonitor
{
private readonly Func<IHub> _getHub;
private readonly SentryOptions _options;
internal readonly IRecursionTracker _recursionTracker;

public BlockingMonitor(Func<IHub> getHub, SentryOptions options)
: this(getHub, options, new StaticRecursionTracker())
{
}

internal BlockingMonitor(Func<IHub> getHub, SentryOptions options, IRecursionTracker recursionTracker)
{
_getHub = getHub;
_options = options;
_recursionTracker = recursionTracker;
}

private static bool ShouldSkipFrame(string? frameInfo) =>
frameInfo?.StartsWith("Sentry.Ben") == true
// Skip frames relating to the TaskBlockingListener
|| frameInfo?.StartsWith("System.Diagnostics") == true
// Skip frames relating to the async state machine
|| frameInfo?.StartsWith("System.Threading") == true;
Comment on lines +29 to +30
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can skip doing this and improve grouping on the server side for this kind of stuff in general.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bitsandfoxes could you link to the associated PR where this logic gets changed on the server?

Also, if we do skip this here, we'll need to make some changes to set a custom fingerprint that excludes all this stuff (since by default the entire stacktrace is used as the fingerprint).

Copy link
Contributor

@bitsandfoxes bitsandfoxes Mar 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll follow up on this. The blocking detection is opt-in and the improvements to the grouping are not blocking this PR. #3202

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll follow up on this. The blocking detection is opt-in and the improvements to the grouping are not blocking this PR. #3202

If we remove the logic above from the client, the grouping will be broken unless some similar logic exists on the server.

I had a chat to Bruno though... Not sure we need to be doing this on the server.


public void BlockingStart(DetectionSource detectionSource)
{
// From Stephen Cleary:
// "The default SynchronizationContext queues its asynchronous delegates to the ThreadPool but executes its
// synchronous delegates directly on the calling thread."
//
// Implicitly then, if we're not on a ThreadPool thread, we're not in an async context.
if (!Thread.CurrentThread.IsThreadPoolThread)
{
return;
}

_recursionTracker.Recurse();

try
{
if (!_recursionTracker.IsFirstRecursion())
{
return;
}

var stackTrace = DebugStackTrace.Create(
_options,
new StackTrace(true),
true,
ShouldSkipFrame
);
var evt = new SentryEvent
{
Level = SentryLevel.Warning,
Message =
"Blocking method has been invoked and blocked, this can lead to ThreadPool starvation. Learn more about it: " +
"https://learn.microsoft.com/en-us/aspnet/core/fundamentals/best-practices#avoid-blocking-calls ",
SentryExceptions = new[]
{
new SentryException
{
ThreadId = Environment.CurrentManagedThreadId,
Mechanism = new Mechanism
{
Type = "BlockingCallDetector",
Handled = false,
Description = "Blocking calls can cause ThreadPool starvation.",
Source = detectionSource.ToString()
},
Type = "Blocking call detected",
Stacktrace = stackTrace,
}
},
};

_getHub().CaptureEvent(evt);
}
catch
{
// ignored
}
}

public void BlockingEnd()
{
if (!Thread.CurrentThread.IsThreadPoolThread)
{
return;
}

_recursionTracker.Backtrack();
}
}

internal enum DetectionSource
{
SynchronizationContext,
EventListener
}
}
Loading
Loading