[Bug]: ResiliencePipelineBuilder AddCircuitBreaker grows stacktrace uncontrollably

[Doviskaune]

Describe the bug

Using ResiliencePipelineBuilder from v8, after Circuit Breaker is open, stacktrace in exception grows by three lines after each ExecuteAsync/Execute call.

Insights:

• Manually closing circuit breaker "resets" stacktrace but does not prevent it from growing
• Manually closing and then opening (isolating) circuit breaker prevents stacktrace from growing
• AdvancedCircuitBreakerAsync from v7 does not have this problem

Expected behavior

Stacktraces should be 3 lines long in all BrokenCircuitException exceptions

Actual behavior

Stacktraces at indices 3 and 8 have 6 lines, stacktraces at indices 4 and 9 have 9 lines. Incorrect stacktraces repeat following lines 2 or 3 times:

   at Polly.Outcome`1.GetResultOrRethrow()
   at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
   at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40

Steps to reproduce

Create new dotnet project "Console Application", add nuget package Polly and put following code into Program.cs file. Entire program output is provided in Exceptions part. To test sync version or v7 version, comment / uncomment lines 40-42.

using Polly;
using Polly.CircuitBreaker;

var manualControl = new CircuitBreakerManualControl();
var resiliencePipeline =
    new ResiliencePipelineBuilder()
        .AddCircuitBreaker(new CircuitBreakerStrategyOptions
        {
            Name = "CircuitBreaker",
            FailureRatio = 0.5,
            SamplingDuration = TimeSpan.FromMinutes(2),
            MinimumThroughput = 2,
            BreakDuration = TimeSpan.FromMinutes(5),
            ShouldHandle = new PredicateBuilder().Handle<Exception>(),
            ManualControl = manualControl,
        })
        .Build();

var policy = Policy<bool>
    .Handle<Exception>()
    .AdvancedCircuitBreakerAsync(0.5, TimeSpan.FromMinutes(2), 2, TimeSpan.FromMinutes(5));

for (int i = 0; i < 15; i++)
{
    if (i == 5)
    {
        Console.WriteLine("Manually closing circuit breaker");
        await manualControl.CloseAsync();
    }
    else if (i == 10)
    {
        Console.WriteLine("Manually closing and then opening circuit breaker");
        await manualControl.CloseAsync();
        await manualControl.IsolateAsync();
    }

    try
    {
        Console.WriteLine("Index: {0}", i);
        await resiliencePipeline.ExecuteAsync<bool>(static _ => { throw new Exception(); });
        // resiliencePipeline.Execute<bool>(static _ => { throw new Exception(); });
        // await policy.ExecuteAsync(static () => { throw new Exception(); });
    }
    catch (BrokenCircuitException ex)
    {
        Console.BackgroundColor = ConsoleColor.Red;
        Console.ForegroundColor = ConsoleColor.Black;
        Console.Write(ex.Message);
        Console.ResetColor();
        Console.WriteLine();
        Console.WriteLine(ex.StackTrace);
        Console.WriteLine();
    }
    catch (Exception ex)
    {
        Console.BackgroundColor = ConsoleColor.DarkYellow;
        Console.ForegroundColor = ConsoleColor.White;
        Console.Write(ex.Message);
        Console.ResetColor();
        Console.WriteLine();
        Console.WriteLine(ex.StackTrace);
        Console.WriteLine();
    }
    finally
    {
        await Task.Delay(100);
    }
}

Exception(s) (if any)

Index: 0
Exception of type 'System.Exception' was thrown.
at Program.<>c.<<Main>$>b__0_0(CancellationToken _) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40
at Polly.ResiliencePipeline.<>c__10`1.<<ExecuteAsync>b__10_0>d.MoveNext()
--- End of stack trace from previous location ---
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40

Index: 1
Exception of type 'System.Exception' was thrown.
at Program.<>c.<<Main>$>b__0_0(CancellationToken _) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40
at Polly.ResiliencePipeline.<>c__10`1.<<ExecuteAsync>b__10_0>d.MoveNext()
--- End of stack trace from previous location ---
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40

Index: 2
The circuit is now open and is not allowing calls.
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40

Index: 3
The circuit is now open and is not allowing calls.
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40

Index: 4
The circuit is now open and is not allowing calls.
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40

Manually closing circuit breaker
Index: 5
Exception of type 'System.Exception' was thrown.
at Program.<>c.<<Main>$>b__0_0(CancellationToken _) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40
at Polly.ResiliencePipeline.<>c__10`1.<<ExecuteAsync>b__10_0>d.MoveNext()
--- End of stack trace from previous location ---
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40

Index: 6
Exception of type 'System.Exception' was thrown.
at Program.<>c.<<Main>$>b__0_0(CancellationToken _) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40
at Polly.ResiliencePipeline.<>c__10`1.<<ExecuteAsync>b__10_0>d.MoveNext()
--- End of stack trace from previous location ---
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40

Index: 7
The circuit is now open and is not allowing calls.
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40

Index: 8
The circuit is now open and is not allowing calls.
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40

Index: 9
The circuit is now open and is not allowing calls.
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40

Manually closing and then opening circuit breaker
Index: 10
The circuit is manually held open and is not allowing calls.
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40

Index: 11
The circuit is manually held open and is not allowing calls.
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40

Index: 12
The circuit is manually held open and is not allowing calls.
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40

Index: 13
The circuit is manually held open and is not allowing calls.
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40

Index: 14
The circuit is manually held open and is not allowing calls.
at Polly.Outcome`1.GetResultOrRethrow()
at Polly.ResiliencePipeline.ExecuteAsync[TResult](Func`2 callback, CancellationToken cancellationToken)
at Program.<Main>$(String[] args) in C:\Users\xxx\RiderProjects\PollyTest\PollyTest\Program.cs:line 40

Polly version

8.2.0

.NET Version

8.0.100 (target framework - net8.0)

Anything else?

No response

[martintmk]

This is indeed an issue. I have opened the PR with the fix (#1878)

[martintmk]

@martincostello This can become issue in high-throughput services.

Wdyt about releasing Polly 8.3 that contains this fix? (on top of other enhancements) I am little worried about the Simmy, since the docs are not ready yet. But maybe we can just hide the APIs for 8.3?

cc @vany0114

[martincostello]

Yeah, I was going to ask @vany0114 - unless that work gets finished soon, then the easiest thing to do is probably create a branch from 8.2.0 and then cherry-pick various fixes over to it to do an 8.2.1 release.

[vany0114]

Sorry for the inconvenience guys, that's still a work in progress, I've made some progress however I haven't had much time lately to work on that, I'm planning to resume it next week.

[martincostello]

OK, I'll look to do a branch that cherry-picks just bug fixes so we can do a patch release before then.

[martincostello]

I've cherry-picked all the bug fixes and CI changes to a release/8.2 branch (diff).

If those look like all the right changes then I can do a 8.2.1 release from there.

[peter-csala]

@vany0114 I can offer my help with documentation if you need extra manpower :)

[martincostello]

@martintmk Can you take a look at the diff please just to check it's what you would expect? Then I will push a release if it looks correct.

[martintmk]

@martintmk Can you take a look at the diff please just to check it's what you would expect? Then I will push a release if it looks correct.

Looks good to me!

[martincostello]

This fix is now available from NuGet.org in Polly 8.2.1.

[vany0114]

@vany0114 I can offer my help with documentation if you need extra manpower :)

@peter-csala That'd be great, I'll open a draft to what I have currently so you can also work from there.