Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IBackgroundTask services throw ObjectDisposedException because the YesSql ISession is disposed #16450

Open
sarahelsaig opened this issue Jul 17, 2024 · 5 comments
Labels
Milestone

Comments

@sarahelsaig
Copy link
Contributor

Describe the bug

Doesn't happen always, but when it happens all background tasks throw the following exception, so it may be something further up the line.

Error while updating settings of background task '...' on tenant 'Default'. System.ObjectDisposedException: Cannot access a disposed object.

Orchard Core version

2.0.0-preview-18268

To Reproduce

I will update this if I find a reliable way to reproduce.

Expected behavior

Background tasks should not use disposed dependencies.

Logs and screenshots

A truncated set of examples from the app log:

2024-07-16 22:48:01.2504|Default|0HN55T59BCOHD||OrchardCore.Modules.ModularBackgroundService|ERROR|Error while updating settings of background task 'OrchardCore.AuditTrail.Services.AuditTrailBackgroundTask' on tenant 'Default'. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Session'.
   at YesSql.Session.CheckDisposed()
   at YesSql.Session.CreateConnectionAsync()
   at YesSql.Services.DefaultQuery.Query`1.FirstOrDefaultImpl()
   at OrchardCore.Data.Documents.DocumentStore.GetOrCreateImmutableAsync[T](Func`1 factoryAsync)
   at OrchardCore.Documents.DocumentManager`1.GetOrCreateImmutableAsync(Func`1 factoryAsync)
   at OrchardCore.BackgroundTasks.Services.BackgroundTaskSettingsProvider.GetSettingsAsync(IBackgroundTask task)
   at OrchardCore.Modules.ModularBackgroundService.<>c__DisplayClass12_1.<<UpdateAsync>b__1>d.MoveNext()    at YesSql.Session.CheckDisposed()
   at YesSql.Session.CreateConnectionAsync()
   at YesSql.Services.DefaultQuery.Query`1.FirstOrDefaultImpl()
   at OrchardCore.Data.Documents.DocumentStore.GetOrCreateImmutableAsync[T](Func`1 factoryAsync)
   at OrchardCore.Documents.DocumentManager`1.GetOrCreateImmutableAsync(Func`1 factoryAsync)
   at OrchardCore.BackgroundTasks.Services.BackgroundTaskSettingsProvider.GetSettingsAsync(IBackgroundTask task)
   at OrchardCore.Modules.ModularBackgroundService.<>c__DisplayClass12_1.<<UpdateAsync>b__1>d.MoveNext()
2024-07-16 22:48:01.2632|Default|0HN55T59BCOHD||OrchardCore.Modules.ModularBackgroundService|ERROR|Error while updating settings of background task 'OrchardCore.Media.Services.ResizedMediaCacheBackgroundTask' on tenant 'Default'. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Session'.
   at YesSql.Session.CheckDisposed()
   at YesSql.Session.CreateConnectionAsync()
   at YesSql.Services.DefaultQuery.Query`1.FirstOrDefaultImpl()
   at OrchardCore.Data.Documents.DocumentStore.GetOrCreateImmutableAsync[T](Func`1 factoryAsync)
   at OrchardCore.Documents.DocumentManager`1.GetOrCreateImmutableAsync(Func`1 factoryAsync)
   at OrchardCore.BackgroundTasks.Services.BackgroundTaskSettingsProvider.GetSettingsAsync(IBackgroundTask task)
   at OrchardCore.Modules.ModularBackgroundService.<>c__DisplayClass12_1.<<UpdateAsync>b__1>d.MoveNext()    at YesSql.Session.CheckDisposed()
   at YesSql.Session.CreateConnectionAsync()
   at YesSql.Services.DefaultQuery.Query`1.FirstOrDefaultImpl()
   at OrchardCore.Data.Documents.DocumentStore.GetOrCreateImmutableAsync[T](Func`1 factoryAsync)
   at OrchardCore.Documents.DocumentManager`1.GetOrCreateImmutableAsync(Func`1 factoryAsync)
   at OrchardCore.BackgroundTasks.Services.BackgroundTaskSettingsProvider.GetSettingsAsync(IBackgroundTask task)
   at OrchardCore.Modules.ModularBackgroundService.<>c__DisplayClass12_1.<<UpdateAsync>b__1>d.MoveNext()
2024-07-16 22:48:01.2640|Default|0HN55T59BCOHD||OrchardCore.Modules.ModularBackgroundService|ERROR|Error while updating settings of background task 'OrchardCore.Media.Services.RemoteMediaCacheBackgroundTask' on tenant 'Default'. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Session'.
   at YesSql.Session.CheckDisposed()
   at YesSql.Session.CreateConnectionAsync()
   at YesSql.Services.DefaultQuery.Query`1.FirstOrDefaultImpl()
   at OrchardCore.Data.Documents.DocumentStore.GetOrCreateImmutableAsync[T](Func`1 factoryAsync)
   at OrchardCore.Documents.DocumentManager`1.GetOrCreateImmutableAsync(Func`1 factoryAsync)
   at OrchardCore.BackgroundTasks.Services.BackgroundTaskSettingsProvider.GetSettingsAsync(IBackgroundTask task)
   at OrchardCore.Modules.ModularBackgroundService.<>c__DisplayClass12_1.<<UpdateAsync>b__1>d.MoveNext()    at YesSql.Session.CheckDisposed()
   at YesSql.Session.CreateConnectionAsync()
   at YesSql.Services.DefaultQuery.Query`1.FirstOrDefaultImpl()
   at OrchardCore.Data.Documents.DocumentStore.GetOrCreateImmutableAsync[T](Func`1 factoryAsync)
   at OrchardCore.Documents.DocumentManager`1.GetOrCreateImmutableAsync(Func`1 factoryAsync)
   at OrchardCore.BackgroundTasks.Services.BackgroundTaskSettingsProvider.GetSettingsAsync(IBackgroundTask task)
   at OrchardCore.Modules.ModularBackgroundService.<>c__DisplayClass12_1.<<UpdateAsync>b__1>d.MoveNext()
2024-07-16 22:48:01.2640|Default|0HN55T59BCOHD||OrchardCore.Modules.ModularBackgroundService|ERROR|Error while updating settings of background task 'OrchardCore.Media.Services.ChunkFileUploadBackgroundTask' on tenant 'Default'. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'Session'.
   at YesSql.Session.CheckDisposed()
   at YesSql.Session.CreateConnectionAsync()
   at YesSql.Services.DefaultQuery.Query`1.FirstOrDefaultImpl()
   at OrchardCore.Data.Documents.DocumentStore.GetOrCreateImmutableAsync[T](Func`1 factoryAsync)
   at OrchardCore.Documents.DocumentManager`1.GetOrCreateImmutableAsync(Func`1 factoryAsync)
   at OrchardCore.BackgroundTasks.Services.BackgroundTaskSettingsProvider.GetSettingsAsync(IBackgroundTask task)
   at OrchardCore.Modules.ModularBackgroundService.<>c__DisplayClass12_1.<<UpdateAsync>b__1>d.MoveNext()    at YesSql.Session.CheckDisposed()
   at YesSql.Session.CreateConnectionAsync()
   at YesSql.Services.DefaultQuery.Query`1.FirstOrDefaultImpl()
   at OrchardCore.Data.Documents.DocumentStore.GetOrCreateImmutableAsync[T](Func`1 factoryAsync)
   at OrchardCore.Documents.DocumentManager`1.GetOrCreateImmutableAsync(Func`1 factoryAsync)
   at OrchardCore.BackgroundTasks.Services.BackgroundTaskSettingsProvider.GetSettingsAsync(IBackgroundTask task)
   at OrchardCore.Modules.ModularBackgroundService.<>c__DisplayClass12_1.<<UpdateAsync>b__1>d.MoveNext()
@Piedone
Copy link
Member

Piedone commented Jul 17, 2024

Yeah this needs some kind of repro but perhaps there's some exception happening earlier in BG task execution that gets swallowed but causes ISession to be disposed? You can try increasing the log level to see more info to at least be able to observe more when this does happen.

@sarahelsaig
Copy link
Contributor Author

Way ahead of you, but somehow adding the extra log made the problem go away. Well, like I said it doesn't happen always. I will keep rerunning it to see if I can catch it with more logs.

@sebastienros sebastienros added this to the 2.x milestone Jul 18, 2024
Copy link
Contributor

We triaged this issue and set the milestone according to the priority we think is appropriate (see the docs on how we triage and prioritize issues).

This indicates when the core team may start working on it. However, if you'd like to contribute, we'd warmly welcome you to do that anytime. See our guide on contributions here.

@sebastienros
Copy link
Member

First step would be to understand what is the lifetime of a background task and how service provider scopes are managed in this case. That could hint at some bad patterns that dispose a Session where it shouldn't.

@gvkries
Copy link
Contributor

gvkries commented Aug 6, 2024

I tried to understand how this ObjectDisposedException is possible. As far as I see, the session is only ever disposed through its parent container, as expected. As it is only ever hold by the active ShellScope, this one must be disposed too early in some cases.

Because of that, it was my suspicion that something is not right with the ShellScopes lifetime. I thoroughly reviewed any possible call stack, but I only found one hypothetical case where a ShellScope gets disposed unexpectedly.

The ShellScopes themselfs are only disposed in ShellScope.UsingAsync() (ignoring some error handling). If one captures the current scope and reuses it later from a nested scope, the scope gets disposed too early. E.g. the following unit test would fail:

[Fact]
public async Task RecursiveUsingAsyncCallsMustNotDisposeScope()
{
    await ShellHost.InitializeAsync();

    var shellContext = await ShellHost.GetOrCreateShellContextAsync(
        new ShellSettings()
        {
            Name = "Tenant1",
            RequestUrlPrefix = "tenant1",
        }
        .AsUninitialized());

    var rootScope = await shellContext.CreateScopeAsync();
    await rootScope.UsingAsync(async (scope) =>
    {
        await (await ShellScope.CreateChildScopeAsync()).UsingAsync(async (childScope) =>
        {
            // Second UsingAsync() of the parent scope must not dispose the scope.
            await scope.UsingAsync(innerRootScope => Task.CompletedTask);
        });

        // rootScope must not be disposed by calling UsingAsync() again if it is a recursive call.
        Assert.False(scope.IsDisposed);
    });

    Assert.True(rootScope.IsDisposed);
}

(Note: This requires to expose the _disposed field of ShellScope.)

This happens, because the AsyncLocal which holds the current scope is used to allow reentrancy of UsingAsync(). But it gets set to the new scope when using a separate child scope.

I did not find this usage pattern anywhere in Orchards source, so I'm pretty sure this is not the actual issue. Also as the error does not occur every time, it is possibly some kind of a difficult to find race condition. But maybe we could guard against this case a little bit better.

This may also be related to #16511, where a ShellScope is also disposed unexcpectedly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants