.NET apps are crashing in pod with entire host file system mounted as volume

[jefflill]

I'm working on the neon-node-agent service and am trying to mount the entire host file system so the service can perform operations directly on the node.

Unfortunately, the service crashes shortly after stopping logging a deep FileSystemWatcher related exception which seems to have been thrown 170K+ times:

This seems to be an issue for non-Windows environments like Linux, where the do watch the entire file system by default and Linux doesn't have a way to watch subfolders without creating new watches link and adding the watches can take 100ms! So this problem impacts services that are not mounting the host filesystem.

λ neon logs -f daemonset/neon-node-agent
[2022-01-07T19:22:33.244+00:00] [INFO] [version:0.3.0-alpha] [index:1] Starting [neon-node-agent:0.3.0-alpha]
Stack overflow.
Repeat 174341 times:
--------------------------------
   at System.IO.FileSystemWatcher+RunningInstance+WatchedDirectory.Write(System.Text.StringBuilder, Boolean)
--------------------------------
   at System.IO.FileSystemWatcher+RunningInstance+WatchedDirectory.GetPath(Boolean, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance..ctor(System.IO.FileSystemWatcher, Microsoft.Win32.SafeHandles.SafeFileHandle, System.String, Boolean, System.IO.NotifyFilters, System.Threading.CancellationToken)
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
   at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean)
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(System.String)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(System.String)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
   at Microsoft.Extensions.Primitives.ChangeToken.OnChange(System.Func`1<Microsoft.Extensions.Primitives.IChangeToken>, System.Action)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(Microsoft.Extensions.Configuration.FileConfigurationSource)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(Microsoft.Extensions.Configuration.IConfigurationBuilder)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at NeonNodeAgent.Service+<OnRunAsync>d__4.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[NeonNodeAgent.Service+<OnRunAsync>d__4, neon-node-agent, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null]](<OnRunAsync>d__4 ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[NeonNodeAgent.Service+<OnRunAsync>d__4, neon-node-agent, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null]](<OnRunAsync>d__4 ByRef)
   at NeonNodeAgent.Service.OnRunAsync()
   at Neon.Service.NeonService+<RunAsync>d__108.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Neon.Service.NeonService+<RunAsync>d__108, Neon.Service, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null]](<RunAsync>d__108 ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Int32, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Neon.Service.NeonService+<RunAsync>d__108, Neon.Service, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null]](<RunAsync>d__108 ByRef)
   at Neon.Service.NeonService.RunAsync(Boolean)
   at NeonNodeAgent.Program+<Main>d__0.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[NeonNodeAgent.Program+<Main>d__0, neon-node-agent, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null]](<Main>d__0 ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[NeonNodeAgent.Program+<Main>d__0, neon-node-agent, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null]](<Main>d__0 ByRef)
   at NeonNodeAgent.Program.Main(System.String[])
   at NeonNodeAgent.Program.<Main>(System.String[])

[jefflill]

I betting that we're running into some kind of inotify limit in the container. The .NET runtime seems to be working OK, so this issue probably isn't related:

dotnet/runtime#27272

After some more research, this might be the issue:

dotnet/runtime#42036

We're seeing this problem when starting the ASPNET site.

[jefflill]

It looks like we need to pass reloadOnChange: false when adding configuration files:

https://github.com/Azure/bicep/pull/3044/commits

Here's one of their commit changes:

Azure/bicep@13277e0

[jefflill]

@marcusbooyah: So I fixed this for the operators by adding this code to disable loading of config files (which we don't currently use):

We should do this for all of our ASPNET services. I'll let you do this for neon-dashboard.

There may also be a problem with Blazor watching for razor template file changes to trigger a recompile. You may also need to find a way to constrain that too.

[marcusbooyah]

DONE