[browser][MT] mono_download_assets - RangeError: Start offset -1 is outside the bounds of the buffer

[pavelsavara]

Log

[11:40:15] fail: [out of order message from the browser]: http://127.0.0.1:33721/_framework/dotnet.runtime.js 2:76111 Uncaught RangeError: Start offset -1 is outside the bounds of the buffer
[11:40:15] fail: Error in mono_download_assets: RangeError: Start offset -1 is outside the bounds of the buffer
[11:40:15] fail: [--000fb3fc-emsc 11:40:15.219] MONO_WASM: Start offset -1 is outside the bounds of the buffer
                 RangeError: Start offset -1 is outside the bounds of the buffer
                     at new Uint8Array (<anonymous>)
                     at http://127.0.0.1:33721/_framework/dotnet.runtime.js:3:76112
                     at Object.sa [as instantiate_asset] (http://127.0.0.1:33721/_framework/dotnet.runtime.js:3:76160)
                     at n (http://127.0.0.1:33721/_framework/dotnet.js:3:8629)
                     at async Promise.all (index 73)

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=674731
Build error leg or test failing:

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorPattern": ["mono_download_assets", "RangeError"],
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=674731
Error message validated: [mono_download_assets RangeError]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 5/15/2024 10:05:35 AM UTC

Report

Build	Definition	Test	Pull Request
678702	dotnet/runtime	WasmTestOnChrome-MT-System.Security.SecureString.Tests.WorkItemExecution	#102363
679376	dotnet/runtime	WasmTestOnChrome-MT-Microsoft.Bcl.AsyncInterfaces.Tests.WorkItemExecution	#102375
679712	dotnet/runtime	WasmTestOnChrome-MT-System.Linq.Parallel.Tests.WorkItemExecution
679157	dotnet/runtime	WasmTestOnChrome-MT-System.Private.Xml.Tests.WorkItemExecution	#102188
678956	dotnet/runtime	WasmTestOnChrome-MT-System.Reflection.InvokeEmit.Tests.WorkItemExecution	#102185
679183	dotnet/runtime	WasmTestOnFirefox-MT-System.Net.WebSockets.Client.Tests.WorkItemExecution	#99818
678900	dotnet/runtime	WasmTestOnFirefox-MT-System.Xml.XmlSerializer.ReflectionOnly.Tests.WorkItemExecution	#102143
678959	dotnet/runtime	WasmTestOnChrome-MT-System.Net.Primitives.Functional.Tests.WorkItemExecution	#102375
678869	dotnet/runtime	WasmTestOnChrome-MT-System.Reflection.MetadataLoadContext.Tests.WorkItemExecution
678827	dotnet/runtime	WasmTestOnChrome-MT-System.Reflection.Metadata.Tests.WorkItemExecution	#100675
678753	dotnet/runtime	WasmTestOnChrome-MT-System.Reflection.InvokeInterpreted.Tests.WorkItemExecution	#102365
677948	dotnet/runtime	WasmTestOnChrome-MT-Microsoft.Extensions.HostFactoryResolver.Tests.WorkItemExecution	#101938
677879	dotnet/runtime	WasmTestOnChrome-MT-System.ServiceModel.Syndication.Tests.WorkItemExecution	#100675
677621	dotnet/runtime	WasmTestOnFirefox-MT-System.Private.Uri.Functional.Tests.WorkItemExecution	#102255
677596	dotnet/runtime	WasmTestOnChrome-MT-System.Diagnostics.StackTrace.Tests.WorkItemExecution	#101975
677577	dotnet/runtime	WasmTestOnChrome-MT-System.Threading.ThreadPool.Tests.WorkItemExecution	#101543
677422	dotnet/runtime	WasmTestOnFirefox-MT-System.Xml.Linq.Axes.Tests.WorkItemExecution
677460	dotnet/runtime	WasmTestOnChrome-MT-System.Xml.XmlSerializer.ReflectionOnly.Tests.WorkItemExecution	#102212
676620	dotnet/runtime	WasmTestOnChrome-MT-System.Collections.Tests.WorkItemExecution	#102143
677282	dotnet/runtime	WasmTestOnFirefox-MT-System.Net.Mail.Functional.Tests.WorkItemExecution	#102292
676525	dotnet/runtime	WasmTestOnFirefox-MT-System.Net.Mail.Unit.Tests.WorkItemExecution	#102265
676609	dotnet/runtime	WasmTestOnChrome-MT-System.Runtime.ReflectionInvokeEmit.Tests.WorkItemExecution	#102185
676344	dotnet/runtime	WasmTestOnChrome-MT-System.IO.UnmanagedMemoryStream.Tests.WorkItemExecution	#101543
676289	dotnet/runtime	WasmTestOnChrome-MT-System.ValueTuple.Tests.WorkItemExecution	#102188
676279	dotnet/runtime	WasmTestOnChrome-MT-System.Drawing.Primitives.Tests.WorkItemExecution	#102212
676251	dotnet/runtime	WasmTestOnChrome-MT-Microsoft.Extensions.Configuration.CommandLine.Tests.WorkItemExecution	#102254
676242	dotnet/runtime	WasmTestOnChrome-MT-System.Xml.Linq.Properties.Tests.WorkItemExecution	#100675
676228	dotnet/runtime	WasmTestOnChrome-MT-Microsoft.Extensions.Configuration.Xml.Tests.WorkItemExecution	#101975
676151	dotnet/runtime	WasmTestOnFirefox-MT-Invariant.Tests.WorkItemExecution	#101252
676144	dotnet/runtime	WasmTestOnFirefox-MT-Invariant.Tests.WorkItemExecution	#102184
674731	dotnet/runtime	WasmTestOnChrome-MT-System.Runtime.Caching.Tests.WorkItemExecution	#100141

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	31	31

[dotnet-policy-service]

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

[pavelsavara]

cc @kg

[kg]

I think this would indicate that malloc/sbrk/mmap failed during startup. Seems like a threading bug (emscripten libc bug?)

[pavelsavara]

This one is the same thing from firefox
Log

JavaScript error: http://127.0.0.1:41545/_framework/dotnet.runtime.js, line 3: RangeError: invalid or out-of-range index
JavaScript error: http://127.0.0.1:41545/_framework/dotnet.runtime.js, line 3: RangeError: invalid or out-of-range index
JavaScript error: http://127.0.0.1:41545/_framework/dotnet.runtime.js, line 3: RangeError: invalid or out-of-range index
JavaScript error: http://127.0.0.1:41545/_framework/dotnet.runtime.js, line 3: RangeError: invalid or out-of-range index
JavaScript error: http://127.0.0.1:41545/_framework/dotnet.runtime.js, line 3: RangeError: invalid or out-of-range index
[08:07:17] fail: Error in mono_download_assets: RangeError: invalid or out-of-range index
[08:07:17] fail: [--000fb3fc-emsc 08:07:17.444] MONO_WASM: invalid or out-of-range index
                 sa/a<@http://127.0.0.1:41545/_framework/dotnet.runtime.js:3:76112
                 sa@http://127.0.0.1:41545/_framework/dotnet.runtime.js:3:76160
                 n@http://127.0.0.1:41545/_framework/dotnet.js:3:8629
[08:07:17] info: WASM EXIT 1

[pavelsavara]

More Log Log

[pavelsavara]

Maybe emscripten update could help. See emscripten-core/emscripten@6a41187

[kg]

The structure of emscripten's implementation of sbrk and v8's implementation of memory.grow both imply that it shouldn't be possible to get -1 unless we run out of memory.

However, looking closer at v8's underlying implementation, if you're using an SAB and multiple threads are racing to grow the heap, it looks like if mprotect() were to fail for any reason, the grow would immediately fail and return -1 even if you're not out of memory (there's no retry/loop in the implementation, regardless of what error code mprotect returns). I only consider this remotely plausible since (AFAIK) this failure is only happening on ubuntu and only affects MT lanes, since this logic is specifically different in V8 for SABs.

Still feels highly unlikely, though...

[kg]

It looks like it's possible for mprotect to return EINTR despite what the docs say, so that could be happening here.

[pavelsavara]

I think we could lock around our allocations or retry
We could replace emscripten's emscripten_resize_heap in JS

[kg]

I think we could lock around our allocations or retry We could replace emscripten's emscripten_resize_heap in JS

I looked into doing this before, but it seems very risky. It could cause problems in emscripten dlmalloc/mimalloc/emmalloc if we were to manage the heap size ourselves.

[kg]

24 hour hit count is 0, closing