Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[browser][publish] Windows has problems with AOT build/publish for long paths #103625

Open
ilonatommy opened this issue Jun 18, 2024 · 17 comments
Open
Assignees
Labels
arch-wasm WebAssembly architecture area-Build-mono wasm-aot-test WebAssembly AOT Test
Milestone

Comments

@ilonatommy
Copy link
Member

ilonatommy commented Jun 18, 2024

Reproduction:

  1. WBT Wasm.Build.Tests.Blazor.BuildPublishTests - edit id to be $"blz_aot_{config}_{GetRandomId()}_TEST_OF_EXTREMELY_LONG_PATH", or anything that will result in path longer than 260 chars.
  2. Run, first error will be connected with g_fopen code:
Unable to open trimming-eligible-methods-outfile specified file C:\Users\username\source\repos\runtime-fork\artifacts\bin\Wasm.Build.Tests\Release\net9.0\win-x64\wbt artifacts\blz_aot_Debug_tumnkn5b_5fc_TEST_OF_LONG_PATH\obj\Debug\net9.0\wasm\for-publish\tokens\Microsoft_Extensions_DependencyInjection_dll_compiled_methods.txt

and can be fixed by editing g_fopen (const gchar *path, const gchar *mode) from gfile.c to:

gchar *path_mod;
#ifdef HOST_WIN32
	// add long-path prefix
	path_mod = g_malloc(strlen(path) + 5);
	strcpy(path_mod, "\\\\?\\");
	strcat(path_mod, path);
	if (is_ascii_string (path_mod) && is_ascii_string (mode)) {
		fp = fopen (path_mod, mode);
	} else {
		gunichar2 *wPath = g_utf8_to_utf16 (path_mod, -1, 0, 0, 0);
		gunichar2 *wMode = g_utf8_to_utf16 (mode, -1, 0, 0, 0);

		if (!wPath || !wMode)
			return NULL;

		fp = _wfopen ((wchar_t *) wPath, (wchar_t *) wMode);
		g_free (wPath);
		g_free (wMode);
	}
    	g_free (path_mod);
#else

Further errors come from aot_printerrf (acfg, "Failed to load methodspec 0x%x due to %s.\n", token, mono_error_get_message (error)); in aot-compiler.c.

mono_get_method_checked Failed to load method 0x60001cc from 'C:\Users\source\repos\runtime-fork\artifacts\bin\Wasm.Build.Tests\Debug\net9.0\win-x64\wbt artifacts\blz_aot_Release_1t1y0tcq_4j1_TEST_OF_EXTREMELY_LONG_PATH\obj\Release\net9.0\wasm\for-publish\aot-in\Microsoft.AspNetCore.Components.WebAssembly.dll'
          [] C:\Users\source\repos\runtime-fork\artifacts\bin\dotnet-latest\packs\Microsoft.NET.Runtime.WebAssembly.Sdk\9.0.0-dev\Sdk\WasmApp.Common.targets(697,5): error : due to Could not load file or assembly 'Microsoft.Extensions.Configuration.Abstractions, Version=9.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60' or one of its dependencies..
          [] C:\Users\source\repos\runtime-fork\artifacts\bin\dotnet-latest\packs\Microsoft.NET.Runtime.WebAssembly.Sdk\9.0.0-dev\Sdk\WasmApp.Common.targets(697,5): error : Run with MONO_LOG_LEVEL=debug for more information.
          [] C:\Users\source\repos\runtime-fork\artifacts\bin\dotnet-latest\packs\Microsoft.NET.Runtime.WebAssembly.Sdk\9.0.0-dev\Sdk\WasmApp.Common.targets(697,5): error : AOT of image C:\Users\source\repos\runtime-fork\artifacts\bin\Wasm.Build.Tests\Debug\net9.0\win-x64\wbt artifacts\blz_aot_Release_1t1y0tcq_4j1_TEST_OF_EXTREMELY_LONG_PATH\obj\Release\net9.0\wasm\for-publish\aot-in\Microsoft.AspNetCore.Components.WebAssembly.dll failed

They are caused by mono_get_method_checked from loader.c getting empty result but I am not able to track down the reason for it.
cc @kg, @BrzVlad any ideas what might be going wrong there?

For debugging it is helpful to switch off parallelization of library precompiling in MonoAOTCompiler (PrecompileLibraryParallel).

@ilonatommy ilonatommy added arch-wasm WebAssembly architecture area-Build-mono wasm-aot-test WebAssembly AOT Test labels Jun 18, 2024
@ilonatommy ilonatommy self-assigned this Jun 18, 2024
Copy link
Contributor

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

@BrzVlad
Copy link
Member

BrzVlad commented Jun 18, 2024

From what I know we open metadata stuff with mono_file_map_open. You could try applying your fix there. Also replacing any uses in the runtime of fopen to g_fopen wouldn't hurt.

@ilonatommy
Copy link
Member Author

@BrzVlad thanks for ideas but the proposed fixes are not enough. Do we have a way to dump stack trace? We are getting to mono_error_set_simple_file_not_found in exception.c and setting the error there. I have hard time collecting the stack manually, are there any other ways than mono_wasm_print_stack_trace?

#ifdef HOST_WASM
#include <emscripten.h>
EM_ASM(
	var err = new Error();
	console.log ("Stacktrace: \n");
	console.log (err.stack);
	);
#endif

It does not print anything (debug mode).

@BrzVlad
Copy link
Member

BrzVlad commented Jun 18, 2024

I don't think we have something to dump native stack. I used in the past for various reasons the backtrace api but I have no idea if it works on wasm/windows. You can check a potentially bitrotten implementation in mono_backtrace.

However, I thought the error happens during app compilation ? My understanding that this is during aot compilation to llvm bytecode ? So I don't think it makes sense to use browser api ?

@BrzVlad
Copy link
Member

BrzVlad commented Jun 18, 2024

If you are running a desktop program, you could always make it assert when that error is set and then I think it would ask you automatically to attach with VS where you can get stack trace and other things.

@ilonatommy
Copy link
Member Author

ilonatommy commented Jun 18, 2024

Yes, it's app build step and yes, JS code does not make sense, though that was the only stack trace dump I found in the code.

If you are running a desktop program

The problems are with WASM app.

@kg
Copy link
Contributor

kg commented Jun 19, 2024

I would strongly advise against gating the long path (the \\?\ stuff) based on how long the input path is. Long paths have different parsing rules, so having our parsing behavior change arbitrarily based on how long the input path is will cause lots of really confusing failure modes for end users. We should either always use it (on targets where it's available - all modern windows, i think) or never use it.

@ilonatommy
Copy link
Member Author

We should either always use it (on targets where it's available - all modern windows, i think) or never use it.

Thank you. I changed the snipped in the description to always append the prefix. Does any reason for Could not load file or assembly ... or one of its dependencies. come to your mind?

@kg
Copy link
Contributor

kg commented Jun 20, 2024

Have you tried MONO_LOG_LEVEL=debug? The answer is probably in the debug log messages.

@kg
Copy link
Contributor

kg commented Jun 20, 2024

From grepping around, you may need to edit mono_file_map_open to use long paths.

@lewing
Copy link
Member

lewing commented Jul 29, 2024

@ilonatommy is there work left for 9.0 here?

@ilonatommy
Copy link
Member Author

ilonatommy commented Jul 30, 2024

Yes, I cannot find the proper place that has to be fixed and I got stuck with the PR that is linked to it (#103766), with no idea how to proceed.

@pavelsavara
Copy link
Member

@jeromelaban shared this workaround

<Target Name="_WorkaroundEmscriptenPathLength" Condition=" '$(OS)' == 'Windows_NT' " BeforeTargets="_SetupEmscripten">
    <Exec Command="mklink /J &quot;$(TMP)\emsdk&quot; &quot;$(EmscriptenSdkToolsPath)..\tools\&quot;" ContinueOnError="true" />
        
    <PropertyGroup>
        <EmscriptenSdkToolsPath>$(TMP)\emsdk\</EmscriptenSdkToolsPath>
        <EmscriptenUpstreamBinPath>$(EmscriptenSdkToolsPath)bin\</EmscriptenUpstreamBinPath>
        <EmscriptenUpstreamEmscriptenPath>$(EmscriptenSdkToolsPath)emscripten\</EmscriptenUpstreamEmscriptenPath>
    </PropertyGroup>
    <ItemGroup>
        <EmscriptenPrependPATH Remove="@(EmscriptenPrependPATH)" />
        <EmscriptenPrependPATH Include="$(EmscriptenUpstreamBinPath)" />
        <EmscriptenPrependPATH Include="$(EmscriptenUpstreamEmscriptenPath)" />
    </ItemGroup>
</Target>

@jeromelaban
Copy link
Contributor

Without the workaround, here's the output:

2024-09-05T21:05:06.2473473Z   system_libs:INFO: compiled 2 inputs in 0.32s
2024-09-05T21:05:06.3910679Z   cache:INFO:  - ok
2024-09-05T21:05:06.3922699Z   cache:INFO: generating system library: sysroot\lib\wasm32-emscripten\libc.a... (this will be cached in "C:\Users\VssAdministrator\AppData\Local\Temp\emsdk\sysroot\lib\wasm32-emscripten\libc.a" for subsequent builds)
2024-09-05T21:05:06.4710051Z    "C:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Sdk.win-x64\9.0.0-rc.2.24455.1\tools\emscripten\emcc.bat" -c -O2 -Wall -fno-unroll-loops -std=c99 -D_XOPEN_SOURCE=700 -Wno-unused-result -Os -fno-inline-functions -fno-builtin -Wno-ignored-attributes -Wno-macro-redefined -Wno-shift-op-parentheses -Wno-string-plus-int -Wno-missing-braces -Wno-logical-op-parentheses -Wno-bitwise-op-parentheses -Wno-unused-but-set-variable -Wno-unused-variable -Wno-unused-label -Wno-pointer-sign -g -sSTRICT -Werror "-IC:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Sdk.win-x64\9.0.0-rc.2.24455.1\tools\emscripten\system\lib\libc\musl\src\internal" "-IC:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Sdk.win-x64\9.0.0-rc.2.24455.1\tools\emscripten\system\lib\libc\musl\src\include" "-IC:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Sdk.win-x64\9.0.0-rc.2.24455.1\tools\emscripten\system\lib\libc\musl\include" "-IC:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Sdk.win-x64\9.0.0-rc.2.24455.1\tools\emscripten\system\lib\libc" "-IC:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Sdk.win-x64\9.0.0-rc.2.24455.1\tools\emscripten\system\lib\pthread" -DNDEBUG "C:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Sdk.win-x64\9.0.0-rc.2.24455.1\tools\emscripten\system\lib\libc\musl\src\exit\_Exit.c" -o C:\Users\VssAdministrator\AppData\Local\Temp\emsdk\build\libc-tmp\_exit__1.o
2024-09-05T21:05:06.4723585Z    "C:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Sdk.win-x64\9.0.0-rc.2.24455.1\tools\emscripten\emcc.bat" @C:\Users\VSSADM~1\AppData\Local\Temp\emscripten_cldyffve.rsp.utf-8
2024-09-05T21:05:06.7282268Z   Traceback (most recent call last):
2024-09-05T21:05:06.7307973Z     File "C:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Sdk.win-x64\9.0.0-rc.2.24455.1\tools\emscripten\emcc.py", line 1582, in <module>
2024-09-05T21:05:06.7323652Z       sys.exit(main(sys.argv))
2024-09-05T21:05:06.7330573Z                ^^^^^^^^^^^^^^
2024-09-05T21:05:06.7353142Z     File "C:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Python.win-x64\9.0.0-rc.2.24455.1\tools\Lib\contextlib.py", line 81, in inner
2024-09-05T21:05:06.7381031Z       return func(*args, **kwds)
2024-09-05T21:05:06.7388552Z              ^^^^^^^^^^^^^^^^^^^
2024-09-05T21:05:06.7390631Z     File "C:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Sdk.win-x64\9.0.0-rc.2.24455.1\tools\emscripten\emcc.py", line 1575, in main
2024-09-05T21:05:06.7392280Z       ret = run(args)
2024-09-05T21:05:06.7393010Z             ^^^^^^^^^
2024-09-05T21:05:06.7401248Z     File "C:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Sdk.win-x64\9.0.0-rc.2.24455.1\tools\emscripten\emcc.py", line 632, in run
2024-09-05T21:05:06.7402864Z       linker_inputs = phase_compile_inputs(options, state, newargs, input_files)
2024-09-05T21:05:06.7405677Z                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-05T21:05:06.7407531Z     File "C:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Python.win-x64\9.0.0-rc.2.24455.1\tools\Lib\contextlib.py", line 81, in inner
2024-09-05T21:05:06.7409071Z       return func(*args, **kwds)
2024-09-05T21:05:06.7409833Z              ^^^^^^^^^^^^^^^^^^^
2024-09-05T21:05:06.7425064Z     File "C:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Sdk.win-x64\9.0.0-rc.2.24455.1\tools\emscripten\emcc.py", line 990, in phase_compile_inputs
2024-09-05T21:05:06.7430957Z       shared.exec_process(cmd)
2024-09-05T21:05:06.7444346Z     File "C:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Sdk.win-x64\9.0.0-rc.2.24455.1\tools\emscripten\tools\shared.py", line 247, in exec_process
2024-09-05T21:05:06.7526237Z       rtn = run_process(cmd, stdin=sys.stdin, check=False).returncode
2024-09-05T21:05:06.7547017Z             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-05T21:05:06.7696777Z     File "C:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Sdk.win-x64\9.0.0-rc.2.24455.1\tools\emscripten\tools\shared.py", line 130, in run_process
2024-09-05T21:05:06.7699364Z       ret = subprocess.run(cmd, check=check, input=input, *args, **kw)
2024-09-05T21:05:06.7701014Z             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-05T21:05:06.7712941Z     File "C:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Python.win-x64\9.0.0-rc.2.24455.1\tools\Lib\subprocess.py", line 548, in run
2024-09-05T21:05:06.7715076Z       with Popen(*popenargs, **kwargs) as process:
2024-09-05T21:05:06.7745063Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-05T21:05:06.7750314Z     File "C:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Python.win-x64\9.0.0-rc.2.24455.1\tools\Lib\subprocess.py", line 1026, in __init__
2024-09-05T21:05:06.7752178Z       self._execute_child(args, executable, preexec_fn, close_fds,
2024-09-05T21:05:06.7753421Z     File "C:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Python.win-x64\9.0.0-rc.2.24455.1\tools\Lib\subprocess.py", line 1538, in _execute_child
2024-09-05T21:05:06.7754873Z       hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
2024-09-05T21:05:06.7758782Z                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-09-05T21:05:06.7760230Z   FileNotFoundError: [WinError 206] The filename or extension is too long
2024-09-05T21:05:06.7763436Z    "C:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Sdk.win-x64\9.0.0-rc.2.24455.1\tools\emscripten\emcc.bat" @C:\Users\VSSADM~1\AppData\Local\Temp\emscripten_kudoq_7e.rsp.utf-8
2024-09-05T21:05:06.7764954Z emcc : error : subprocess 2/6 failed (returned 1)! (cmdline: "C:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Sdk.win-x64\9.0.0-rc.2.24455.1\tools\emscripten\emcc.bat" @C:\Users\VSSADM~1\AppData\Local\Temp\emscripten_cldyffve.rsp.utf-8) [D:\a\1\s\testconsole01\testconsole01.csproj]
2024-09-05T21:05:06.9615697Z emcc : error : response file not found: C:\Users\VSSADM~1\AppData\Local\Temp\emscripten_kudoq_7e.rsp.utf-8 [D:\a\1\s\testconsole01\testconsole01.csproj]
2024-09-05T21:05:06.9637328Z C:\Program Files\dotnet\packs\Microsoft.NET.Runtime.WebAssembly.Sdk\9.0.0-rc.2.24454.7\Sdk\BrowserWasmApp.targets(492,5): error MSB3073: The command "emcc "@C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Runtime.Mono.browser-wasm\9.0.0-rc.2.24454.7\runtimes\browser-wasm\native\src\emcc-default.rsp" -msimd128 "@C:\Program Files\dotnet\packs\Microsoft.NETCore.App.Runtime.Mono.browser-wasm\9.0.0-rc.2.24454.7\runtimes\browser-wasm\native\src\emcc-link.rsp" "@D:\a\1\s\testconsole01\obj\Release\net9.0\browser-wasm\wasm\for-publish\emcc-link.rsp"" exited with code 1. [D:\a\1\s\testconsole01\testconsole01.csproj]
2024-09-05T21:05:07.0594155Z 
2024-09-05T21:05:07.2899507Z ##[error]PowerShell exited with code '1'.
2024-09-05T21:05:07.3304326Z ##[section]Finishing: PowerShell

Repro in azure deovops:

pool:
  vmImage: 'windows-2022'

trigger:
  - main
  - release/*

stages:
- stage: Build
  jobs:
  - job: Build

    steps:
    - checkout: self
      clean: 'true'
      fetchDepth: 0
      
    - pwsh: |
        $DotNetRoot = "C:\Program Files\dotnet"
        Invoke-WebRequest "https://dot.net/v1/dotnet-install.ps1" -OutFile ".\installcli.ps1"
        & .\installcli.ps1 -c 9.0 -q daily -InstallDir $DotNetRoot

      condition: and(succeeded(), eq( variables['Agent.OS'], 'Windows_NT' ))
      displayName: 'Setup .NET (Windows)'

    - pwsh: |
        dotnet nuget add source -n "dotnet-public" https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public/nuget/v3/index.json
        dotnet nuget add source -n "dotnet-tools" https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-tools/nuget/v3/index.json
        dotnet nuget add source -n "dotnet-eng" https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-eng/nuget/v3/index.json
        dotnet nuget add source -n "dotnet-libraries" https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-libraries/nuget/v3/index.json
        dotnet nuget add source -n "dotnet-libraries-transport" https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-libraries-transport/nuget/v3/index.json
        dotnet nuget add source -n "dotnet9" https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet9/nuget/v3/index.json
        dotnet nuget add source -n "dotnet9-transport" https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet9-transport/nuget/v3/index.json
        dotnet nuget add source -n "dotnet10" https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet10/nuget/v3/index.json
        dotnet nuget add source -n "dotnet10-transport" https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet10-transport/nuget/v3/index.json

      displayName: Setup feeds

    - pwsh: |
        dotnet --version
        dotnet --list-sdks
        dotnet workload list
        dotnet workload install wasm-tools
        dotnet workload install wasm-experimental

      displayName: Setup Workloads

    - pwsh: dotnet new wasmconsole -o testconsole01 -f net9.0
    
    - pwsh: |
        cd testconsole01
        mkdir $env:TMP\emsdk
        dotnet publish -p:WasmCachePath=$env:TMP\emsdk -p:WasmBuildNative=true /bl:$(Build.ArtifactStagingDirectory)\msbuild.binlog


    - task: PublishPipelineArtifact@1
      condition: always()
      inputs:
        targetPath: $(Build.ArtifactStagingDirectory)
        artifactName: dotnet-sdk-binaries

Associated binlog: msbuild (15).zip

@ilonatommy
Copy link
Member Author

ilonatommy commented Sep 6, 2024

Thank you. Similarly, I can reproduce on azure, locally not. In both cases emsdk paths are same (C:\Program Files\dotnet\packs\Microsoft.NET.Runtime.Emscripten.3.1.56.Sdk.win-x64\9.0.0-rc.2.24455.1\tools). Comparing the response files for linking and compilation, I do not see any reason for such behavior. Long paths on Azure's Windows is enabled.
Repository for provided repro: https://github.com/ilonatommy/issue-103625.

ToDo:
can we add logging or try-catch to https://github.com/dotnet/emscripten/blob/38a38efa792ef422ff90a48258cbdf57ef552674/tools/shared.py#L130 ?

@jeromelaban
Copy link
Contributor

jeromelaban commented Sep 6, 2024

@ilonatommy The only notable difference that I could see is the fact that the sources and the workload are not on the same drive. This is why I chose to put the WasmCachePath is in TMP, not in IntermediateOutputPath, as it would otherwise fail for a different reason invoking the impossibility to generate relative paths.

@ilonatommy
Copy link
Member Author

Different drive does not help, I have a local project that produces 1:1 same emcc-link.rsp, including all the path contents - still no failure. Leaving WasmCachePath with default value does not change it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arch-wasm WebAssembly architecture area-Build-mono wasm-aot-test WebAssembly AOT Test
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants