Loading errors/warnings

[charleskawczynski]

There seem to be a lot of errors and warnings when running ClimaAtmos on the clima A100:

Precompiling ClimaAtmos
  ✓ ClimaCore
  ✓ ClimaCore → KrylovExt
  ✓ ClimaCoreTempestRemap
  ✓ ClimaDiagnostics
CUDAUtilsExt Waiting for background task / IO / timer.
[pid 2213019] waiting for IO to finish:
 Handle type        uv_handle_t->data
 fs_event           0x1d976c0->0x7fad3e15ef80
 timer              0x26e9d60->0x7fad3e15efb0
This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.
  ✓ ClimaUtilities → CUDAUtilsExt
  ✓ ClimaUtilities → DataHandlingExt
  ✓ ClimaUtilities → TimeVaryingInputsExt
  ✓ ClimaUtilities → InterpolationsRegridderExt
  ✓ ClimaUtilities → SpaceVaryingInputsExt
  ✓ ClimaUtilities → TempestRegridderExt
        Info Given ClimaAtmos was explicitly requested, output will be shown live 
┌ Warning: CUDA runtime library libcublasLt.so.12 was loaded from a system path. This may cause errors.
│ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
└ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
┌ Warning: CUDA runtime library libnvJitLink.so.12 was loaded from a system path. This may cause errors.
│ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
└ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
┌ Warning: CUDA runtime library libcusparse.so.12 was loaded from a system path. This may cause errors.
│ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
└ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
  ✓ ClimaAtmos
  11 dependencies successfully precompiled in 77 seconds. 415 already precompiled.
  11 dependencies had output during precompilation:
┌ ClimaAtmos
│  [Output was shown above]
└  
┌ ClimaUtilities → CUDAUtilsExt
│  ┌ Warning: CUDA runtime library libcublasLt.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libnvJitLink.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libcusparse.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libcublasLt.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libnvJitLink.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libcusparse.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: Module CUDAUtilsExt with build ID ffffffff-ffff-ffff-0027-227933d255f0 is missing from the cache.
│  │ This may mean CUDAUtilsExt [44558d62-efde-57ee-a437-0474b5fb1348] does not support precompilation but is imported by a module that does.
│  └ @ Base loading.jl:1948
│  ┌ Error: Error during loading of extension CUDAUtilsExt of ClimaUtilities, use `Base.retry_load_extensions()` to retry.
│  │   exception =
│  │    1-element ExceptionStack:
│  │    Declaring __precompile__(false) is not allowed in files that are being precompiled.
│  │    Stacktrace:
│  │      [1] _require(pkg::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:1952
│  │      [2] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│  │        @ Base ./loading.jl:1812
│  │      [3] #invoke_in_world#3
│  │        @ ./essentials.jl:926 [inlined]
│  │      [4] invoke_in_world
│  │        @ ./essentials.jl:923 [inlined]
│  │      [5] _require_prelocked
│  │        @ ./loading.jl:1803 [inlined]
│  │      [6] _require_prelocked
│  │        @ ./loading.jl:1802 [inlined]
│  │      [7] run_extension_callbacks(extid::Base.ExtensionId)
│  │        @ Base ./loading.jl:1295
│  │      [8] run_extension_callbacks(pkgid::Base.PkgId)
│  │        @ Base ./loading.jl:1330
│  │      [9] run_package_callbacks(modkey::Base.PkgId)
│  │        @ Base ./loading.jl:1164
│  │     [10] __require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:1819
│  │     [11] #invoke_in_world#3
│  │        @ ./essentials.jl:926 [inlined]
│  │     [12] invoke_in_world
│  │        @ ./essentials.jl:923 [inlined]
│  │     [13] _require_prelocked(uuidkey::Base.PkgId, env::String)
│  │        @ Base ./loading.jl:1803
│  │     [14] macro expansion
│  │        @ ./loading.jl:1790 [inlined]
│  │     [15] macro expansion
│  │        @ ./lock.jl:267 [inlined]
│  │     [16] __require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:1753
│  │     [17] #invoke_in_world#3
│  │        @ ./essentials.jl:926 [inlined]
│  │     [18] invoke_in_world
│  │        @ ./essentials.jl:923 [inlined]
│  │     [19] require(into::Module, mod::Symbol)
│  │        @ Base ./loading.jl:1746
│  │     [20] include
│  │        @ ./Base.jl:495 [inlined]
│  │     [21] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
│  │        @ Base ./loading.jl:2222
│  │     [22] top-level scope
│  │        @ stdin:3
│  │     [23] eval
│  │        @ ./boot.jl:385 [inlined]
│  │     [24] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
│  │        @ Base ./loading.jl:2076
│  │     [25] include_string
│  │        @ ./loading.jl:2086 [inlined]
│  │     [26] exec_options(opts::Base.JLOptions)
│  │        @ Base ./client.jl:316
│  │     [27] _start()
│  │        @ Base ./client.jl:552
│  └ @ Base loading.jl:1301
│  
│  [pid 2213019] waiting for IO to finish:
│   Handle type        uv_handle_t->data
│   fs_event           0x1d976c0->0x7fad3e15ef80
│   timer              0x26e9d60->0x7fad3e15efb0
│  This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.
└  
┌ ClimaDiagnostics
│  ┌ Warning: CUDA runtime library libcublasLt.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libnvJitLink.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libcusparse.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
└  
┌ ClimaUtilities → SpaceVaryingInputsExt
│  ┌ Warning: CUDA runtime library libcublasLt.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libnvJitLink.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libcusparse.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
└  
┌ ClimaCoreTempestRemap
│  ┌ Warning: CUDA runtime library libcublasLt.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libnvJitLink.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libcusparse.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
└  
┌ ClimaCore → KrylovExt
│  ┌ Warning: CUDA runtime library libcublasLt.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libnvJitLink.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libcusparse.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libcublasLt.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libnvJitLink.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libcusparse.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
└  
┌ ClimaCore
│  ┌ Warning: CUDA runtime library libcublasLt.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libnvJitLink.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libcusparse.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
└  
┌ ClimaUtilities → InterpolationsRegridderExt
│  ┌ Warning: CUDA runtime library libcublasLt.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libnvJitLink.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libcusparse.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
└  
┌ ClimaUtilities → TempestRegridderExt
│  ┌ Warning: CUDA runtime library libcublasLt.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libnvJitLink.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libcusparse.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
└  
┌ ClimaUtilities → TimeVaryingInputsExt
│  ┌ Warning: CUDA runtime library libcublasLt.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libnvJitLink.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libcusparse.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
└  
┌ ClimaUtilities → DataHandlingExt
│  ┌ Warning: CUDA runtime library libcublasLt.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libnvJitLink.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
│  ┌ Warning: CUDA runtime library libcusparse.so.12 was loaded from a system path. This may cause errors.
│  │ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
│  └ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
└  
┌ Warning: CUDA runtime library libcublasLt.so.12 was loaded from a system path. This may cause errors.
│ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
└ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
┌ Warning: CUDA runtime library libnvJitLink.so.12 was loaded from a system path. This may cause errors.
│ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
└ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187
┌ Warning: CUDA runtime library libcusparse.so.12 was loaded from a system path. This may cause errors.
│ Ensure that you have not set the LD_LIBRARY_PATH environment variable, or that it does not contain paths to CUDA libraries.
└ @ CUDA ~/.julia/packages/CUDA/jdJ7Z/src/initialization.jl:187

[charleskawczynski]

cc @Sbozzolo

[Sbozzolo]

Even if the first error points to ClimaUtilities, this has nothing to do with ClimaUtilities (specifically). It might be a CUDA misconfiguration, but

CUDA runtime library libcublasLt.so.12 was loaded from a system path

Isn't this what we want? We want to use the native CUDA we have on the machine. I am not very familiar with how the CUDA runtime should be set for optimal performance

[Sbozzolo]

Can you also provide a way to reproduce?

I don't get these errors and I don't see them on buildkite

[charleskawczynski]

I see them every time I run on the clima node in a fresh session. Also, it's in the ClimaAtmos GHA CI: https://github.com/CliMA/ClimaAtmos.jl/actions/runs/9143626464/job/25140532009#step:6:995.

[Sbozzolo]

What error are you referring to?

The Error: Error during loading of extension TimeVaryingInputs0DExt of ClimaUtilities, use Base.retry_load_extensions() to retry. is a Julia upstream problem with precompilation and extensions, eg
JuliaLang/julia#53081
JuliaLang/julia#52511

(I think) it was fixed only two weeks ago: JuliaLang/julia#53972

[Sbozzolo]

CUDA is no longer part of the test environment and extensions in ClimaUtilities have been reordered to avoid triggering the error, so I think we can close this.