Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

External Pkg always recompiles upon loading #54940

Closed
KristofferC opened this issue Jun 26, 2024 · 12 comments · Fixed by #55066
Closed

External Pkg always recompiles upon loading #54940

KristofferC opened this issue Jun 26, 2024 · 12 comments · Fixed by #55066
Labels
compiler:precompilation Precompilation of modules domain:packages Package management and loading
Milestone
1.12

Comments

@KristofferC
Copy link
Sponsor Member

While being in a Pkg.jl repo:

❯ julia +nightly --project -q

julia> using Pkg
[ Info: Precompiling Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f] (cache misses: wrong source (1), mismatched flags (1))

❯ julia +nightly --project -q
julia> using Pkg
[ Info: Precompiling Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f] (cache misses: for different buildid (2), mismatched flags (2))

❯ julia +nightly --project -q
julia> using Pkg
[ Info: Precompiling Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f] (cache misses: for different buildid (2), mismatched flags (2))

The two original misses are from the bundled cache files so those are expected. This is also a bit weird:

julia> pkg = Base.identify_package("Pkg")
Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f]

julia> Base.isprecompiled(pkg)
true

julia> using Pkg
[ Info: Precompiling Pkg [44cfe95a-1eb2-52ea-b672-e2afdf69b78f] (cache misses: for different buildid (2), mismatched flags (2))
@KristofferC KristofferC added domain:packages Package management and loading compiler:precompilation Precompilation of modules labels Jun 26, 2024
@vchuravy
Copy link
Member

What does JULIA_DEBUG=loading say?

@KristofferC
Copy link
Sponsor Member Author 

┌ Debug: Rejecting cache file /home/kc/.julia/compiled/v1.12/Pkg/tUTdb_sLFSB.ji because required dependency Base.PkgId(Base.UUID("2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"), "Base64") failed to load from cache file for /home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/Base64/D7K0n_Ip9tx.ji.
│   exception = Error reading package image file.

weird..

@KristofferC
Copy link
Sponsor Member Author

I am also surprised this doesn't print anything at all for Base64:

julia> ENV["JULIA_DEBUG"] = "loading"
"loading"

julia> using Base64

julia> using Example
┌ Debug: Loading object cache file /home/kc/.julia/compiled/v1.12/Example/lLvWP_sLFSB.so for Example [7876af07-990d-54b4-ab0e-23690620f79a]
└ @ Base loading.jl:1227

@KristofferC
Copy link
Sponsor Member Author

I had some thought that the nightly build juliaup uses was corrupt somehow but building a local Julia on master it gives the exact same behavior.

@IanButterworth
Copy link
Sponsor Member

IanButterworth commented Jun 26, 2024 •
edited
Loading

Base64 is an indirect dep of REPL, so it's already loaded?

@KristofferC
Copy link
Sponsor Member Author

This method of loading Pkg seems to work:

❯ julia +nightly --project -iqe 'using REPL; using Pkg'
julia> pkgdir(Pkg)
"/home/kc/JuliaPkgs/Pkg.jl"

@KristofferC
Copy link
Sponsor Member Author

KristofferC commented Jun 26, 2024 •
edited
Loading

Base64 is an indirect dep of REPL, so it's already loaded?

Okay, that makes sense, I looked in Base.loaded_modules and it wasn't there so I assumed it wasn't loaded already:

julia> Base.loaded_modules
Dict{Base.PkgId, Module} with 12 entries:
  PkgId(UUID("4536629a-c528-5b80-bd46-f80d51c5b363"), "OpenBLAS_jll")          => OpenBLAS_jll
  PkgId(UUID("8f399da3-3557-5675-b5ff-fb832c97cbdb"), "Libdl")                 => Libdl
  PkgId(UUID("ea8e919c-243c-51af-8825-aaa63cd721ce"), "SHA")                   => SHA
  PkgId(UUID("7b1f6079-737a-58dc-b8bc-7a2ca5c1b5ee"), "FileWatching")          => FileWatching
  PkgId(UUID("56f22d72-fd6d-98f1-02f0-08ddc0907c33"), "Artifacts")             => Artifacts
  PkgId(UUID("8e850b90-86db-534c-a0d3-1478176c7d93"), "libblastrampoline_jll") => libblastrampoline_jll
  PkgId(nothing, "Main")                                                       => Main
  PkgId(nothing, "Core")                                                       => Core
  PkgId(UUID("37e2e46d-f89d-539d-b4ee-838fcccc9c8e"), "LinearAlgebra")         => LinearAlgebra
  PkgId(nothing, "Base")                                                       => Base
  PkgId(UUID("9a3f8284-a2c9-5f02-9a11-845980a1fd5c"), "Random")                => Random
  PkgId(UUID("6462fe0b-24de-5631-8697-dd941f90decc"), "Sockets")               => Sockets

so I guess things that are loaded as part of require_stdlib are now invisible but when you do using on them they grab the invisible module and makes it visible?

But that shouldn't make Pkg have to compile over and over due to error when reading the Base64 pkgimage file I guess.

@fatteneder
Copy link
Member 

┌ Debug: Rejecting cache file /home/kc/.julia/compiled/v1.12/Pkg/tUTdb_sLFSB.ji because required dependency Base.PkgId(Base.UUID("2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"), "Base64") failed to load from cache file for /home/kc/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/Base64/D7K0n_Ip9tx.ji.
│   exception = Error reading package image file.

weird..

I have seen this too when trying to debug JuliaPy/PythonCall.jl#513.
There @kshyatt nailed down the problem to 71fa11f.

@KristofferC
Copy link
Sponsor Member Author

There @kshyatt nailed down the problem to 71fa11f.

Same thing happens when I revert that commit.

@kshyatt
Copy link
Contributor

kshyatt commented Jun 26, 2024 via email

@KristofferC
Copy link
Sponsor Member Author

Yes, I think they are unrelated.

@KristofferC
Copy link
Sponsor Member Author

Same happens with OhMyREPL on nightly:

┌ Debug: Loading object cache file /Users/kristoffercarlsson/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/StyledStrings/UcVoM_44pG1.dylib for StyledStrings [f489334b-da3d-4c2e-b8f0-e476e12c162b]
└ @ Base loading.jl:1227
┌ Debug: Rejecting cache file /Users/kristoffercarlsson/.julia/compiled/v1.12/OhMyREPL/08e1i_J2Ppx.ji because required dependency Base.PkgId(Base.UUID("f489334b-da3d-4c2e-b8f0-e476e12c162b"), "StyledStrings") failed to load from cache file for /Users/kristoffercarlsson/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/StyledStrings/UcVoM_44pG1.ji.
│   exception = Error reading package image file.
└ @ Base loading.jl:1962
┌ Debug: Loading object cache file /Users/kristoffercarlsson/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/StyledStrings/UcVoM_44pG1.dylib for StyledStrings [f489334b-da3d-4c2e-b8f0-e476e12c162b]
└ @ Base loading.jl:1227
┌ Debug: Rejecting cache file /Users/kristoffercarlsson/.julia/compiled/v1.12/OhMyREPL/08e1i_J2Ppx.ji because required dependency Base.PkgId(Base.UUID("f489334b-da3d-4c2e-b8f0-e476e12c162b"), "StyledStrings") failed to load from cache file for /Users/kristoffercarlsson/.julia/juliaup/julia-nightly/share/julia/compiled/v1.12/StyledStrings/UcVoM_44pG1.ji.
│   exception = Error reading package image file.

@KristofferC KristofferC added this to the 1.12 milestone Jun 29, 2024
This was referenced Jun 29, 2024
Merged
Merged
@KristofferC KristofferC mentioned this issue Jul 8, 2024
Closed
vtjnash added a commit that referenced this issue Jul 8, 2024
@vtjnash
fix loading of repeated/concurrent modules
a0c1326 
More followup to fix issues with require. There was an accidental
variable reuse (build_id) that caused it to be unable to load cache
files in many cases. There was also missing check for a dependency
already being loaded, resulting in trying to load it twice. Finally, the
start_loading code may drop the require_lock, but the surrounding code
was not prepared for that. Now integrate the necessary checks into
start_loading, instead of needing to duplicate them before and
afterwards.

Fixes #53983
Fixes #54940
Closes #55064
@vtjnash vtjnash mentioned this issue Jul 8, 2024
Merged
vtjnash added a commit that referenced this issue Jul 8, 2024
@vtjnash
fix loading of repeated/concurrent modules
9460d5c 
More followup to fix issues with require. There was an accidental
variable reuse (build_id) that caused it to be unable to load cache
files in many cases. There was also missing check for a dependency
already being loaded, resulting in trying to load it twice. Finally, the
start_loading code may drop the require_lock, but the surrounding code
was not prepared for that. Now integrate the necessary checks into
start_loading, instead of needing to duplicate them before and
afterwards.

Fixes #53983
Fixes #54940
Closes #55064
vtjnash added a commit that referenced this issue Jul 8, 2024
@vtjnash
fix loading of repeated/concurrent modules
7b70775 
More followup to fix issues with require. There was an accidental
variable reuse (build_id) that caused it to be unable to load cache
files in many cases. There was also missing check for a dependency
already being loaded, resulting in trying to load it twice. Finally, the
start_loading code may drop the require_lock, but the surrounding code
was not prepared for that. Now integrate the necessary checks into
start_loading, instead of needing to duplicate them before and
afterwards.

Fixes #53983
Fixes #54940
Closes #55064
KristofferC pushed a commit that referenced this issue Jul 23, 2024
@vtjnash @KristofferC
fix loading of repeated/concurrent modules (#55066)
d4f9808 
More followup to fix issues with require. There was an accidental
variable reuse (build_id) that caused it to be unable to load cache
files in many cases. There was also missing check for a dependency
already being loaded, resulting in trying to load it twice. Finally, the
start_loading code may drop the require_lock, but the surrounding code
was not prepared for that. Now integrate the necessary checks into
start_loading, instead of needing to duplicate them before and
afterwards.

Fixes #53983
Fixes #54940
Closes #55064

(cherry picked from commit fba928d)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
compiler:precompilation Precompilation of modules domain:packages Package management and loading
Projects
None yet
Milestone
1.12
5 participants
@vchuravy @kshyatt @KristofferC @IanButterworth @fatteneder