pidlock cache file precompilation

[IanButterworth]

This avoids multiple processes entering precompile cache file generation for the same package at the same time, allowing the first to enter and the others will wait to use the cache file that the first generated.

Terminal 1

julia> @time using Revise
[ Info: Precompiling Revise [295af30f-e4ad-537b-8983-00126c2a3abe]
 18.165747 seconds (321.97 k allocations: 19.684 MiB)

Terminal 2 (started just after above)

julia> @time using Revise
[ Info: Waiting for another process to precompile Revise [295af30f-e4ad-537b-8983-00126c2a3abe]
 17.033804 seconds (325.99 k allocations: 19.775 MiB, 0.07% compilation time: 100% of which was recompilation)

julia> 

Perhaps the 2nd process should inform the user that it's waiting for another process to finish. Implemented

Making this specific to the package source was relatively straightforward. Making it specific to the resulting cache file path seems basically impossible because of the way the cache file name is generated.

[KristofferC]

Perhaps the 2nd process should inform the user that it's waiting for another process to finish.

Was going to say this.

[IanButterworth]

Added

julia> using Revise
[ Info: Waiting for another process to precompile Revise [295af30f-e4ad-537b-8983-00126c2a3abe]

julia> 

[IanButterworth]

Maybe this should accept a function to print however wanted, as showing a @info log might be too opinionated?

[IanButterworth]

I was hoping this would mean one could

julia> @async Pkg.precompile(io=devnull)

julia> using Foo

where the environment contains packages bigger than Foo, for instance, but because Pkg.precompile precompiles depth first the using Foo could be first to the pidlock. Achieving that would require Pkg.precompile to take the pidlock early

[fredrikekre]

Would it not make sense to have a trymkpidlock instead, and handle the calling of f_waited on the outside?

[IanButterworth]

I think that would require knowing which file you need to wait for. The path of the cachefile isn't determinable before spawning off the child process, AFAICT

[IanButterworth]

Oh, I see what you mean

function cachefile_lock(f_first::Function, f_waited::Function, pkg::Base.PkgId, path::String)
    pidfile = string(path, ".pidlock")
    if trymkpidlock(f_first, pidfile) == false
        mkpidlock(f_waited, pidfile; wait_msg = "Waiting for another process to precompile $pkg")
    end
end

[IanButterworth]

No urgency but I'm going to wait for Jameson's review/opinion before working more on this

[vtjnash]

This seems like a great direction! Should we schedule a call to review details?

[IanButterworth]

@vtjnash I've tried to rewrite this to follow what we discussed

[IanButterworth]

@vtjnash this seems to be in a working state now.

For aligning Pkg.precompile I'm thinking it would use Base.maybe_cachefile_lock and two options come to mind:

1. Allow Pkg.precompile to enter the parallel phase and lock at the dependency level. Maybe even add some text beside the package with the spinner indicating it's being precompiled elsewhere
2. Prevent entering the parallel phase until all dependencies are not being precompiled elsewhere (needs some thought to not be racy)

[IanButterworth]

Bump @vtjnash

[vtjnash]

I don't love the trymkpidlock API of return Union{Bool,Any} or the naming of PidlockedError, but overall seems fine

[vchuravy]

So in JuliaLang/Pkg.jl#3458 we found it to be possible for a fork-bomb to occur.
That makes me wonder how this will handle accidental recursion. Will it now just hang since the parent obtains the pid-lock, then shells out, and the child hangs?

[vchuravy]

We could maybe even try to detect this specific deadlock scenario by calling getppid https://linux.die.net/man/3/getppid

The other question is how robust is this against cancellation, what happens on a killall -9 julia

[IanButterworth]

I'll add to NEWS later as this just manages non-interactive code load precompilation currently, needs a Pkg.precompile update