Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backports for Julia 1.8.2 #46663

Merged
merged 34 commits into from
Sep 21, 2022
Merged

Backports for Julia 1.8.2 #46663

merged 34 commits into from
Sep 21, 2022

Conversation

KristofferC
Copy link
Sponsor Member

@KristofferC KristofferC commented Sep 7, 2022

Backported PRs:

Non-merged PRs with backport label:

mortenpi and others added 6 commits September 7, 2022 09:46
…ractPlatform)` (#46547)

This improvement prevents invalidations in binaryplatforms.jl when loading Static.jl.

(cherry picked from commit 25a07bf)
This prevents invalidations from loading Static.jl

(cherry picked from commit adc72d1)
This fixes invalidations from loading Static.jl

(cherry picked from commit 7537cb3)
@KristofferC KristofferC added the kind:release Release management and versioning. label Sep 7, 2022
…#46581)

Added in #43415, this was too aggressive for many cases. Unlike the
comment suggested, it is unneeded in many cases, so only do it when it
is expected to be maximally profitable.

Fixes #46492

```
julia> @time norm(C_212)
before 45.959497 seconds (81.85 M allocations: 6.976 GiB, 6.31% gc time, 100.00% compilation time)
after  15.781804 seconds (20.81 M allocations: 1.294 GiB, 6.32% gc time, 100.00% compilation time)
```
@ranocha
Copy link
Member

ranocha commented Sep 9, 2022

#46553, #46673 and #46661 are additional non-merged PRs with backport label.

@ufechner7
Copy link

Can this commit also be added: #46492 ?

…`JuliaIO/Tar.jl` repo (not the `master` branch of the `JuliaIO/Tar.jl` repo) (#46733)
@ranocha
Copy link
Member

ranocha commented Sep 14, 2022

#46732

aviatesk and others added 3 commits September 14, 2022 14:39
It turns out that previously we didn't concretize `invoke` callsite
correctly, as we didn't take into account whether the call being
concretized is `invoke`-d or not, e.g.:
```
julia> invoke_concretized2(a::Int) = a > 0 ? :int : nothing
invoke_concretized2 (generic function with 1 method)

julia> invoke_concretized2(a::Integer) = a > 0 ? :integer : nothing
invoke_concretized2 (generic function with 2 methods)

julia> let
           Base.Experimental.@force_compile
           Base.@invoke invoke_concretized2(42::Integer)
       end
:int # this should return `:integer` instead
```

This commit fixes that up by propagating information `invoke`-d callsite
to `concrete_eval_call`. Now we should be able to pass the following test cases:
```julia
invoke_concretized1(a::Int) = a > 0 ? :int : nothing
invoke_concretized1(a::Integer) = a > 0 ? "integer" : nothing
@test Base.infer_effects((Int,)) do a
    @invoke invoke_concretized1(a::Integer)
end |> Core.Compiler.is_foldable
@test Base.return_types() do
    @invoke invoke_concretized1(42::Integer)
end |> only === String

invoke_concretized2(a::Int) = a > 0 ? :int : nothing
invoke_concretized2(a::Integer) = a > 0 ? :integer : nothing
@test Base.infer_effects((Int,)) do a
    @invoke invoke_concretized2(a::Integer)
end |> Core.Compiler.is_foldable
@test let
    Base.Experimental.@force_compile
    @invoke invoke_concretized2(42::Integer)
end === :integer
```
For some reason `tab_unpack` gained an `@noinline` annotation as well as a conflicting
`@inline`. The lack of inlining was causing performance problems, so remove both of
those, since the function should be inlineable by default.
@fxcoudert
Copy link
Contributor

Could I ask that #45411 be backported to 1.8.2? Or at least, the parts that provide forward build compatibility with libgit2 >= 1.4.0, i.e., this part:

diff --git a/stdlib/LibGit2/src/consts.jl b/stdlib/LibGit2/src/consts.jl
index 2bc9edaf89..55887ebe2a 100644
--- a/stdlib/LibGit2/src/consts.jl
+++ b/stdlib/LibGit2/src/consts.jl
@@ -247,6 +247,11 @@ const RESET_HARD  = Cint(3) # MIXED plus changes in working tree discarded
                             REBASE_OPERATION_FIXUP  = Cint(4),
                             REBASE_OPERATION_EXEC   = Cint(5))
 
+# git_remote_redirect_t
+const GIT_REMOTE_REDIRECT_NONE    = Cint(0)
+const GIT_REMOTE_REDIRECT_INITIAL = Cint(1)
+const GIT_REMOTE_REDIRECT_ALL     = Cint(2)
+
 # fetch_prune
 const FETCH_PRUNE_UNSPECIFIED = Cint(0)
 const FETCH_PRUNE             = Cint(1)
diff --git a/stdlib/LibGit2/src/types.jl b/stdlib/LibGit2/src/types.jl
index 9ffcaa3646..98d938df65 100644
--- a/stdlib/LibGit2/src/types.jl
+++ b/stdlib/LibGit2/src/types.jl
@@ -343,6 +343,9 @@ The fields represent:
     @static if LibGit2.VERSION >= v"0.25.0"
         proxy_opts::ProxyOptions       = ProxyOptions()
     end
+    @static if LibGit2.VERSION >= v"1.4.0"
+        follow_redirects::Cint         = Consts.GIT_REMOTE_REDIRECT_INITIAL
+    end
     @static if LibGit2.VERSION >= v"0.24.0"
         custom_headers::StrArrayStruct = StrArrayStruct()
     end
@@ -674,6 +677,9 @@ The fields represent:
     @static if LibGit2.VERSION >= v"0.25.0"
         proxy_opts::ProxyOptions       = ProxyOptions()
     end
+    @static if LibGit2.VERSION >= v"1.4.0"
+        follow_redirects::Cint         = Consts.GIT_REMOTE_REDIRECT_INITIAL
+    end
     @static if LibGit2.VERSION >= v"0.24.0"
         custom_headers::StrArrayStruct = StrArrayStruct()
     end

Currently, julia 1.8.x fails to build against recent libgit2 >= 1.4.0. In Homebrew, we have to patch it as above, and linux distros (like archlinux) do it as well. It would be nicer to actually provide this for source build.

In the official builds, since the dependency is not actually upgraded, it will change nothing. But it will make unpatched source builds possible for us.

@KristofferC
Copy link
Sponsor Member Author

You can open a PR with that as a commit against this branch or just post a commit SHA and I will push it here.

ranocha and others added 7 commits September 16, 2022 11:15
This prevents some invalidations in `mightalias(A::AbstractArray, B::AbstractArray)`
in abstractarray.jl when loading Static.jl.

Here we specialize on the function instead of using map since
`broadcasting` returns a BitArray, `map` returns a Vector{Bool}.

(cherry picked from commit 31d4c22)
The way we were counting the number of bits was assigning a negative number
to `0`, which could lead to a negative total number of bits. Better to just
exit early in this case.
Also, the estimate was slightly off because we were counting the number of leading
zeros in the least significant limb, instead of the most significant.

(cherry picked from commit f7b4ebe)
* improve inferrability of `NamedTuple{names}(nt::NamedTuple) where {names}`
* improve inferrability of `recursive_dotcalls!(ex, args, i=1)`

(cherry picked from commit 5d9807d)
This fixes a few hundred invalidations when loading Static/jl/ArrayInterface.jl.

(cherry picked from commit 70bfa3f)
…_uuid::Base.UUID)` (#46661)

* improve type stability of `process_overrides(artifact_dict::Dict, pkg_uuid::Base.UUID)`

This fixes some invalidations when loading Static.jl

(cherry picked from commit b4af0e5)
* improve type stability of `sort!(v::AbstractVector, lo::Integer, hi::Integer, ::InsertionSortAlg, o::Ordering)`
* improve type stability of `fmt(buf, pos, arg, spec::Spec{T}) where {T <: Strings}`

This fixes some invalidations when loading Static.jl.

(cherry picked from commit 9b1780d)
- serialize bits values and QuoteNodes in IR encoding
  Using this instead of the `roots` array avoids scaling problems in
  functions with many constants.
- move roots created by codegen to a single global table, allowing
  reuse of egal values and keeping method roots lists shorter

(cherry picked from commit 425f6ff)
@fxcoudert
Copy link
Contributor

@KristofferC I have opened a PR at #46795

barucden and others added 4 commits September 16, 2022 11:36
The previous regex for credential helpers included even the following
credential:

```
[credential "helperselector"]
    selected = manager-core
```

which gets introduced by Git for Windows and fails our assumption about
what a credential helper is.

The commit also removes a test that mirrors what `credential_helpers`
does (otherwise, we would have to maintain the same regex at two places,
the definition of `credential_helpers` and the test case).

Fixes #45693

(cherry picked from commit ea74208)
This fixes some invalidations when loading ChainRulesCore.jl.

(cherry picked from commit e758982)
* Also merge var with unchanged bounds.
The current `env` might not be valid if the var's bounds get fixed by another Unions decision.
* Replace `v->var->lb` with `simple_meet`
* Remove the unneeded NUll check.

Co-Authored-By: Jameson Nash <vtjnash+github@gmail.com>
(cherry picked from commit 6557542)
vtjnash and others added 3 commits September 19, 2022 10:29
Includes several fixes:
JuliaLang/libuv@1b2d164...3f7038d
And should make read-eof notification slightly faster in some cases
(rather than waiting for close notification).

(cherry picked from commit fa986d9)
…46799)

Sometimes `Core.Compiler.findall(::Type, ::CachedMethodTable; limit::Int)`
is called with different `limit` setting (in particularity
`return_type_tfunc` calls it with `limit=-1`). The query should return
different results given different `limit` settings, so its cache should
also have different keys per different `limit` settings.

fix #46722
@maleadt
Copy link
Member

maleadt commented Sep 20, 2022

@nanosoldier runtests(ALL, vs = ":release-1.8", configuration = (buildflags=["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"],), vs_configuration=(buildflags=["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"],))

@maleadt
Copy link
Member

maleadt commented Sep 20, 2022

@nanosoldier runbenchmarks(ALL, vs=":release-1.8")

@nanosoldier
Copy link
Collaborator

Your package evaluation job has completed - possible new issues were detected. A full report can be found here.

@KristofferC
Copy link
Sponsor Member Author

@nanosoldier runtests(["FHIRClient", "Flamenco", "GAP", "HydrophoneCalibrations", "SwagUI"], vs = ":release-1.8", configuration = (buildflags = ["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"],), vs_configuration = (buildflags = ["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"],))

@nanosoldier
Copy link
Collaborator

Your package evaluation job has completed - no new issues were detected. A full report can be found here.

@maleadt
Copy link
Member

maleadt commented Sep 20, 2022

Those results cannot be right; something was off with my retry logic. I've also wiped the packages store once again because there were too many failures. Let's try again:

@nanosoldier runtests(ALL, vs = ":release-1.8", configuration = (buildflags=["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"],), vs_configuration=(buildflags=["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"],))

@nanosoldier
Copy link
Collaborator

Your job failed.

@maleadt
Copy link
Member

maleadt commented Sep 20, 2022

I presume the upload was too large again? cc @vtjnash

@maleadt
Copy link
Member

maleadt commented Sep 20, 2022

The most significant of these regressions seems real:

              _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.1 (2022-09-06)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using BenchmarkTools

julia> v = Vector{Nothing}(undef, 100000);

julia> @benchmark Vector{Nothing}($v)
BenchmarkTools.Trial: 10000 samples with 997 evaluations.
 Range (min … max):  19.789 ns …  1.056 μs  ┊ GC (min … max): 0.00% … 96.26%
 Time  (median):     21.294 ns              ┊ GC (median):    0.00%
 Time  (mean ± σ):   22.657 ns ± 25.521 ns  ┊ GC (mean ± σ):  3.47% ±  3.04%

    ▅▆▄▃▇█▇▅▅▃▂▁                            ▁▂▂▂▁▁            ▂
  ███████████████▇▇▅▅▅▄▄▁▁▁▁▁▄▁▁▁▁▄▃▁▁▄▅▆███████████▇▇▅▄▄▆▆▆▇ █
  19.8 ns      Histogram: log(frequency) by time      31.7 ns <

 Memory estimate: 48 bytes, allocs estimate: 1.
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.8.1 (2022-09-20)
 _/ |\__'_|_|_|\__'_|  |  backports-release-1.8/39ae07ca802 (fork: 355 commits, 215 days)
|__/                   |

julia> using BenchmarkTools
[ Info: Precompiling BenchmarkTools [6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf]
WARNING: method definition for dpeekbyte at /home/tim/Julia/depot/packages/Parsers/34hDN/src/utils.jl:199 declares type variable T but does not use it.

julia> v = Vector{Nothing}(undef, 100000);

julia> @benchmark Vector{Nothing}($v)
BenchmarkTools.Trial: 10000 samples with 995 evaluations.
 Range (min … max):  28.099 ns … 809.528 ns  ┊ GC (min … max): 0.00% … 94.47%
 Time  (median):     28.724 ns               ┊ GC (median):    0.00%
 Time  (mean ± σ):   30.064 ns ±  20.183 ns  ┊ GC (mean ± σ):  2.05% ±  2.96%

  ▃▇██▇▆▅▄▁                               ▁▁▂▂▂▁▁▁             ▂
  █████████▇▇▇▇▆▄▄▄▄▃▃▄▃▃▄▃▁▄▃▃▁▃▃▁▃▄▄▆▆████████████▇▆▄▄▅▆▅▇▆▆ █
  28.1 ns       Histogram: log(frequency) by time      39.2 ns <

 Memory estimate: 48 bytes, allocs estimate: 1.

@gbaraldi
Copy link
Member

gbaraldi commented Sep 20, 2022

Any idea where the extra time is coming from?

@nanosoldier
Copy link
Collaborator

Your package evaluation job has completed - possible new issues were detected. A full report can be found here.

@maleadt
Copy link
Member

maleadt commented Sep 21, 2022

Hmm, maybe this is a false alarm. Even on 1.8.1 that benchmark sometimes takes 19-20us, but in some sessions (i.e. it's consistent until you restart julia) it regresses to 28us.

@KristofferC
Copy link
Sponsor Member Author

@nanosoldier runtests(["ArgoData", "BayesianQuadrature", "Flamenco", "FluxTraining", "GeometryPrimitives", "HydrophoneCalibrations", "OceanBasins", "OptimKit", "RainFARM", "SimpleFWA"], vs = ":release-1.8", configuration = (buildflags = ["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"],), vs_configuration = (buildflags = ["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"],))

@nanosoldier
Copy link
Collaborator

Your package evaluation job has completed - no new issues were detected. A full report can be found here.

@KristofferC KristofferC merged commit 4e8ce61 into release-1.8 Sep 21, 2022
@KristofferC KristofferC deleted the backports-release-1.8 branch September 21, 2022 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:release Release management and versioning.
Projects
None yet
Development

Successfully merging this pull request may close these issues.