Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

release-1.8: Backports for 1.8-rc4 #46075

Merged
merged 45 commits into from
Aug 7, 2022
Merged

release-1.8: Backports for 1.8-rc4 #46075

merged 45 commits into from
Aug 7, 2022

Conversation

KristofferC
Copy link
Sponsor Member

@KristofferC KristofferC commented Jul 17, 2022 •
edited by IanButterworth
Loading

Backported PRs:

Need manual backport:

Contains multiple commits, manual intervention needed:

Non-merged PRs with backport label:

KristofferC and others added 2 commits July 17, 2022 09:02
@KristofferC
fix convert call in Artifacts.jl (#46040)
08f12b9 
(cherry picked from commit e96b19d)
@staticfloat @KristofferC
Allow BUILDKITE_BRANCH to provide branch name (#46053)
78ad6ff 
* Allow `BUILDKITE_BRANCH` to provide branch name

Our CI system checks commits out as a detached head, which breaks our
`Base.GIT_VERSION_INFO.branch` information.

* Fix typo

(cherry picked from commit d117975)
@KristofferC KristofferC added the kind:release Release management and versioning. label Jul 17, 2022
@IanButterworth @KristofferC
fixup time_imports news entry
7674873 
(cherry picked from commit dad1c7f)
@KristofferC KristofferC mentioned this pull request Jul 17, 2022
Closed
giordano and others added 3 commits July 17, 2022 19:06
@giordano @KristofferC
[LibCURL_jll] Update to v7.84 (#45967)
c8c6cc7 
(cherry picked from commit 9629154)
@vtjnash @KristofferC
fix Libc.rand and seed problems (#44432)
3dc152b 
Continuation from #43606

 - Replaces thread-unsafe function `rand` with `jl_rand`.
 - Fixes `_ad_hoc_entropy_source` fallback in Random.
 - Uses uv_random for more direct access to quality-randomness (usually
   a syscall rather than a file.)
 - Ensures Array{Bool} are valid when created from RandomDevice.

(cherry picked from commit b4bed71)
@JeffBezanson @KristofferC
give finalizers their own RNG state (#45212)
6a05297 
fixes #42752

(cherry picked from commit 85b895b)
maleadt and others added 15 commits July 19, 2022 14:23
@maleadt
Update LLVM to include additional patches. (#46091)
6ba1d23
@haampie @vchuravy
Unset NDEBUG in analyzers
2bb4aec 
(cherry picked from commit 2ad2e85)
@vchuravy
Don't segfault when running atexit before jl_threads_init
dcd915b 
(cherry picked from commit 3b77feb)
@staticfloat @vchuravy
Explicitly test for a failing exit code, do not include signals
47cd899 
We accidentally ignored some test failures because we only tested for `!success(p)`, which passes even if `p` segfaulted.

(cherry picked from commit 67a43c5)
@staticfloat @vchuravy
Don't call jl_errorf() when we're early in bringup
9bb9625 
When calling `jl_error()` or `jl_errorf()`, we must check to see if we
are so early in the bringup process that it is dangerous to attempt to
construct a backtrace because the data structures used to provide line
information are not properly setup.

This can be easily triggered by running:

```
julia -C invalid
```

On an `i686-linux-gnu` build, this will hit the "Invalid CPU Name"
branch in `jitlayers.cpp`, which calls `jl_errorf()`.  This in turn
calls `jl_throw()`, which will eventually call `jl_DI_for_fptr` as part
of the backtrace printing process, which fails as the object maps are
not fully initialized.  See the below `gdb` stacktrace for details:

```
$ gdb -batch -ex 'r' -ex 'bt' --args ./julia -C invalid
...
fatal: error thrown and no exception handler available.
ErrorException("Invalid CPU name "invalid".")

Thread 1 "julia" received signal SIGSEGV, Segmentation fault.
0xf75bd665 in std::_Rb_tree<unsigned int, std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo>, std::_Select1st<std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo> >, std::greater<unsigned int>, std::allocator<std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo> > >::lower_bound (__k=<optimized out>, this=0x248) at /usr/local/i686-linux-gnu/include/c++/9.1.0/bits/stl_tree.h:1277
1277    /usr/local/i686-linux-gnu/include/c++/9.1.0/bits/stl_tree.h: No such file or directory.
 #0  0xf75bd665 in std::_Rb_tree<unsigned int, std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo>, std::_Select1st<std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo> >, std::greater<unsigned int>, std::allocator<std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo> > >::lower_bound (__k=<optimized out>, this=0x248) at /usr/local/i686-linux-gnu/include/c++/9.1.0/bits/stl_tree.h:1277
 #1  std::map<unsigned int, JITDebugInfoRegistry::ObjectInfo, std::greater<unsigned int>, std::allocator<std::pair<unsigned int const, JITDebugInfoRegistry::ObjectInfo> > >::lower_bound (__x=<optimized out>, this=0x248) at /usr/local/i686-linux-gnu/include/c++/9.1.0/bits/stl_map.h:1258
 #2  jl_DI_for_fptr (fptr=4155049385, symsize=symsize@entry=0xffffcfa8, slide=slide@entry=0xffffcfa0, Section=Section@entry=0xffffcfb8, context=context@entry=0xffffcf94) at /cache/build/default-amdci5-4/julialang/julia-master/src/debuginfo.cpp:1181
 #3  0xf75c056a in jl_getFunctionInfo_impl (frames_out=0xffffd03c, pointer=4155049385, skipC=0, noInline=0) at /cache/build/default-amdci5-4/julialang/julia-master/src/debuginfo.cpp:1210
 #4  0xf7a6ca98 in jl_print_native_codeloc (ip=4155049385) at /cache/build/default-amdci5-4/julialang/julia-master/src/stackwalk.c:636
 #5  0xf7a6cd54 in jl_print_bt_entry_codeloc (bt_entry=0xf0798018) at /cache/build/default-amdci5-4/julialang/julia-master/src/stackwalk.c:657
 #6  jlbacktrace () at /cache/build/default-amdci5-4/julialang/julia-master/src/stackwalk.c:1090
 #7  0xf7a3cd2b in ijl_no_exc_handler (e=0xf0794010) at /cache/build/default-amdci5-4/julialang/julia-master/src/task.c:605
 #8  0xf7a3d10a in throw_internal (ct=ct@entry=0xf070c010, exception=<optimized out>, exception@entry=0xf0794010) at /cache/build/default-amdci5-4/julialang/julia-master/src/task.c:638
 #9  0xf7a3d330 in ijl_throw (e=0xf0794010) at /cache/build/default-amdci5-4/julialang/julia-master/src/task.c:654
 #10 0xf7a905aa in ijl_errorf (fmt=fmt@entry=0xf7647cd4 "Invalid CPU name \"%s\".") at /cache/build/default-amdci5-4/julialang/julia-master/src/rtutils.c:77
 #11 0xf75a4b22 in (anonymous namespace)::createTargetMachine () at /cache/build/default-amdci5-4/julialang/julia-master/src/jitlayers.cpp:823
 #12 JuliaOJIT::JuliaOJIT (this=<optimized out>) at /cache/build/default-amdci5-4/julialang/julia-master/src/jitlayers.cpp:1044
 #13 0xf7531793 in jl_init_llvm () at /cache/build/default-amdci5-4/julialang/julia-master/src/codegen.cpp:8585
 #14 0xf75318a8 in jl_init_codegen_impl () at /cache/build/default-amdci5-4/julialang/julia-master/src/codegen.cpp:8648
 #15 0xf7a51a52 in jl_restore_system_image_from_stream (f=<optimized out>) at /cache/build/default-amdci5-4/julialang/julia-master/src/staticdata.c:2131
 #16 0xf7a55c03 in ijl_restore_system_image_data (buf=0xe859c1c0 <jl_system_image_data> "8'\031\003", len=125161105) at /cache/build/default-amdci5-4/julialang/julia-master/src/staticdata.c:2184
 #17 0xf7a55cf9 in jl_load_sysimg_so () at /cache/build/default-amdci5-4/julialang/julia-master/src/staticdata.c:424
 #18 ijl_restore_system_image (fname=0x80a0900 "/build/bk_download/julia-d78fdad601/lib/julia/sys.so") at /cache/build/default-amdci5-4/julialang/julia-master/src/staticdata.c:2157
 #19 0xf7a3bdfc in _finish_julia_init (rel=rel@entry=JL_IMAGE_JULIA_HOME, ct=<optimized out>, ptls=<optimized out>) at /cache/build/default-amdci5-4/julialang/julia-master/src/init.c:741
 #20 0xf7a3c8ac in julia_init (rel=<optimized out>) at /cache/build/default-amdci5-4/julialang/julia-master/src/init.c:728
 #21 0xf7a7f61d in jl_repl_entrypoint (argc=<optimized out>, argv=0xffffddf4) at /cache/build/default-amdci5-4/julialang/julia-master/src/jlapi.c:705
 #22 0x080490a7 in main (argc=3, argv=0xffffddf4) at /cache/build/default-amdci5-4/julialang/julia-master/cli/loader_exe.c:59
```

To prevent this, we simply avoid calling `jl_errorf` this early in the
process, punting the problem to a later PR that can update guard
conditions within `jl_error*`.

(cherry picked from commit 21ab24e)
@JeffBezanson
more friendly/useful error for stack overflow in type inference (#44971)
6d7539b 
fixes #44852

(cherry picked from commit 401d578)
@oscardssmith
mark two_mul as consistent (#46022)
d6e8e5d 
should let the compiler prove that float64^int always gives the same result.

(cherry picked from commit fd70eab)
Export jl_is_foreign_type
ef18da7
@giordano
[nghttp2_jll] Upgrade to v1.48.0 (#45980)
54a6806 
* [nghttp2_jll] Upgrade to v1.48.0

* [nghttp2_jll] Fix test

(cherry picked from commit 707f59b)
@JeffBezanson
fix #45825, BitArray methods assuming 1-indexing of AbstractArray (#4…
9fa6b7f 
…5835)

(cherry picked from commit 820c08b)
@IanButterworth
remove examples of time_imports nesting given it was a bug (#46072)
bcac3c2 
(cherry picked from commit 29586ef)
@N5N3
Fix fast path for strides(::ReinterpretArray) with dense parent. (#…
169567d 
…46114)

(cherry picked from commit 017228a)
bump Pkg version to latest 1.8
bfc7e3e
@maleadt
Specialize tuple setindex to avoid ntuple-related performance regress…
1549064 
…ion. (#46050)

(cherry picked from commit d7c56ba)
@chriselrod
Use max(1, Sys.CPU_THREADS) BLAS threads for apple aarch64. (#46085)
f3bc6bf 
(cherry picked from commit 97df6db)
@vchuravy
Copy link
Member

@aviatesk Failure on windows looks relevant:

Error in testset math:
Test Failed at C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\test\math.jl:1426
  Expression: Core.Compiler.is_foldable(Base.infer_effects(^, (T, Int)))
Error in testset math:
Test Failed at C:\buildbot\worker-tabularasa\tester_win64\build\share\julia\test\math.jl:1447
  Expression: Core.Compiler.is_foldable(Base.infer_effects(^, (Float32, Int)))

@aviatesk
Copy link
Sponsor Member

Thanks, I backported #45299 to fix it. Now I'm a bit worried that the backports ends up being fairly big...

aviatesk and others added 7 commits July 29, 2022 06:21
@aviatesk
inference: fix #46207, make sure we never form nested Conditional (#…
9b38226 
…46208)
@IanButterworth
Fix timev compilation time tracking and add tests (#46100)
c02b4e1 
(cherry picked from commit f3c2227)
@IanButterworth
update Pkg branch & commit
20ef7d9
@vtjnash @vchuravy
Prefix Float16 intrinsics
a9bf474 
(cherry picked from commit f651866)
@vchuravy
Define aliases to FP16 crt in the OJIT
71d062c 
(cherry picked from commit ff36015)
@vchuravy @maleadt
Emit aliases into the system image
7ff795c 
- Put the interposer in llvm.compiler.used.
- Injecting the aliases after optimization:
  Our multiversioning pass interacts badly with the llvm.compiler.used gvar.
- run `postopt` passes independent of optimization passes.

(cherry picked from commit 3407fb3)

Co-authored-by: Tim Besard <tim.besard@gmail.com>
Co-authored-by: Valentin Churavy <v.churavy@gmail.com>
@vchuravy
Merge pull request #46110 from JuliaLang/vc/fp16_bb
4983135 
Backport "Emit aliases for FP16 conversion routines" (#45649) to 1.8
@KristofferC
Copy link
Sponsor Member Author

@nanosoldier runtests(ALL, vs = ":release-1.8", buildflags=["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"], vs_buildflags=["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"])

@nanosoldier
Copy link
Collaborator

Something went wrong when running your job:

Base.IOError("unlink(\"/tmp/jl_c8vjoA\"): no such file or directory (ENOENT)", -2)

cc @maleadt

@KristofferC
Copy link
Sponsor Member Author

@nanosoldier runtests(ALL, vs = ":release-1.8", buildflags=["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"], vs_buildflags=["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"])

@maleadt
Copy link
Member

maleadt commented Aug 3, 2022 •
edited
Loading

I'm actually still doing some work on PkgEval (currently trying-out JuliaCI/PkgEval.jl#108, if that works I think we could significantly reduce the time it takes to run all tests). However, I could as well do those tests on a useful comparison. It's possible the results might be off though, and I might have to run it again if another bug surfaces.

@maleadt
Copy link
Member

maleadt commented Aug 4, 2022

@nanosoldier runtests(ALL, vs = ":release-1.8", buildflags=["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"], vs_buildflags=["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"])

@maleadt
Copy link
Member

maleadt commented Aug 5, 2022

Pushed it manually: https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_hash/4983135_vs_843b322/report.html

aviatesk and others added 3 commits August 6, 2022 18:51
@aviatesk @KristofferC
improve cat inferrability (#45028)
4b5ea08 
Make `cat` inferrable even if its arguments are not fully constant:
```julia
julia> r = rand(Float32, 56, 56, 64, 1);

julia> f(r) = cat(r, r, dims=(3,))
f (generic function with 1 method)

julia> @inferred f(r);

julia> last(@code_typed f(r))
Array{Float32, 4}
```

After descending into its call graph, I found that constant propagation
is prohibited at `cat_t(::Type{T}, X...; dims)` due to the method instance
heuristic, i.e. its body is considered to be too complex for successful
inlining although it's explicitly annotated as `@inline`.
But for this case, the constant propagation is greatly helpful both for
abstract interpretation and optimization since it can improve the return
type inference.

Since it is not an easy task to improve the method instance heuristic,
which is our primary logic for constant propagation, this commit does
a quick fix by helping inference with the `@constprop` annotation.

There is another issue that currently there is no good way to properly
apply `@constprop`/`@inline` effects to a keyword function (as a note,
this is a general issue of macro annotations on a method definition).
So this commit also changes some internal helper functions of `cat`
so that now they are not keyword ones: the changes are also necessary
for the `@inline` annotation on `cat_t` to be effective to trick
the method instance heuristic.

(cherry picked from commit 65b9be4)
@N5N3 @KristofferC
copyto! fix for BitArray/AbstractArray, fixes #25968 (#46161)
5cd8444 
1. map `copyto!(::BitArray, n1, ::BitArray, n2, l)` to `Base.unsafe_copyto!`
2. add missing unaliasing in `copyto!` for `AbstractArray`

(cherry picked from commit 0ea2b2d)
@vtjnash @KristofferC
dump: ensure we generate the whole backedge graph (#46171)
2415a14 
The edge-restore algorithm here is pretty bad now, but this should hopefully fix #45444

(cherry picked from commit 6b51780)
@KristofferC
Copy link
Sponsor Member Author

@nanosoldier runtests(["Balloons", "COPT", "CellListMap", "CitableImage", "CountdownNumbers", "CrystalInfoFramework", "DataDrivenDiffEq", "DiffEqDevTools", "FlameGraphs", "Folds", "GPUCompiler", "GasChromatographySimulator", "InformationGeometry", "MatrixPencils", "NODAL", "Optimization", "OteraEngine", "PDENLPModels", "PoreMatMod", "PowerFlowData", "ProfileView", "ProxSDP", "PsychomotorVigilanceTask", "QuadraticToBinary", "RadiativeTransfer", "RetroCap", "SimpleFWA", "Singular", "StochasticPrograms", "StrBase", "StressTest", "SymbolicUtils", "TensorBoardLogger", "TensorToolbox", "TransitionalMCMC", "UrlDownload", "YAAD", "vSmartMOM"], vs = ":release-1.8", buildflags=["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"], vs_buildflags=["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"])

@maleadt
Copy link
Member

maleadt commented Aug 7, 2022

Your package evaluation job has completed - possible new issues were detected. A full report can be found here.

@KristofferC
Copy link
Sponsor Member Author

@nanosoldier runtests(["COPT", "CountdownNumbers", "FlameGraphs", "Folds", "GPUCompiler", "PowerFlowData", "RetroCap", "StrBase", "StressTest", "SymbolicUtils", "TensorBoardLogger", "vSmartMOM"], vs = ":release-1.8", buildflags=["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"], vs_buildflags=["LLVM_ASSERTIONS=1", "FORCE_ASSERTIONS=1"])

@maleadt
Copy link
Member

maleadt commented Aug 7, 2022

Your package evaluation job has completed - possible new issues were detected. A full report can be found here.

@KristofferC KristofferC mentioned this pull request Aug 7, 2022
Closed
@KristofferC KristofferC merged commit f32a5c6 into release-1.8 Aug 7, 2022
@KristofferC KristofferC deleted the backports-release-1.8 branch August 7, 2022 19:34
@maleadt maleadt mentioned this pull request Aug 8, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
kind:release Release management and versioning.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.