Stack overflow in type inference… julia 1.9.0-DEV

[ghjwp7]

edit: See program toverflow7.jl and its discussion in a later comment. toverflow7 is 28 lines long instead of the 140 of toverflow5.jl. The runs-ok version now is toverflow8, obtained by commands A=toverflow7;B=toverflow8;sed -e 's/if 1<2/if 1>2/' $A.jl > $B.jl; chmod u+x $B.jl. Program toverflow5.jl appears below. Program toverflow3.jl is made from toverflow5.jl by the command sed -e 's/if 1<2/if 1>2/' toverflow5.jl > toverflow3.jl. These programs illustrate that at the moment, seemingly-minor changes (using one print instead of two, writing to file instead of stdout) trigger a few error messages for Julia 1.9.0-DEV, or thousands of traceback and error messages for Julia 1.7.2, which is disconcerting.

This may be related to problems reported in issues type inference stackoverflow #43050 and StackOverflow Type Inference Error #44852. I also posted about this in discourse.julialang.org at Stack overflow in type inference… julia 1.9.0-DEV.

toverflow3 runs ok via Julia 1.7.2 (which is julia on my system, see versioninfo() at end of post) and via Julia 1.9.0-DEV.485 (which is julia-latest, ditto), writing 1 line to stdout and 2 lines to outP00.

Command julia-latest toverflow5.jl produces 4 identical 2-line error-message sets and then produces expected results. The message sets are like:

Internal error: stack overflow in type inference of (::toverflow.var"#findMore#7"{Array{Array{Float64, 1}, 1}, Array{Tuple, 1}, Array{Array{Float64, 1}, 1}, Array{Int32, 1}, Array{Int32, 1}, Array{Float64, 1}, Int64, Float64})(Int64).
This might be caused by recursion over very long tuples or argument lists.

Running 1.7.2, the command julia toverflow5.jl produces four similar groups of 19028 error lines each. For example, grep -n 'Internal error' over-script-1.7.2b shows:

3:Internal error: encountered unexpected error in runtime:
19031:Internal error: encountered unexpected error in runtime:
38059:Internal error: encountered unexpected error in runtime:
57087:Internal error: encountered unexpected error in runtime:

where over-script-1.7.2b is result of running toverflow5 within script over-script-1.7.2b.

Note, the difference between toverflow3 and toverflow5 is that the former uses two printfmt calls to print the same things that toverflow5 uses one printfmt for. The single-character difference between the files (as shown in the following output from diff toverflow3.jl toverflow5.jl) makes an if-test false for toverflow3 and true for toverflow5. The false case of the if contains 2 x 8-argument printfmts while the true case has one 14-argument printfmt.

58c58
<     if 1>2           # sed changes < to > for other version
---
>     if 1<2           # sed changes < to > for other version

Note, if the output to outP00 is instead written to stdout (ie, if outP00 isn't opened/written to/closed) the stack overflow message goes away, otherwise I would not have file-writing included in the problem's working example. Of course that makes the argument list to the many-argument printfmt have one less argument; however, when writing to stdout I ran a test with 50 arguments in the printfmt, as follows, and it worked ok:

printfmt("({:2d}, {:2d}, {:2d}, {:2d}, {:2d} {:2d}, {:2d}, {:2d}, {:2d}, {:2d},{:2d}, {:2d}, {:2d}, {:2d}, {:2d}{:2d}, {:2d}, {:2d}, {:2d}, {:2d} {:2d}, {:2d}, {:2d}, {:2d}, {:2d},{:2d}, {:2d}, {:2d}, {:2d}, {:2d} {:2d}, {:2d}, {:2d}, {:2d}, {:2d}, {:2d}, {:5.2f}, {:5.2f}, {:5.2f}, {:5.2f}, [{:5.2f}, {:5.2f}, {:5.2f}], [{:5.2f}, {:5.2f}, {:5.2f}], ", DRP.RPnumber, n, n+2, n+6, n+7, n+3, n, n+12, n+16, n+17, n+13, n, n-2, n-6, n-7, n-3, n, n-12, n-16, n-17, n-13, n, n+2, n+6, n+7, n+3, n, n+12, n+16, n+17, n+13, n, n-2, n-6, n-7, n-3, n, n-12, n-16, n-17, n-13, r, s, raxi, zaxi, c[1],c[2],c[3], nnnv[1],nnnv[2],nnnv[3])

suggesting number of arguments is not the whole story.

Note, to some extent the number of error messages that print can be decreased or increased by moving the ]# characters earlier or later within the cots = ... array.

Note, if the recursion-call findMore(level+1) is commented out, then toverflow3 results are unchanged, while julia-latest toverflow5.jl runs with only one error-message pair:

Internal error: stack overflow in type inference of toverR().
This might be caused by recursion over very long tuples or argument lists.

while julia toverflow5.jl runs with about 19030 error message lines before it produces ok results.

Here is program toverflow5: edit: Instead of toverflow5 as was shown below, see toverflow7 in next comment. For brevity I've removed the 140-line listing of toverflow5. end edit

versioninfo() results are shown next.

> julia <<< 'versioninfo()'
Julia Version 1.7.2
Commit bf53498635 (2022-02-06 15:21 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, skylake)

> julia-latest <<< 'versioninfo()'
Julia Version 1.9.0-DEV.485
Commit 6e06132243 (2022-05-07 08:12 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: 4 × Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, skylake)
  Threads: 1 on 4 virtual cores

[ghjwp7]

Shown below is program toverflow7.jl. It's half 1/3 1/5 as long as toverflow5. toverflow7.jl exhibits the following error message when run via julia-latest toverflow7.jl (on my system julia-latest is v. 1.9.0-DEV.485) :

Internal error: stack overflow in type inference of toverR().
This might be caused by recursion over very long tuples or argument lists.

Program toverflow8, obtained by commands A=toverflow7;B=toverflow8;sed -e 's/if 1<2/if 1>2/' $A.jl > $B.jl; chmod u+x $B.jl, runs ok. Listing of toverflow7.jl is next, followed by verbose versioninfo.
[edit: replaced a wrong listing, and also removed using LinearAlgebra; ca. 28 lines in following.]

# -*- mode: julia; -*- jiw - 12 May 2022 - exhibit error "Internal
# error: stack overflow in type inference of toverR()" triggered by
# use of one long arg list in a printfmt(), and not triggered by use
# of two shorter lists doing the same thing.  Note, problem also goes
# away if DRP.f is a plain var instead of a struct element.
using Formatting
struct DataRP
    f
end
function toverR()
    DRP=DataRP(open("outP00", "w"))
    RPcounts = zeros(Int32, 12)
    function findMore()::Nothing
        s, r, nnnv, c = 2.0, 1.414213, [0.0, 0.0, -4.0], [13.,13.,13.]
        RPcounts[4] = n = raxi = zaxi = 13
        if 1<2           # sed changes < to > for other version
            printfmt(DRP.f, "({:2d}, {:2d}, {:5.2f}, {:5.2f}, {:5.2f}, {:5.2f}, [{:5.2f}, {:5.2f}, {:5.2f}], [{:5.2f}, {:5.2f}, {:5.2f}], \n", 13, n, r, s, raxi, zaxi, c[1],c[2],c[3], nnnv[1],nnnv[2],nnnv[3])
        else
            printfmt(DRP.f, "({:2d}, {:2d}, {:5.2f}, {:5.2f}, {:5.2f}, {:5.2f}, ", 13, n, r, s, raxi, zaxi)
            printfmt(DRP.f, "[{:5.2f}, {:5.2f}, {:5.2f}], [{:5.2f}, {:5.2f}, {:5.2f}], \n", c[1],c[2],c[3], nnnv[1],nnnv[2],nnnv[3])
        end
        return nothing
    end
    findMore()
    close(DRP.f)
    println("RPtotals = $RPcounts")
end
toverR()

Verbose versioninfo except PATH:

Julia Version 1.9.0-DEV.485
Commit 6e06132243 (2022-05-07 08:12 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
      Ubuntu 20.04.4 LTS
  uname: Linux 5.4.0-109-generic #123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022 x86_64 x86_64
  CPU: Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz: 
              speed         user         nice          sys         idle          irq
       #1  3691 MHz     448332 s        809 s     119042 s    8231382 s          0 s
       #2  3678 MHz     457086 s        722 s     124489 s     420950 s          0 s
       #3  3693 MHz     452091 s        769 s     121692 s     421421 s          0 s
       #4  3692 MHz     459232 s        671 s     118750 s     420648 s          0 s
  Memory: 31.2266845703125 GB (14078.85546875 MB free)
  Uptime: 1.25151872e6 sec
  Load Avg:  0.31  0.32  0.3
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, skylake)
  Threads: 1 on 4 virtual cores
Environment:
  WINDOWPATH = 2
  HOME = /home/j-waldby
  TERM = xterm-256color

[ghjwp7]

Here is (1) toverflow7 with some of the cruft cleaned out, (2) output from run78 shell script, (3) run78 shell script:

(1) toverflow7.jl:

# -*- mode: julia; -*-
# jiw - 12 May 2022 - exhibit error "Internal error: stack overflow in
# type inference of toverR()" triggered by use of one long arg list in
# a printfmt(), and not triggered by use of two shorter lists doing
# the same thing.  Note, problem also goes away if D.f is a plain var
# instead of a struct element.
using Formatting
struct Dat
    f
end
function rover()
    D=Dat(open("outP00", "w"))
    function tover()
        if 1<2           # sed changes < to > for other version
            printfmt(D.f, "R({} {} {} {} {} {} {} {} {} {} {})\n", 1,2,3,4,5,6,7,8,9,10,11)
        else
            printfmt(D.f, "R({} {} {} {} {} {}", 1,2,3,4,5,6)
            printfmt(D.f, " {} {} {} {} {})\n", 7,8,9,10,11)
        end
    end
    tover()
    printfmt(D.f, "S({} {} {} {} {} {} {} {} {} {} {})\n", 1,2,3,4,5,6,7,8,9,10,11)
    close(D.f)
    printfmt("T({} {} {} {} {} {} {} {} {} {} {})\n", 1,2,3,4,5,6,7,8,9,10,11)
end
rover()

(2) output from run78 shell script:

Running toverflow7
Internal error: stack overflow in type inference of rover().
This might be caused by recursion over very long tuples or argument lists.
T(1 2 3 4 5 6 7 8 9 10 11)
outP00 contents:
R(1 2 3 4 5 6 7 8 9 10 11)
S(1 2 3 4 5 6 7 8 9 10 11)
toverflow7 done in ~ 5 seconds
sed t7→t8

Running toverflow8
T(1 2 3 4 5 6 7 8 9 10 11)
outP00 contents:
R(1 2 3 4 5 6 7 8 9 10 11)
S(1 2 3 4 5 6 7 8 9 10 11)
toverflow8 done in ~ 1 seconds

(3) run78 shell script:

#!/bin/sh
A=toverflow7
B=toverflow8
running(){
    P=$1
    S=$(date +%s)
    echo; echo Running $P
    julia-latest $P.jl
    S=$(($(date +%s)-S))
    echo "outP00 contents:"
    cat outP00
    echo "$P done in ~ $S seconds"
}
running $A
echo sed t7→t8
sed -e 's/if 1<2/if 1>2/' $A.jl > $B.jl
running $B