Infinitely growing compilation recursion

[andreasnoack]

On 0.4 I get

julia> UpperTriangular(randn(3,3))\sub(randn(3), [1,2,3])
ERROR: StackOverflowError:
 in \ at linalg/generic.jl:323
 in \ at linalg/generic.jl:326 (repeats 74845 times)

but on 0.5.0-dev+3878 (2016-05-04 17:25 UTC) Julia just hangs and I cannot break out of recursion so I have to kill Julia.

Update: This is on OS X.

[yuyichao]

What happens when you run this in a debugger?

[vtjnash]

it's sitting in an infinitely growing compilation recursion, for example:
Base.LinAlg.\(Base.#\, Base.LinAlg.LowerTriangular{Float64, Base.LinAlg.UpperTriangular{Float64, Base.LinAlg.LowerTriangular{Float64, Base.LinAlg.UpperTriangular{Float64, Base.LinAlg.LowerTriangular{Float64, Base.LinAlg.UpperTriangular{Float64, Base.LinAlg.LowerTriangular{Float64, Base.LinAlg.UpperTriangular{Float64, Base.LinAlg.LowerTriangular{Float64, Base.LinAlg.UpperTriangular{Float64, Base.LinAlg.LowerTriangular{Float64, Base.LinAlg.UpperTriangular{Float64, Base.LinAlg.LowerTriangular{Float64, Base.LinAlg.UpperTriangular{Float64, Base.LinAlg.LowerTriangular{Float64, Base.LinAlg.UpperTriangular{Float64, Base.LinAlg.LowerTriangular{Float64, Base.LinAlg.UpperTriangular{Float64, Base.LinAlg.LowerTriangular{Float64, Base.LinAlg.UpperTriangular{Float64, Base.LinAlg.LowerTriangular{Float64, Base.LinAlg.UpperTriangular{Float64, Base.LinAlg.LowerTriangular{Float64, Base.LinAlg.UpperTriangular{Float64, Base.LinAlg.LowerTriangular{Float64, Base.LinAlg.UpperTriangular{Float64, Base.LinAlg.LowerTriangular{Float64, Base.LinAlg.UpperTriangular{Float64, Base.LinAlg.LowerTriangular{Float64, Base.LinAlg.UpperTriangular{Float64, Base.LinAlg.LowerTriangular{Float64, Base.LinAlg.UpperTriangular{Float64, Base.LinAlg.LowerTriangular{Float64, Base.LinAlg.UpperTriangular{Float64, Array{Float64, 2}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}}, Base.SubArray{Float64, 1, Array{Float64, 1}, Tuple{Array{Int64, 1}}, false})

[yuyichao]

Regression on #14009 ?

[vtjnash]

probably related, yes. but this is in codegen, not inference.

[andreasnoack]

This bug complicates the work in #6837 because I have to kill Julia when a method is missing. It's a slow process to figure out if the tests are just slow (like the 15 minutes of triangular.jl) or if they are hanging because of this issue.

[andreasnoack]

I've bisected this and there seems to two issues. When executing

UpperTriangular(randn(3,3))\sub(randn(3), [1,2,3])

1. My changes in 87e26c5 cause an infinite loop in inference instead of an instant StackOverflowError but it is still possible to break out of the loop with ctrl+c.
2. @vtjnash's changes in 8f4238a make the hang fatal and Julia has to be killed.

Right now, I'm working on a fix for the specific statement above such the dispatch isn't recursive but I would still expect that the behavior in 1. and 2. are bugs, i.e. type inference should never go into an infinite loop and you should always be able to break out of a loop with ctrl+c.

[andreasnoack]

This is kind of fixed by #16288. I've changed the title to reflect the issue with infinite compilation recursions. Isn't there a way to detect this?

[StefanKarpinski]

TODO: add the test case to #16692.