At_mul_B and gemv is broken on Travis

[lindahua]

In these two days, I observe unusual travis failure in several packages, including NumericExtensions and Distributions. The failing part passes tests perfectly in my local machines.

Examining the codes, I found that those parts that suddenly failed recently involve the calling to either At_mul_B or gemv. So I specially created a gemvbug branch on NumericExtensions.jl and added the simple test as follows, and see whether it can pass travis.

using Base.Test

m = 7
n = 8
x = rand(m, n)
a = rand(m)

# a slow but safe function to compute At_mul_B
function my_mul(x::Matrix{Float64}, a::Vector{Float64})
    m = size(x, 1)
    n = size(x, 2)

    r = zeros(n)
    for j = 1:n
        s = 0.
        for i = 1:m
            s += x[i,j] * a[i]
        end
        r[j] = s
    end
    return r
end

function blas_mul(x::Matrix{Float64}, a::Vector{Float64})
    n = size(x, 2)
    r = Array(Float64, n)
    At_mul_B(r, x, a)
    r
end

println("x = $x")
println("a = $a")

r0 = my_mul(x, a)
println("my_mul => $r0")

r1 = blas_mul(x, a)
println("blas_mul => $r1")

@test_approx_eq r0 r1  # this failed tests

The implementation is correct and it passes local testing. But travis failed.
Here is what travis produces:

my_mul => [1.6623387959220757,1.021323256792092,1.3800204235755555,1.9019911669465768,1.4643172831039295,1.9058177790830482,2.1335438803632534,1.6529451897943939]
blas_mul => [1.6623387959220757,0.42876508001595803,1.3800204235755555,1.508452276637747,1.4643172831039297,1.91331093372259,2.1335438803632534,0.9207130848734909]
ERROR: assertion failed: |r0 - r1| <= 7.579870528087629e-11
  r0 = [1.6623387959220757,1.021323256792092,1.3800204235755555,1.9019911669465768,1.4643172831039295,1.9058177790830482,2.1335438803632534,1.6529451897943939]
  r1 = [1.6623387959220757,0.42876508001595803,1.3800204235755555,1.508452276637747,1.4643172831039297,1.91331093372259,2.1335438803632534,0.9207130848734909]
  difference = 0.732232104920903 > 7.579870528087629e-11

Travis page: https://travis-ci.org/lindahua/NumericExtensions.jl (build #19).

Clearly, At_mul_B generates correct results in r1[1], r1[3], ..., and incorrect results in r1[2], r1[4], .... The difference is 0.7322, which is obviously not just because of numerical truncation.

I am not completely sure what is the culprit -- perhaps the openblas installed in Travis?

gemv yields obviously incorrect results is a serious bug that affects a lot of computation, which needs to be fixed soon.

cc: @staticfloat @ViralBShah

[staticfloat]

@lindahua; you are just a little too fast for me! I think I've found an (unsatisfying) solution to this. OpenBLAS just got a bunch of new AMD Piledriver-optimized codes, and Travis uses Piledriver CPUs. These codes seem to have some bugs in them, so I'm currently figuring out the best way to recompile OpenBLAS without using Piledriver codes.

[timholy]

You folks are amazing.

[lindahua]

It is working now.