kernel_4x4_haswell.S build error on scientific linux 6

[floswald]

I cannot build Julia from source on an hpc system. It errors with

../kernel/x86_64/dgemm_kernel_4x4_haswell.S:2548: Error: no such instruction: vpermpd $ 0xb1,%ymm3,%ymm3'

I tried to set the option as suggested here #7476 but that didn't change anything. my version is

cat /proc/version
Linux version 2.6.32-279.19.1.el6.x86_64 (mockbuild@sl6.fnal.gov) (gcc version 4.4.6     20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Tue Dec 18 17:22:54 CST 2012

[timholy]

Does the solution in #7240 fix it?

[floswald]

It DID fix it! thanks!

[tkelman]

If you're trying to build for a heterogeneous cluster with a variety of processors, you're going to need OPENBLAS_DYNAMIC_ARCH enabled. Did you actually apply the patch from #7476 to your local copy? It hasn't been merged into master yet so you would need to manually replicate the additional lines of code in your local makefile.

[floswald]

@tkelman I did apply your patch into my make.user locally. that didn't change the failing build.
doing your trick from #7240 however worked:

OPENBLAS_DYNAMIC_ARCH=0
JULIA_CPU_TARGET=core2

this build works on my cluster. Are you saying that it should not? I'm unsure how the core2 target interacts with this issue, just that if I don't set this, I get an arch mismatch error. Entirely possible that I'm just lucky here - please let me know!

[tkelman]

JULIA_CPU_TARGET is related to the code generation in the precompiled Julia system image, rather than openblas. I'm very surprised OPENBLAS_DYNAMIC_ARCH = 0 works. If you had compiled Julia on a newer machine with a newer instruction set, then openblas compiled on that machine would not work on an older processor. You're saying setting OPENBLAS_NO_AVX = 1 and applying my patch (to deps/Makefile, as well as setting the option in Make.user) was still giving the same openblas error message? Did you clean out the openblas build between changing this setting?

By "apply the patch", I mean specifically adding these 3 lines to deps/Makefile, around line 790:

# Allow disabling AVX for older binutils
ifeq ($(OPENBLAS_NO_AVX), 1)
OPENBLAS_BUILD_OPTS += NO_AVX=1
endif

[floswald]

i did actually wipe the entire julia directory and clone a fresh copy between both trials. but i'm not sure if I applied your patch correctly. so I just have to set echo OPENBLAS_NO_AVX = 1 > Make.user and copy those 4 lines into my deps/Makefile? I'm tempted not to try this again, since it works now. i'll build a second version if I have time later on.

[tkelman]

so I just have to set echo OPENBLAS_NO_AVX = 1 > Make.user and copy those 4 lines into my deps/Makefile?

Yes, that should do it. If NO_AVX=1 was properly getting sent to OpenBLAS and you were still getting the same no such instruction error, then I'd consider it a bug somewhere in OpenBLAS's build system.

[tkelman]

Relevant cross-reference: OpenMathLib/OpenBLAS#401

Disabling all AVX instructions could hurt performance on Sandy Bridge, Bulldozer, and Piledriver, but the problematic vpermpd instruction is actually AVX2 and only used on Haswell so far.