gcc-4.8: Fix Linux portability problems

[stuarteberg]

Edit: This thread is getting long, so I'll try to keep an up-to-date summary at the top here.

<tl;dr>Without this patch, the gcc package is worthless on nearly all Linux distros. But with this patch, it works on all non-ancient Linux distros.</tl;dr>

Conda's gcc package was built on CentOS 5.11, and only works on that distro. However, we can make it work on more distros by adding a post-link.sh file.

In this PR, post-link.sh does three important things:

1. Remove the headers generated by gcc's "fixincludes" build step. They were made specifically for CentOS 5.11, and don't work elsewhere. If your target OS already has ANSI-compliant libc headers, you don't need the extra "fixed" headers provided by gcc. (On newer Linux distros, this seems to be the case.)
2. Help gcc locate the system libc headers by adding an -I flag to the default gcc flags (via a specs file).
3. Help gcc locate certain system libc object files (e.g. crt1.o) by creating some symlinks to them.

Also, post-link.sh has the following features:

I. If you are installing onto OSX, do nothing (post-link.sh is a no-op).
II. If you are installing onto the same OS that was used to create the gcc package, do nothing. (Here, the definition of "same OS" means: compare the output of cat /etc/*-release at build-time and install-time.)
III. In interest of "failing early" if you're trying this gcc package on a system that just can't use it, the post-link.sh script compiles a "Hello World" program at install time. If something is broken, running conda install gcc will simply fail.

Without this PR, the gcc package is basically non-portable across Linux distros.

It's true: even with this PR, we still can't support every Linux distro under the sun, but at least now we can support a lot more. And thanks to item II above, you can always just build the gcc recipe yourself (in which case post-link.sh becomes a no-op). This means the Anaconda build server won't be affected by these changes.

(Sadly, if your Linux distro is so old that it has "bad" libc headers, even this PR can't help you. In that case your only option is to build gcc yourself, or obtain it through other means.)

Good news! You can try this PR on own system without needing to build gcc yourself:

conda install -c stuarteberg gcc=4.8.4.99
conda install -c asmeurer gcc=4.8.5

That package is known to work on the following distros:

• CentOS 5.11
• CentOS 6.3
• CentOS 6.6
• Fedora 20
• Ubuntu 12.04
• Ubuntu 14.04
• Archlinux

---

This PR attempts to fix the issue described on the mailing list here:
https://groups.google.com/a/continuum.io/d/msg/conda/HwUazgD-hJ0/aofO0vD-MhcJ

To summarize:

• The gcc build scripts run a step called "fixincludes" which generates custom versions of several system header files.
• Those files are compatible ONLY with the system gcc was built on, so including them in the gcc package makes the gcc binary non-portable to other systems.
• Apparently, many Linux distros simply remove these "fixed" headers as a workaround for this problem. (See http://ewontfix.com/12). This PR does the same.
• As long as the target system has ANSI-compatible headers (which don't use C language syntax that gcc doesn't support yet), it should be okay to omit the "fixed" versions.

---

Update: As described below, now this PR also address two other portability issues: locating system headers and locating system libc code on various distros.

[asmeurer]

Should this also run on OS X? If not, add an if statement to the script.

[asmeurer]

What is the best way to test this?

[stuarteberg]

Should this also run on OS X? If not, add an if statement to the script.

After some testing on Mac OS 10.10, apparently not. (I'll update the patch.) The only "fixed" header is stdint.h, and it doesn't seem to cause compilation errors on the programs I've tried so far.

On a different point, the conda gcc package doesn't work out-of-the-box on Mac OS X. (It doesn't know where to find the system headers.) I'll open a separate issue for that later.

What is the best way to test this?

Well, ideally, we'd just try compiling a big stack (all of the default Anaconda packages?) on multiple Linux distributions. If nothing seems to break, I guess we're good to go.

[stuarteberg]

Should this also run on OS X? If not, add an if statement to the script.

Done.

BTW, I was wrong about this Mac issue I mentioned above:

On a different point, the conda gcc package doesn't work out-of-the-box on Mac OS X. (It doesn't know where to find the system headers.) I'll open a separate issue for that later.

The Mac I borrowed for this testing had XCode installed, but didn't have the "xcode command-line developer tools". Running xcode-select --install fixed my build issues

I suppose it would be possible to help boneheads like me by running a post-install script that verifies the command-line tools are installed, but it's probably not necessary -- most devs have them installed already. Also, verifying that they are installed is not simple.

[asmeurer]

It was not my intention for gcc to depend on the developer tools. Requiring them (almost) defeats the purpose. What error did you get? According to conda inspect linkages gcc, gcc linkages against the system packages libSystem.B.dylib, libgcc_s.1.dylib and libiconv.2.dylib (all in /usr/lib/). I suppose libgcc_s.1.dylib is the issue. I didn't realize that it would not be installed unless you have the developer tools installed. Or was it another issue?

[stuarteberg]

Just to clarify, I'm not reporting a problem with using the gcc or libgcc package as a runtime dependency for pre-built binaries. As far as I know, they work just fine for that purpose. I'm interested in using the gcc package as my main compiler, for building my own packages from source.

The problem I ran into is that gcc doesn't know where to find the libc headers (e.g. wchar.h, math.h, etc.). I think the headers aren't even installed by default (most users would never need them). If you install XCode, then you'll get a copy, but they're buried inside the XCode package in an SDK directory:

$ export SDK_DIR=`xcrun --show-sdk-path`
$ echo $SDK_DIR
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk
$ ls $SDK_DIR/usr/include/std*.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/include/stdbool.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/include/stddef.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/include/stdint.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/include/stdio.h
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/usr/include/stdlib.h

gcc doesn't know about this directory. But if you install the "Xcode command-line tools", then all the standard headers are copied into /usr/include, which gcc DOES know about.

$ xcode-select --install
$ ls /usr/include/std*.h
/usr/include/stdbool.h  /usr/include/stddef.h   /usr/include/stdint.h   /usr/include/stdio.h    /usr/include/stdlib.h

(Actually, the headers were only part of the problem. If I provided the headers via -I/$SDK_DIR/usr/include, other things, like crt1.o were also missing. Installing the command-line tools fixed that, too.)

I don't think it's unreasonable to expect users who want to compile their own packages to have XCode and the command-line tools already installed.

[asmeurer]

I don't think it's unreasonable to expect users who want to compile their own packages to have XCode and the command-line tools already installed.

It would be nice to avoid it if possible, since installing XCode requires root permissions, but I don't plan to dedicate a ton of effort to work around it myself.

[asmeurer]

Well, ideally, we'd just try compiling a big stack (all of the default Anaconda packages?) on multiple Linux distributions. If nothing seems to break, I guess we're good to go.

I was hoping for something a little more directed, i.e., something that is broken and easily reproducible with the current package.

[stuarteberg]

As I mentioned before, this PR works on Fedora 20. But when I tested on Ubuntu 12.04, I discovered yet more portability problems:

• Ubuntu keeps the std C headers in /usr/include/x86_64-linux-gnu, but the conda-built gcc doesn't know to look there. Therefore, "hello world" can't compile due to missing headers.

$ gcc hello.c
In file included from /usr/include/stdio.h:28:0,
                 from hello.c:1:
/home/vagrant/miniconda/envs/delme/lib/gcc/x86_64-unknown-linux-gnu/4.8.2/include-fixed/features.h:338:25: fatal error: sys/cdefs.h: No such file or directory
 #  include <sys/cdefs.h>
                         ^
compilation terminated.

Of course, one workaround is to pass explicit -I option:

gcc -I/usr/include/x86_64-linux-gnu hello.c

... but then we hit the next problem:

• The conda gcc binary can't find the C runtime object files crt1.0, crti.o, and crtn.o, resulting in a link error.

$ gcc -I/usr/include/x86_64-linux-gnu hello.c
/usr/bin/ld: cannot find crt1.o: No such file or directory
/usr/bin/ld: cannot find crti.o: No such file or directory
collect2: error: ld returned 1 exit status

This, too, can be worked around, this time with the -B option:

$ gcc -I/usr/include/x86_64-linux-gnu -B/usr/lib/x86_64-linux-gnu/ hello.c

Instead of requiring users to pass these flags to gcc every time they use it, we can augment the gcc binary ass follows:

• Fix the first issue by adding the -I option to the default set via a gcc specs file.
• Fix the second issue with symlinks to crt1.o, etc. inside $PREFIX/lib/gcc/..., which is always searched. (As suggested in gcc, binutils: link to Cellar instead of system libs Linuxbrew/legacy-linuxbrew#200.)

I've just added a post-link.sh script to this PR, which implements both of the above fixes. (I know post-link scripts are to be avoided when possible, but I think it's the right thing to do here. I guess we shouldn't be surprised that the gcc package is special in some ways.)

[asmeurer]

Hmm, conda-build really ought to have a way to not even include the file on OS X.

[frol]

@asmeurer This file is not only for OS X though. However, since you said that post-list.sh shouldn't be included on OS X. Could you please say why?

[asmeurer]

Is any of this process needed on OS X?

[frol]

@asmeurer The part below fi statement is kind of needed.

[asmeurer]

Why is it needed? As far as I can tell, all it does is test the compiler. I haven't heard of any issues with the existing gcc package on OS X.

[stuarteberg]

This entire PR is really for Linux. The things that this PR fixes ("fixed" std headers, inconsistent std header locations, and inconsistent locations for crt0.o, etc.) aren't issues on Mac. If OSX had multiple "distributions" like Linux, that would be a different story. It looks like we can just skip all this stuff on Mac.

The part below fi statement is kind of needed.

The part below the last fi merely verifies that "Hello World" compiles. Generally, it should always pass -- unless the user didn't install the xcode command-line tools on her machine. (Otherwise gcc can't find the standard library headers.)

[asmeurer]

If you have to have a post-link, fine. Just make sure it is very reliable (if it fails, the install will be broken), and very portable.

[stuarteberg]

Regarding testing:

I was hoping for something a little more directed, i.e., something that is broken and easily reproducible with the current package.

Since this issue is about fixing portability issues, the only way to demonstrate the problem(s) is to try using the package on different OSes.

Just to reiterate the above discussion, there are three known portability issues addressed in this PR:

1. The bogus fixincludes step
2. Can't find C library headers
3. Can't find C library runtime object files

Ubuntu 12.04 appears to be affected by all three issues. To demonstrate issues 2 and 3, try compiling this C program on Ubuntu 12.04 using conda's gcc binary:

$ cat > hello.c
#include <stdio.h>
int main(){}

$ gcc hello.c

$ OR:
$ gcc -I/usr/include/x86_64-linux-gnu hello.c

To demonstrate issue 1, try compiling this C++ program on Ubuntu 12.04:

$ cat > hello.cpp
#include <iostream>
int main(){}

$ g++ -I/usr/include/x86_64-linux-gnu -B/usr/lib/x86_64-linux-gnu/ hello.cpp

I don't know which other Linux distributions are affected by these issues, but I happen to know that Fedora 20 is affected by issue 1, and @jakirkham's issue referenced above suggests that CentOS 6.3 is also affected.

[stuarteberg]

BTW, my binstar channel has a pre-packaged version of these changes, which you can try out on the VM of your choice:

conda install -c stuarteberg gcc-no-fixincludes

(Disclosure: I didn't actually build it myself. I just copied your gcc package, removed the include-fixed headers, and added the post-link.sh file. That's also why it has a build number of 2.)

[stuarteberg]

There may be a simpler way for me to get the locations of crt1.o, etc. I'll revise this patch on Monday.

[jakirkham]

I don't know which other Linux distributions are affected by these issues, but I happen to know that Fedora 20 is affected by issue 1, and @jakirkham's issue referenced above suggests that CentOS 6.3 is also affected.

@stuarteberg So, I tried your binary ( https://binstar.org/stuarteberg/gcc-no-fixincludes ) and it did indeed resolve my build issues regarding NumPy on CentOS 6.3. It also worked on SciPy.

[stuarteberg]

I've added a test step in two places: run_test.sh and in post-link.sh. The post-link.sh step seems warranted because these portability issues can only be detected on the target system, not the build system.

It's worth noting that the addition of this new test in post-link.sh means that gcc should never be used as a run dependency. The libgcc package should be used instead. (BTW, Now would be a good time to move @asmeurer's libgcc package from binstar into conda's defaults...)

In other news, I've made some minor simplifications to post-link.sh and also added a pre-unlink.sh script. I think this PR is basically complete, so I'm hoping for comments.

[asmeurer]

You should bump the build number in the meta.yaml

[jakirkham]

@stuarteberg What OS's you tested these pre and post link steps?

[stuarteberg]

What OS's you tested these pre and post link steps?

Fedora 20, Ubuntu 12.04, ScientificLinux 6.3, and Mac OS 10.10 (for which I pushed some minor compatibility fixes just now).

You should bump the build number in the meta.yaml

OK, done. I bumped it from 1 to 3 because somehow the version in conda already has a build number of 2.

[stuarteberg]

What OS's you tested these pre and post link steps?

Fedora 20, Ubuntu 12.04, ScientificLinux 6.3, and Mac OS 10.10

Apparently I didn't test thoroughly enough. The post-link.sh test was using the system's gcc executable, not the conda-installed gcc (I didn't realize the PATH variable isn't updated until AFTER post-link.sh is executed...). Anyway, I pushed a fix to the post-link.sh, and updated my test package on binstar: stuarteberg/gcc-no-fixincludes (build number: 3).

[stuarteberg]

This thread was getting long, so I updated the description with a fresh summary of the changes here. See above.

[stuarteberg]

should i686 proceed i386 in the list or does it matter?

Sigh. I'm not sure. I guess the best thing would be to check the system gcc on the 32-bit VM. If both directories are listed, then we should use the same order it uses. This command will show the header search directories (among other things):

$ echo | gcc -v -x c -E -

I don't happen to have a 32-bit Ubuntu VM handy, so I can't test this myself.

[asmeurer]

This is what I did.

[stuarteberg]

This is what I did.

That should be just fine. Have you tested it yet?

FWIW, I just spun up a new Ubuntu 32-bit VM just to check the include path order Only /usr/include/i386-linux-gnu appears in the search path. I think there's no harm in having the other directory listed, though.

[asmeurer]

It seems to install and remove fine. I haven't tested it beyond that, though. The builds are on my binstar.

[izaid]

@asmeurer Can we merge it then? Or are we waiting for something else?

[stuarteberg]

Can we merge it then? Or are we waiting for something else?

He already merged it into conda-recipes master. The last step would be to add it to the default conda package repo, so it can be installed via conda install gcc instead of conda install -c asmeurer gcc

[izaid]

Yes, sorry, that's what I meant. @asmeurer is this something we can do?

[asmeurer]

I've also put it in on the r channel.

[izaid]

@asmeurer Is this something that we can get added to the default conda repo? Or is that unlikely to happen for whatever reason?

[asmeurer]

It will happen. I just want it to get tested a little bit before doing so.

[izaid]

Great, thanks!

Question: Is it worth making a gcc-libraries recipe that simply grabs the important libraries (libstdc++, ...) from gcc? This way other recipes can depend on the gcc libraries without needing all of GCC.

I've done this for DyND and it works nicely.

[jakirkham]

@izaid, I believe this is already done. ( https://github.com/conda/conda-recipes/blob/master/libgcc/meta.yaml )

[izaid]

@jakirkham Ah, okay, good.

But, don't we need another one for the gcc-4.8 related to this PR?

[jakirkham]

I have a PR for bumping the version to be consistent with the version of gcc here. ( #371 )

[jakirkham]

Also, @izaid, there is a copy of libgcc 4.8.4 in the anaconda channel that @asmeurer had added. ( https://anaconda.org/anaconda/libgcc ). Though I suppose the version should be bumped.

[asmeurer]

oh I forgot about libgcc. I should rebuild that as well.

[dfroger]

Hello,

It may be usefull to create a symbolic link cc to gcc, otherwise CMake (don't know for other build system) choose /usr/bin/cc instead of gcc of the conda package.

For C++, it seems ok (CMake choose c++ of the conda package).

[asmeurer]

That sounds like a good idea, although I'm wondering if that would break some of my OS X recipes that are currently working with clang.

[stuarteberg]

It may be usefull to create a symbolic link cc to gcc

Already done: @mjuric already did that in 510897e, which was merged into this PR before it was accepted by @asmeurer.

@dfroger, I'm guessing you are using conda install -c stuarteberg gcc=4.8.4.99, which I created before the @mjuric provided the cc fix. Sorry about that. Please try conda install -c asmeurer gcc=4.8.5 instead. It contains everything from this PR, plus more. (Hopefully his version will become the official gcc conda package soon...)

I'm wondering if that would break some of my OS X recipes that are currently working with clang

If you don't include the gcc package as a build dependency, then there should be no problems, right? If gcc isn't installed, it won't be on your path (and neither will the new cc symlink), and CMake won't find it.

[asmeurer]

I guess I am primarily thinking of R but that is managed via https://github.com/conda/conda-recipes/blob/7e78b47686c193fb81973d33b3797ed259eaccdf/r-base/build.sh#L61.

[dfroger]

@stuarteberg exactly, I was usingconda install -c stuarteberg gcc=4.8.4.99, I'll switch to conda install -c asmeurer gcc=4.8.5. Thanks a lot!

[stuarteberg]

Yes, it's best for all recipes to be explicit about which compiler they're using, especially if they involve C++ code. That means explicitly setting the CC and CXX environment variables, and also explicitly setting CMAKE_C_COMPILER and CMAKE_CXX_COMPILER when using cmake.

For my own project, the above guidelines are spelled out here: https://github.com/ilastik/ilastik-build-conda#appendix-compiler-details

[dfroger]

Thanks for the guidelines. By the way, I'm using this Vagrant box: https://github.com/dfroger/conda-build-env