BOOT_LDFLAGS=-Wl,-headerpad_max_install_names ineffective

[ilg-ul]

I updated the xPack GCC build to use the latest gcc-11.2.0-arm-20211124 from @fxcoudert, and the build passed on macOS 11.6 both on Apple Silicon and Intel, but failed on an older macOS 10.13 x64, during the post-processing step, due to a limited headerpad:

## Preparing g++-mapper-server /Users/ilg/Work/gcc-11.2.0-2/darwin-x64/install/gcc/libexec/gcc/x86_64-apple-darwin17.7.0/11.2.0/g++-mapper-server libraries...

[install_name_tool -rpath @loader_path /Users/ilg/Work/gcc-11.2.0-2/darwin-x64/install/gcc/libexec/gcc/x86_64-apple-darwin17.7.0/11.2.0 /Users/ilg/Work/gcc-11.2.0-2/darwin-x64/install/gcc/libexec/gcc/x86_64-apple-darwin17.7.0/11.2.0/g++-mapper-server]
error: /Library/Developer/CommandLineTools/usr/bin/install_name_tool: changing install names or rpaths can't be redone for: /Users/ilg/Work/gcc-11.2.0-2/darwin-x64/install/gcc/libexec/gcc/x86_64-apple-darwin17.7.0/11.2.0/g++-mapper-server (for architecture x86_64) because larger updated load commands do not fit (the program must be relinked, and you may need to use -headerpad or -headerpad_max_install_names)

I'm invoking make with the same trick as in HomeBrew:

make all-gcc "BOOT_LDFLAGS=-Wl,-headerpad_max_install_names"

but it looks like this setting does not reach the linker:

/Users/ilg/Work/gcc-11.2.0-2/darwin-x64/build/gcc-gcc-11.2.0-arm-20211124/./gcc/xg++ -B/Users/ilg/Work/gcc-11.2.0-2/darwin-x64/build/gcc-gcc-11.2.0-arm-20211124/./gcc/ -nostdinc++ `if test -f /Users/ilg/Work/gcc-11.2.0-2/darwin-x64/build/gcc-gcc-11.2.0-arm-20211124/x86_64-apple-darwin17.7.0/libstdc++-v3/scripts/testsuite_flags; then /bin/bash /Users/ilg/Work/gcc-11.2.0-2/darwin-x64/build/gcc-gcc-11.2.0-arm-20211124/x86_64-apple-darwin17.7.0/libstdc++-v3/scripts/testsuite_flags --build-includes; else echo -funconfigured-libstdc++-v3 ; fi` -L/Users/ilg/Work/gcc-11.2.0-2/darwin-x64/build/gcc-gcc-11.2.0-arm-20211124/x86_64-apple-darwin17.7.0/libstdc++-v3/src -L/Users/ilg/Work/gcc-11.2.0-2/darwin-x64/build/gcc-gcc-11.2.0-arm-20211124/x86_64-apple-darwin17.7.0/libstdc++-v3/src/.libs -L/Users/ilg/Work/gcc-11.2.0-2/darwin-x64/build/gcc-gcc-11.2.0-arm-20211124/x86_64-apple-darwin17.7.0/libstdc++-v3/libsupc++/.libs -B/Users/ilg/Work/gcc-11.2.0-2/darwin-x64/build/gcc-gcc-11.2.0-arm-20211124/x86_64-apple-darwin17.7.0/libstdc++-v3/src/.libs -B/Users/ilg/Work/gcc-11.2.0-2/darwin-x64/build/gcc-gcc-11.2.0-arm-20211124/x86_64-apple-darwin17.7.0/libstdc++-v3/libsupc++/.libs -B/Users/ilg/Work/gcc-11.2.0-2/darwin-x64/install/gcc/x86_64-apple-darwin17.7.0/bin/ -B/Users/ilg/Work/gcc-11.2.0-2/darwin-x64/install/gcc/x86_64-apple-darwin17.7.0/lib/ -isystem /Users/ilg/Work/gcc-11.2.0-2/darwin-x64/install/gcc/x86_64-apple-darwin17.7.0/include -isystem /Users/ilg/Work/gcc-11.2.0-2/darwin-x64/install/gcc/x86_64-apple-darwin17.7.0/sys-include    -static-libstdc++ -static-libgcc  -fPIE -o g++-mapper-server server.o resolver.o ../libcody/libcody.a ../gcc/version.o ../libiberty/libiberty.a 

If I remove the g++-mapper-server binary from the install folder, the post-processing step finishes fine.

The same script with the previous gcc-11.1.0-arm-20210504, passed, probably because 11.1 uses a simpler rpath mechanism (?).

[iains]

There are some differences between linker behaviour w.r.t header name padding IIRC.
but first... what is the install_name_tool operation intending to achieve?
if there's a bug in the rpath handling that should be fixed in the compiler...

[ilg-ul]

install_name_tool allows to modify the rpath in the binary.

I use a post-processing step to make all binaries relocatable.

[iains]

yes, but with the @rpath change - they are already relocatable... if not, then there's a bug to fix ;)

[ilg-ul]

I would not call it a 'big bug' yet; my build scripts are also doing some library re-arrangements, plus that the scripts should also work with other packages, which do not generate relocatable binaries.

Anyway, the point is that the trick with BOOT_LDFLAGS does not work for g++-mapper-server, and there should be a way to allow further changes with install_name_tool.

[ilg-ul]

I did a test and, after adding -Wl,-headerpad_max_install_names to the global LDFLAGS, the build script passed, so the ineffective HomeBrew trick is no longer needed.

I'll restart the full build and we'll know in a few hours how it went.

[iains]

c++tools (which provides g++-mapper-server) is not bootstrapped but built at the final stage. So I'd not be surprised to see that BOOT_xxxxx does not apply (but xxxFLAGS would).

NOTES: It is installed into
libexec/gcc/aarch64-apple-darwinNN/<version>/

so that @loader_path would not actually find any dependent libraries anyway (unless you have a very non-standard re-layout).

In my build it has no dependent libraries (from the gcc build) in any case (of course libSystem has a non-@rpath entry):

$ otool -Lv c++tools/g++-mapper-server 
c++tools/g++-mapper-server:
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5)
	time stamp 2 Wed Dec 31 17:00:02 1969

So it should not / does not need any rpath?

[ilg-ul]

c++tools (which provides g++-mapper-server) is not bootstrapped but built at the final stage.
So I'd not be surprised to see that BOOT_xxxxx does not apply

Ok, so this explains why BOOT_LFLAGS has no effect.

(but xxxFLAGS would).

Unfortunately LDFLAGS does not apply either. I see -Wl,-headerpad_max_install_names in the logs applied almost everywhere (there are 52 references in the make output log), except g++-mapper-server.

So it should not / does not need any rpath?

Good point. I'll check the scripts, perhaps I can optimise them to no longer use install_name_tool at all.

---

Before considering the port to Apple Silicon, the xPack build scripts used the GNU 11.2 GCC, which did not use @rpath and generated absolute paths for shared libraries. Making the binaries relocatable was easy, just copy the libraries to libexec, compute the relative path from the binary to the library, and add a @relative_path reference.

When I started work with the Mini M1, and used the GCC 11.1 HomeBrew branch, I was surprised to find all binaries using @rpath. Although this in itself is not wrong (the binaries were able to run properly in the build environment), this made the logic to transform the binaries into fully relocatable more complex. Now I have to scan the list of shared libraries (which no longer show the absolute path), search each binary in the @rpath list, identify the absolute location, copy the library in libexec, and recompute the relative path.

So, although in principle the idea to use @rpath is not bad, the current implementation just makes life more difficult in some cases; in other words, the behaviour of the GNU GCC 11.2, which generated absolute paths, was more convenient.

I cannot talk for all applications, but since this change affected my builds, I expect others to be affected in a negative way too.

[iains]

(but xxxFLAGS would).

Unfortunately LDFLAGS does not apply either. I see -Wl,-headerpad_max_install_names in the logs applied almost everywhere (there are 52 references in the make output log), except g++-mapper-server.

OK - there is some build/config glitch in GCC then... something to look at sometime.

=====

Before considering the port to Apple Silicon, the xPack build scripts used the GNU 11.2 GCC, which did not use @rpath and generated absolute paths for shared libraries. Making the binaries relocatable was easy, just copy the libraries to libexec, compute the relative path from the binary to the library, and add a @relative_path reference.

OK - that's a very non-standard installation - if one wants the libraries in a different place, one usually uses the slibdir/libdir config options...

... what is your objective in terms of "relocatability"?

When I started work with the Mini M1, and used the GCC 11.1 HomeBrew branch, I was surprised to find all binaries using @rpath. Although this in itself is not wrong (the binaries were able to run properly in the build environment), this made the logic to transform the binaries into fully relocatable more complex. Now I have to scan the list of shared libraries (which no longer show the absolute path), search each binary in the @rpath list, identify the absolute location, copy the library in libexec, and recompute the relative path.

Actually, with the rpath implementation the compiler is already position-independent / relocatable, you don't need to do anything.

An exe built with the compiler does, of course, contain abs path references to where the libraries are - that is the same as the absolute path case (see below about position-independent apps).

So, although in principle the idea to use @rpath is not bad, the current implementation just makes life more difficult in some cases; in other words, the behaviour of the GNU GCC 11.2, which generated absolute paths, was more convenient.

because od SIP, the compiler cannot be configured/built properly with absolute paths on any version of macOS > 10.10. It is/has been a bad situation which has taken a lot of effort to seek alternative mechanisms to solve it.

the only viable current mechanism I've found that works within the vendor's security model is @rpaths - but I am open to suggestions of course (i.e. "patches are always welcome").

I cannot talk for all applications, but since this change affected my builds, I expect others to be affected in a negative way too.

typically apps are packaged with @rpath libraries and frameworks and @executable_path / @loader_path run paths to make the whole app package relocatable in a drag-and-drop sense.

We don't aim to do that with the compiler - because we expect the user to want a simple process where they use the compiler's runtime libraries from the installation.

A GCC user can, of course, do this with their built exes by using an @executable_path and copying the compiler libraries into their application package (exactly as developer advice goes).

[iains]

(clang does not have to worry about this [yet?] because the clang runtimes are already installed on the system) .. I say "yet" because with a growing set of front-ends, the LLVM project will also end up growing more runtimes. At present, they already use @rpath in the builds.

[ilg-ul]

if one wants the libraries in a different place, one usually uses the slibdir/libdir config options...

Yeah, just that this does not work for all libraries. When using custom toolchains, the resulting binaries also have references to the compiler internal libraries. Some can be linked statically, some can not. Thus, the most generic solution is to post-process the resulting binaries and bring all dependencies in the new package, which later will be distributed as standalone.

what is your objective in terms of "relocatability"?

Ah, probably the term is not very inspired, by relocatability I don't mean PIE, PIC, etc, but a package that does not need any install, it is a folder that runs the same from any location in the filesystem.

because of SIP, the compiler cannot be configured/built properly with absolute paths on any version of macOS > 10.10.

This is probably right if you want to use system folders, but all my packages are expected to be unpacked in user folders, and I did not have any problem to run them on macOS 11.x.

[iains]

what is your objective in terms of "relocatability"?

Ah, probably the term is not very inspired, by relocatability I don't mean PIE, PIC, etc, but a package that does not need any install, it is a folder that runs the same from any location in the filesystem.

If the current @rpath compiler does not do this "out of the box" then please let me know - because it should do this just fine with no modification.

[iains]

because of SIP, the compiler cannot be configured/built properly with absolute paths on any version of macOS > 10.10.

This is probably right if you want to use system folders, but all my packages are expected to be unpacked in user folders, and I did not have any problem to run them on macOS 11.x.

It has nothing to do with permissions (or whether a rewritten compiler can run in a different place) - it is to do with the fact that DYLD_xxxx are stripped from the environment when using key system utilities - like /bin/sh which is hard-wired into configure scripts.

this defeats (some of) the compiler's configuration and build steps (and makes in-tree testing unreliable unless the target libraries are first installed into their intended paths).

=====

by making the compiler position-independent from the get-go, we avoid these problems (there should be no need to change anything to have it run from your user dir, or from /Users/Shared or /opt/cool_toolchain.)

[ilg-ul]

If the current @rpath compiler does not do this "out of the box" then please let me know - because it should do this just fine with no modification.

Then I'm probably missing something :-(

Even worse, I don't even know how this would be possible, since I see references like @rpath/libgcc_s.1.1.dylib, which simply cannot work out the box, these libraries are no longer present in a standalone application; the only solution I found was to copy them to the standalone distribution, and this requires changing the references, and removing the rpath definitions.

[iains]

Suppose you are a compiler user (forget for a moment that you are packaging toolchains).

You unpack the compiler in /Users/me/gcc-11-2

you compile some code

/Users/me/gcc-11-2/bin/gcc hello.c -o hello

you run the built code:

./hello

Nothing more should be needed - it does not matter what happens "under the hood" so to speak.

===== for the toolchain packager - this is what happens under the hood.

The runtime libraries are in /Users/me/gcc-11-2/lib and they are @rpath/....

When the compiler links the user's hello exe, it automatically adds embedded runpaths for the places that libraries are found ; e.g. /Users/me/gcc-11-2/lib /Users/me/gcc-11-2/lib/gcc// etc.

So instead of hard-wiring the compiler's installation path into the libraries - we now provide the exe with the same information.

IFF you add your own -rpaths, then we assume that you are packaging as per the developer documentation, and then you have to supply all the necessary rpaths (and put the runtime libraries somewhere special, it's assumed).

(for a typical user, of course, none of this is relevant - they just build exes as before - the only difference is that they can place the compiler where they prefer).

** if you don't like this - you can always force the configuration off (--disable-darwin-at-rpath) but then you are back to the problems I tried to outline.

So ... in summary;

• If the compiler is not 'position independent' when built normally then please file a bug

• If you don't like this way of doing things then you can switch it off; and rewrite stuff to achieve similar ends. However, you're on your own w.r.t misconfiguration of libstc++.

[iains]

For this actual bug (configuration parameters for c++tools) we probably need a GCC PR - it's nothing specific to aarch64-darwin.

[ilg-ul]

Suppose you are a compiler user ... Nothing more should be needed

Right, custom compiler, regular compile line; the elf runs, details are not relevant, user is happy, :-)

for the toolchain packager ... When the compiler links the user's hello exe, it automatically adds embedded runpaths for the places that libraries are found

In other words, the elf compiled with the custom toolchain will include the absolute paths to the folders with the compiler libraries.

For example, using the 11.1 GCC compiled from the HomeBrew sources, while compiling a hello.cpp, I got:

hello:
	@rpath/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.29.0)
	@rpath/libgcc_s.1.1.dylib (compatibility version 1.0.0, current version 1.1.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5)

Load command 16
          cmd LC_RPATH
      cmdsize 32
         path @loader_path (offset 12)
Load command 17
          cmd LC_RPATH
      cmdsize 104
         path /Users/ilg/.local/xbb/usr/arm64-apple-darwin20.6.0/lib/gcc/arm64-apple-darwin20.6.0/11.1.0 (offset 12)
Load command 18
          cmd LC_RPATH
      cmdsize 152
         path /Users/ilg/.local/xbb/usr/arm64-apple-darwin20.6.0/lib/gcc/arm64-apple-darwin20.6.0/11.1.0/../../../../../arm64-apple-darwin20.6.0/lib (offset 12)
Load command 19
          cmd LC_RPATH
      cmdsize 112
         path /Users/ilg/.local/xbb/usr/arm64-apple-darwin20.6.0/lib/gcc/arm64-apple-darwin20.6.0/11.1.0/../../.. (offset 12)

So, the compiler libraries are relative to @rpath, which is a list of absolute paths inside the toolchain.

If you add your own -rpaths, then we assume that you are packaging as per the developer documentation, and then you have to supply all the necessary rpaths (and put the runtime libraries somewhere special, it's assumed).

Well, (I'm not adding my own -rpath, and it is not a matter that I like it or not), but anyway here is where I noticed that complications arise.

Case 1:

What happens if, in the previous scenario, the toolchain is removed?

$ ./hello
dyld: Library not loaded: @rpath/libgcc_s.1.1.dylib
  Referenced from: /Users/ilg/tmp/my-project/./hello
  Reason: image not found
zsh: abort      ./hello

If this is expected to work, the libraries must be copied in a folder with the elf and some paths adjusted.

Case 2:

What if I want to pack the hello application and distribute it to be executed on another system, where the custom toolchain is no longer installed in the same location, or not at all.

The absolute LC_RPATH entries need to be removed, the dynamic libraries referred via @rpath/xxx.dylib must be copied from the absolute paths somewhere into the distribution (I use libexec) and references to the dynamic libraries must be changed to something relative to @loader_path, or @executable_path.

This is exactly what my post-processing step does. I did not find an automated method to do this, so I hat to write a complicated shell script, invoking install_name_tool (which failed due to headerpad limitations, and brought us to this discussion).

If this approach is not correct, either from the GCC point of view, or macOS point of view, sure, I'm ready to reconsider.

For example I noticed that you add @loader_path to the LC_RPATH list and keep the libraries relative to @rpath, instead of making the libraries directly relative to @loader_path.

Q: Has this solution (libraries relative to @rpath and LC_RPATH including a path relative to @loader_path) any advantages over directly making the libraries relative to @loader_path and having no LC_RPATH at all?

nothing specific to aarch64-darwin.

Yes, rather something specific to recent macOS.

Given that you already pushed a fix for #52, I'll wait for @fxcoudert to update the HomeBrew branch and test if the bug was fixed.

Then I'll try to update my build environment from 11.1 to the very latest and greatest 11.2, and revise the post-processing script to make the best use of the new rpath mechanisms, but I'm not sure I can avoid using install_name_tool when processing g++-mapper-server.

If you have suggestions on how to improve/simplify things, sure, highly appreciated.

[iains]

Case 1:

What happens if, in the previous scenario, the toolchain is removed?

$ ./hello
dyld: Library not loaded: @rpath/libgcc_s.1.1.dylib
Referenced from: /Users/ilg/tmp/my-project/./hello
Reason: image not found
zsh: abort ./hello

Well that is no different from what happens if you delete the toolchain with absolute pathnames in the libraries.

Perhaps you can explain how your installation can do this?

... the general method used on macOS (i.e. as per the Developer Documentation on the topic) is as below.

Case 2:

What if I want to pack the hello application and distribute it to be executed on another >system, where the custom toolchain is no longer installed in the same location, or not at all.

The usual method is to package all the runtimes you are going to use (whether compiler or general libs is academic) alongside your exe (usually exe in the MacOS folder and libraries some adjacent folder in the application package

• but also people build unix-y stand-alone packages like:

my_oss_package/
bin/
my_exe
lib/
... lots of libs

then the exe has an rpath like @executable_path/../lib
and 'lots of libs' have @rpath/xxxx and maybe @loader_path to find dependents.

The compiler will honour this - if you supply any -rpath or -Wl,-rpaths it will suppress the automatic ones, and use the ones you provide.

I don't think there is any other generic way of doing it - e.g. absolute library paths require privs that your user might not have. I suppose you can install them all in ~/lib (but that makes a nightmare in managing versions). The system only searches a small set of places by default - and we already established we cannot use DYLD_xxxx to deal with this.

nothing specific to aarch64-darwin.
Yes, rather something specific to recent macOS.

No, I don't think so; the configuration problem lies with GCC in a generic sense (whether command line options reach the build for g++-mapper-server) it makes no difference what the target is ... (you have noticed it here, it just happens no-one else has [yet]).

[iains]

the bottom line is that a built exe refers to the dependent libraries

either

• using the /absolute/path/to/libxxx.dylib
• or by using rpath entries and @rpath/libyyy.dylib

unless you link statically (which is not always possible) if you move or delete the dependent libraries Bad Things happen, no real surprise - and, of course, you cannot assume those libraries exist on some other user's machine.

to make your application independently distributable (ie. remove the dependencies on the toolchain used to build it) - either you have to make sure you only use system-installed libraries (/usr/lib/xxxxx) [statically link everything else] or you have to package the extra libs with the application (e.g. as described in the Developer Doc.).

[ilg-ul]

as described in the Developer Doc

Oops, I read a few developer docs, but I missed the one with the recommended structure of @rpath. Do you remember what document was that?

So, the recommendation is to refer to dynamic libraries as @rpath/name.dylib, not @loader_path/name.dylib?

[iains]

I don't have an immediate link to it - 'using dynamic libraries' or something like that.

but neither makes a magic solution to the underlying issue - if you (re)move the dependent libs, then application is broken.

[ilg-ul]

but neither makes a magic solution to the underlying issue - if you (re)move the dependent libs, then application is broken.

I'm afraid this is a misunderstanding, the idea is not to remove anything, or to play with the configurations, or do nasty things to see what happens.

The very specific purpose is to build a standalone, multi-version, cross-platform native GCC distribution.

By cross-platform I mean there are separate binaries for Windows/macOS/Linux, in any reasonable 32/64-bit, Intel/Arm combinations, and a uniform mechanism to install the right one for the supported platform (via xpm install).

By standalone I mean the binaries have minimum dependencies on system libraries, and the archive can be unpacked in any location in the file-system and run from there, without any install step.

By multi-version I mean that multiple versions of the same toolchain can be installed in separate folders, and different projects can be locked to different versions, on the same system, at the same time. The typical use case for this requirement is to run unit-tests, which depend on specific toolchain versions, and either the same project needs to build the tests with multiple toolchain versions, or different projects need different toolchain versions.

The specific project that I'm referring to is The xPack GNU Compiler Collection (GCC), which already has 2 releases (https://github.com/xpack-dev-tools/gcc-xpack/releases), and which I'm now trying to expand to also support Apple Silicon..

The xPack GCC is expected to be used to build:

• any simple applications, where, as you mentioned, the details are not relevant, references to the toolchain libraries are acceptable, and the toolchain is expected to be present during the lifespan of the program (usually unit tests)
• future releases of other similar stand-alone packages, including the xPack GCC itself, when an elaborated build procedure will copy libraries into the destination and adjust paths.

So far I do not know of any other similar distribution to get inspiration from, so, if someone can provide any links, they'll be highly appreciated.

[iains]

this is the section I was thinking of - it explains @rpath/ installation names and some of the run path stuff (at least @loader_path)
.. man dyld explains the rest. I do not think I've seen anything where the installation name includes @loader_path ... I'm quite surprised that even works.

https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/RunpathDependentLibraries.html#//apple_ref/doc/uid/TP40008306-SW1

NOTE: also the caption:
"Note: Run-path dependent libraries can also be used as regular dependent libraries by specifying absolute pathnames instead of run-path–relative pathnames in -rpath clauses and ensuring that the libraries reside at the specified locations."

... which makes it safe to replace a dylib that used to have an absolute name, with one beginning with @rpath/ (which we want to allow for backwards-compatibility with toolchains installed at a fixed path).

[ilg-ul]

this is the section I was thinking of ...

Thank you, I'll take a look.

Now I'm rebuilding the development environments on all macOS versions to use the latest gcc-11.2.0-arm-20211126, and later I'll see if g++-mapper-server can be used as-is, or requires further post-processing.

But please note that although the development builds run on macOS 11.6, the Intel production builds will continue to run on macOS 10.13, and the build scrips, including the post-processing scripts, should accommodate all versions.

[fxcoudert]

I can confirm that Homebrew has now completely phased out the use of BOOT_LDFLAGS=-Wl,-headerpad_max_install_names as the compiler is relocatable.

[iains]

JFTR, I did look at the c++tools config and it seems to pass LDFLAGS - so the question would be why isn't that working.

Anyway - if there's a bug here - then it's not specific to the aarch64-darwin branch and we really should add it to GCC BZ and not here (but first we need a reproducer).

[iains]

is this issue still of any relevance?
if not, please can we close it?