macOS Catalina and C header files

[willcl-ark]

Message edited 2020-04-28

Issue and Steps to Reproduce

1. Upgrade your Mac to macOS Catalina 😔

./configure
make

Compiling seems to work after this -- usable binaries are created -- but there are still many lines in the output of make that I don't like the look of, even if they only seem to be for auto-generated tests...

System config:

Apple clang version 11.0.3 (clang-1103.0.32.29)
GNU bash, version 5.0.11(1)-release (x86_64-apple-darwin19.0.0)
automake (GNU automake) 1.16.1
autoconf (GNU Autoconf) 2.69
gettext (GNU gettext-runtime) 0.20.1
GNU Make 3.81
sqlite3 3.28.0 2019-04-15

Output error:

...

checking for ANSI C header files... Undefined symbols for architecture x86_64:
  "_fromwire_amount_msat", referenced from:
      _fromwire_tlv_test_n1_tlv3 in ccj4zKdV.o
  "_fromwire_bool", referenced from:
      _fromwire_test_msg in ccj4zKdV.o
      _fromwire_test_msg_option_short_id in ccj4zKdV.o
      _fromwire_test_msg_option_one in ccj4zKdV.o
      _fromwire_test_msg_option_two in ccj4zKdV.o
  "_fromwire_test_enum", referenced from:
      _fromwire_test_msg in ccj4zKdV.o
      _fromwire_test_msg_option_short_id in ccj4zKdV.o
      _fromwire_test_msg_option_one in ccj4zKdV.o
      _fromwire_test_msg_option_two in ccj4zKdV.o
  "_fromwire_tlvs", referenced from:
      _fromwire_test_tlv1 in ccj4zKdV.o
      _fromwire_test_tlv2 in ccj4zKdV.o
      _fromwire_test_tlv3 in ccj4zKdV.o
  "_fromwire_tu32", referenced from:
      _fromwire_tlv_test_n2_tlv2 in ccj4zKdV.o
  "_fromwire_tu64", referenced from:
      _fromwire_tlv_test_n1_tlv1 in ccj4zKdV.o
      _fromwire_tlv_test_n2_tlv1 in ccj4zKdV.o
  "_fromwire_u16", referenced from:
      _fromwire_test_features in ccj4zKdV.o
      _fromwire_subtype_var_assign in ccj4zKdV.o
      _fromwire_subtype_arrays in ccj4zKdV.o
      _fromwire_tlv_test_n1_tlv4 in ccj4zKdV.o
      _fromwire_tlv_test_n3_tlv3 in ccj4zKdV.o
      _fromwire_test_msg in ccj4zKdV.o
      _fromwire_test_tlv1 in ccj4zKdV.o
      ...
  "_fromwire_u32", referenced from:
      _fromwire_tlv_test_n3_tlv3 in ccj4zKdV.o
      _fromwire_test_msg in ccj4zKdV.o
      _fromwire_test_msg_option_short_id in ccj4zKdV.o
      _fromwire_test_msg_option_one in ccj4zKdV.o
      _fromwire_test_msg_option_two in ccj4zKdV.o
  "_fromwire_u64", referenced from:
      _fromwire_test_short_id in ccj4zKdV.o
      _fromwire_tlv_test_n3_tlv3 in ccj4zKdV.o
  "_fromwire_u8", referenced from:
      _fromwire_subtype_var_assign in ccj4zKdV.o
      _fromwire_subtype_var_len in ccj4zKdV.o
      _fromwire_subtype_varlen_varsize in ccj4zKdV.o
      _fromwire_tlv_test_n3_tlv3 in ccj4zKdV.o
  "_fromwire_u8_array", referenced from:
      _fromwire_test_features in ccj4zKdV.o
      _fromwire_subtype_arrays in ccj4zKdV.o
      _fromwire_tlv_test_n3_tlv3 in ccj4zKdV.o
      _fromwire_test_msg in ccj4zKdV.o
      _fromwire_test_msg_option_short_id in ccj4zKdV.o
      _fromwire_test_msg_option_one in ccj4zKdV.o
      _fromwire_test_msg_option_two in ccj4zKdV.o
      ...
  "_main", referenced from:
     implicit entry/start for main executable
  "_tal_alloc_", referenced from:
      _fromwire_test_features in ccj4zKdV.o
      _fromwire_subtype_var_assign in ccj4zKdV.o
      _fromwire_subtype_var_len in ccj4zKdV.o
      _fromwire_subtype_varlen_varsize in ccj4zKdV.o
      _tlv_test_n1_new in ccj4zKdV.o
      _fromwire_tlv_test_n1_tlv1 in ccj4zKdV.o
      _fromwire_tlv_test_n1_tlv2 in ccj4zKdV.o
      ...
  "_tal_alloc_arr_", referenced from:
      _fromwire_test_features in ccj4zKdV.o
      _fromwire_subtype_var_assign in ccj4zKdV.o
      _fromwire_subtype_var_len in ccj4zKdV.o
      _fromwire_subtype_varlen_varsize in ccj4zKdV.o
      _towire_tlv_test_n1_tlv1 in ccj4zKdV.o
      _towire_tlv_test_n1_tlv2 in ccj4zKdV.o
      _towire_tlv_test_n1_tlv3 in ccj4zKdV.o
      ...
  "_tal_bytelen", referenced from:
      _towire_test_features in ccj4zKdV.o
      _towire_subtype_var_assign in ccj4zKdV.o
      _towire_subtype_var_len in ccj4zKdV.o
      _towire_subtype_varlen_varsize in ccj4zKdV.o
      _towire_tlv_test_n3_tlv3 in ccj4zKdV.o
      _towire_test_msg in ccj4zKdV.o
      _fromwire_test_msg in ccj4zKdV.o
      ...
  "_towire_amount_msat", referenced from:
      _towire_tlv_test_n1_tlv3 in ccj4zKdV.o
  "_towire_bool", referenced from:
      _towire_test_msg in ccj4zKdV.o
      _towire_test_msg_option_short_id in ccj4zKdV.o
      _towire_test_msg_option_one in ccj4zKdV.o
      _towire_test_msg_option_two in ccj4zKdV.o
  "_towire_test_enum", referenced from:
      _towire_test_msg in ccj4zKdV.o
      _towire_test_msg_option_short_id in ccj4zKdV.o
      _towire_test_msg_option_one in ccj4zKdV.o
      _towire_test_msg_option_two in ccj4zKdV.o
  "_towire_tlvs", referenced from:
      _towire_test_tlv1 in ccj4zKdV.o
      _towire_test_tlv2 in ccj4zKdV.o
      _towire_test_tlv3 in ccj4zKdV.o
  "_towire_tu32", referenced from:
      _towire_tlv_test_n2_tlv2 in ccj4zKdV.o
  "_towire_tu64", referenced from:
      _towire_tlv_test_n1_tlv1 in ccj4zKdV.o
      _towire_tlv_test_n2_tlv1 in ccj4zKdV.o
  "_towire_u16", referenced from:
      _towire_test_features in ccj4zKdV.o
      _towire_subtype_var_assign in ccj4zKdV.o
      _towire_subtype_arrays in ccj4zKdV.o
      _towire_tlv_test_n1_tlv4 in ccj4zKdV.o
      _towire_tlv_test_n3_tlv3 in ccj4zKdV.o
      _towire_test_msg in ccj4zKdV.o
      _towire_test_tlv1 in ccj4zKdV.o
      ...
  "_towire_u32", referenced from:
      _towire_tlv_test_n3_tlv3 in ccj4zKdV.o
      _towire_test_msg in ccj4zKdV.o
      _towire_test_msg_option_short_id in ccj4zKdV.o
      _towire_test_msg_option_one in ccj4zKdV.o
      _towire_test_msg_option_two in ccj4zKdV.o
  "_towire_u64", referenced from:
      _towire_test_short_id in ccj4zKdV.o
      _towire_tlv_test_n3_tlv3 in ccj4zKdV.o
  "_towire_u8", referenced from:
      _towire_subtype_var_assign in ccj4zKdV.o
      _towire_subtype_var_len in ccj4zKdV.o
      _towire_subtype_varlen_varsize in ccj4zKdV.o
      _towire_tlv_test_n3_tlv3 in ccj4zKdV.o
  "_towire_u8_array", referenced from:
      _towire_test_features in ccj4zKdV.o
      _towire_subtype_arrays in ccj4zKdV.o
      _towire_tlv_test_n3_tlv3 in ccj4zKdV.o
      _towire_test_msg in ccj4zKdV.o
      _towire_test_msg_option_short_id in ccj4zKdV.o
      _towire_test_msg_option_one in ccj4zKdV.o
      _towire_test_msg_option_two in ccj4zKdV.o
      ...
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
make[1]: *** [tools/test/gen_test.c.tmp] Error 1
make: *** [tools/test/gen_test.c] Error 1

It seems that these headers are supposed to be generated by tools/generate-wire.py during compilation... Is it trying to put them in /usr/include by any chance? This location does not now exist on Catalina and I don't know if it's writable...

Full compile log:
configure.txt
make.txt

[willcl-ark]

I have updated the above description as it's still present on master with MacOS Catalina.

Possibly ping @niftynei as I see your name at the top of tools/generate-wire.py :) Can you help me find where this file is searching for these apparently missing headers?

[niftynei]

So the errors in the message are from a generated test source files (that are useful for checking that generate-wire.py is working as intended). There's a make step that's supposed to fill in mocks for all of those missing linked methods but given the make output that you've posted it looks like the LD failure output has been updated, which is breaking our auto-mocking script.

I put a small patch up for this in #3756, can you check if this fixes the problem?

[Moller40]

Get similar error still with macOS Mojave, but only if I first configure with --enable-developer

System config:

Apple LLVM version 10.0.1 (clang-1001.0.46.4)
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin18)
automake (GNU automake) 1.16.1
autoconf (GNU Autoconf) 2.69
gettext (GNU gettext-runtime) 0.21
GNU Make 3.81
sqlite3 3.24.0 2018-06-04

Error:

tools/generate-wire.py --page impl tools/test/gen_test.h test_type < tools/test/test_cases > tools/test/gen_test.c.tmp.c
mocking out tools/test/gen_test.c.tmp.c:
Undefined symbols for architecture x86_64:
  "_fromwire_amount_msat", referenced from:
      _fromwire_tlv_test_n1_tlv3 in gen_test-fe786f.o
  "_fromwire_bigsize", referenced from:
      _fromwire_test_n1 in gen_test-fe786f.o
      _fromwire_test_n2 in gen_test-fe786f.o
      _fromwire_test_n3 in gen_test-fe786f.o
  "_fromwire_bool", referenced from:
      _fromwire_test_msg in gen_test-fe786f.o
      _fromwire_test_msg_option_short_id in gen_test-fe786f.o
      _fromwire_test_msg_option_one in gen_test-fe786f.o
      _fromwire_test_msg_option_two in gen_test-fe786f.o
  "_fromwire_fail", referenced from:
      _fromwire_test_n1 in gen_test-fe786f.o
      _fromwire_test_n2 in gen_test-fe786f.o
      _fromwire_test_n3 in gen_test-fe786f.o
  "_fromwire_test_enum", referenced from:
      _fromwire_test_msg in gen_test-fe786f.o
      _fromwire_test_msg_option_short_id in gen_test-fe786f.o
      _fromwire_test_msg_option_one in gen_test-fe786f.o
      _fromwire_test_msg_option_two in gen_test-fe786f.o
  "_fromwire_tu32", referenced from:
      _fromwire_tlv_test_n2_tlv2 in gen_test-fe786f.o
  "_fromwire_tu64", referenced from:
      _fromwire_tlv_test_n1_tlv1 in gen_test-fe786f.o
      _fromwire_tlv_test_n2_tlv1 in gen_test-fe786f.o
  "_fromwire_u16", referenced from:
      _fromwire_test_msg in gen_test-fe786f.o
      _fromwire_test_features in gen_test-fe786f.o
      _fromwire_subtype_var_assign in gen_test-fe786f.o
      _fromwire_subtype_arrays in gen_test-fe786f.o
      _fromwire_test_tlv1 in gen_test-fe786f.o
      _fromwire_test_tlv2 in gen_test-fe786f.o
      _fromwire_test_tlv3 in gen_test-fe786f.o
      ...
  "_fromwire_u32", referenced from:
      _fromwire_test_msg in gen_test-fe786f.o
      _fromwire_test_msg_option_short_id in gen_test-fe786f.o
      _fromwire_test_msg_option_one in gen_test-fe786f.o
      _fromwire_test_msg_option_two in gen_test-fe786f.o
      _fromwire_tlv_test_n3_tlv3 in gen_test-fe786f.o
  "_fromwire_u64", referenced from:
      _fromwire_test_short_id in gen_test-fe786f.o
      _fromwire_tlv_test_n3_tlv3 in gen_test-fe786f.o
  "_fromwire_u8", referenced from:
      _fromwire_subtype_var_assign in gen_test-fe786f.o
      _fromwire_subtype_var_len in gen_test-fe786f.o
      _fromwire_subtype_varlen_varsize in gen_test-fe786f.o
      _fromwire_tlv_test_n3_tlv3 in gen_test-fe786f.o
  "_fromwire_u8_array", referenced from:
      _fromwire_test_msg in gen_test-fe786f.o
      _fromwire_test_features in gen_test-fe786f.o
      _fromwire_subtype_arrays in gen_test-fe786f.o
      _fromwire_test_msg_option_short_id in gen_test-fe786f.o
      _fromwire_test_msg_option_one in gen_test-fe786f.o
      _fromwire_test_msg_option_two in gen_test-fe786f.o
      _fromwire_tlv_test_n3_tlv3 in gen_test-fe786f.o
      ...
  "_main", referenced from:
     implicit entry/start for main executable
  "_tal_alloc_", referenced from:
      _tlv_test_n1_new in gen_test-fe786f.o
      _tlv_test_n2_new in gen_test-fe786f.o
      _tlv_test_n3_new in gen_test-fe786f.o
      _fromwire_test_msg in gen_test-fe786f.o
      _fromwire_test_features in gen_test-fe786f.o
      _fromwire_subtype_var_assign in gen_test-fe786f.o
      _fromwire_subtype_var_len in gen_test-fe786f.o
      ...
  "_tal_alloc_arr_", referenced from:
      _tlv_test_n1_new in gen_test-fe786f.o
      _tlv_test_n2_new in gen_test-fe786f.o
      _tlv_test_n3_new in gen_test-fe786f.o
      _towire_test_msg in gen_test-fe786f.o
      _fromwire_test_msg in gen_test-fe786f.o
      _fromwire_test_features in gen_test-fe786f.o
      _fromwire_subtype_var_assign in gen_test-fe786f.o
      ...
  "_tal_bytelen", referenced from:
      _towire_test_n1 in gen_test-fe786f.o
      _fromwire_test_n1 in gen_test-fe786f.o
      _test_n1_is_valid in gen_test-fe786f.o
      _towire_test_n2 in gen_test-fe786f.o
      _fromwire_test_n2 in gen_test-fe786f.o
      _test_n2_is_valid in gen_test-fe786f.o
      _towire_test_n3 in gen_test-fe786f.o
      ...
  "_tal_dup_", referenced from:
      _fromwire_test_n1 in gen_test-fe786f.o
      _fromwire_test_n2 in gen_test-fe786f.o
      _fromwire_test_n3 in gen_test-fe786f.o
  "_tal_free", referenced from:
      _towire_test_n1 in gen_test-fe786f.o
      _towire_test_n2 in gen_test-fe786f.o
      _towire_test_n3 in gen_test-fe786f.o
  "_tal_resize_", referenced from:
      _fromwire_test_n1 in gen_test-fe786f.o
      _fromwire_test_n2 in gen_test-fe786f.o
      _fromwire_test_n3 in gen_test-fe786f.o
      _fromwire_subtype_var_assign in gen_test-fe786f.o
      _fromwire_subtype_var_len in gen_test-fe786f.o
      _fromwire_subtype_varlen_varsize in gen_test-fe786f.o
      _fromwire_subtype_arrays in gen_test-fe786f.o
      ...
  "_towire", referenced from:
      _towire_test_n1 in gen_test-fe786f.o
      _towire_test_n2 in gen_test-fe786f.o
      _towire_test_n3 in gen_test-fe786f.o
     (maybe you meant: _towire_test_msg_option_two, _towire_test_msg_option_one , _towire_test_msg_option_short_id , _towire_test_tlv1 , _towire_test_n3 , _towire_test_msg , _towire_test_n2 , _towire_test_tlv2 , _towire_test_tlv3 , _towire_test_n1 )
  "_towire_amount_msat", referenced from:
      _towire_tlv_test_n1_tlv3 in gen_test-fe786f.o
  "_towire_bigsize", referenced from:
      _towire_test_n1 in gen_test-fe786f.o
      _towire_test_n2 in gen_test-fe786f.o
      _towire_test_n3 in gen_test-fe786f.o
  "_towire_bool", referenced from:
      _towire_test_msg in gen_test-fe786f.o
      _towire_test_msg_option_short_id in gen_test-fe786f.o
      _towire_test_msg_option_one in gen_test-fe786f.o
      _towire_test_msg_option_two in gen_test-fe786f.o
  "_towire_test_enum", referenced from:
      _towire_test_msg in gen_test-fe786f.o
      _towire_test_msg_option_short_id in gen_test-fe786f.o
      _towire_test_msg_option_one in gen_test-fe786f.o
      _towire_test_msg_option_two in gen_test-fe786f.o
  "_towire_tu32", referenced from:
      _towire_tlv_test_n2_tlv2 in gen_test-fe786f.o
  "_towire_tu64", referenced from:
      _towire_tlv_test_n1_tlv1 in gen_test-fe786f.o
      _towire_tlv_test_n2_tlv1 in gen_test-fe786f.o
  "_towire_u16", referenced from:
      _towire_test_msg in gen_test-fe786f.o
      _towire_test_features in gen_test-fe786f.o
      _towire_subtype_var_assign in gen_test-fe786f.o
      _towire_subtype_arrays in gen_test-fe786f.o
      _towire_test_tlv1 in gen_test-fe786f.o
      _towire_test_tlv2 in gen_test-fe786f.o
      _towire_test_tlv3 in gen_test-fe786f.o
      ...
  "_towire_u32", referenced from:
      _towire_test_msg in gen_test-fe786f.o
      _towire_test_msg_option_short_id in gen_test-fe786f.o
      _towire_test_msg_option_one in gen_test-fe786f.o
      _towire_test_msg_option_two in gen_test-fe786f.o
      _towire_tlv_test_n3_tlv3 in gen_test-fe786f.o
  "_towire_u64", referenced from:
      _towire_test_short_id in gen_test-fe786f.o
      _towire_tlv_test_n3_tlv3 in gen_test-fe786f.o
  "_towire_u8", referenced from:
      _towire_subtype_var_assign in gen_test-fe786f.o
      _towire_subtype_var_len in gen_test-fe786f.o
      _towire_subtype_varlen_varsize in gen_test-fe786f.o
      _towire_tlv_test_n3_tlv3 in gen_test-fe786f.o
  "_towire_u8_array", referenced from:
      _towire_test_msg in gen_test-fe786f.o
      _towire_test_features in gen_test-fe786f.o
      _towire_subtype_arrays in gen_test-fe786f.o
      _towire_test_msg_option_short_id in gen_test-fe786f.o
      _towire_test_msg_option_one in gen_test-fe786f.o
      _towire_test_msg_option_two in gen_test-fe786f.o
      _towire_tlv_test_n3_tlv3 in gen_test-fe786f.o
      ...
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [tools/test/gen_test.c.tmp] Error 1
make: *** [tools/test/gen_test.c] Error 1
rm tools/test/gen_test.c.tmp.c

[willcl-ark]

@Moller40 which branch are you building? Since #3756 got merged, I have not experienced this again, but perhaps you don't have that patch in whichever branch you're using?

Edit: I've just done a make distclean && ./configure --enable-developer && make on branch v0.9.1 and all seems to be well still (Catalina 10.15.6 (19G2021))

[Moller40]

Building master with #3756 included.
But with Mojave 10.14.6 (18G6020)

[willcl-ark]

Closing as stale. Let me know if you want it re-opened.