Unable to build arm64 binaries (segfault during v8 bytecode_builtins_list_generator) #17100

timonwong · 2021-06-23T10:27:52Z

Hello,
I'm encountering a strange error when building arm64(aarch64) binaries (It's worked for amd64 build though):

Using following command from https://github.com/envoyproxy/envoy/tree/main/ci#on-linux:

./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.release.server_only'

And during wee8 building, there is a strange error:

[130/1236] CXX obj/third_party/inspector_protocol/crdtp/cbor.o
[131/1236] LINK ./bytecode_builtins_list_generator
[132/1236] ACTION //:generate_bytecode_builtins_list(//build/toolchain/linux/unbundle:default)
FAILED: gen/builtins-generated/bytecodes-builtins-list.h
python ../../tools/run.py ./bytecode_builtins_list_generator gen/builtins-generated/bytecodes-builtins-list.h
Return code is -11
[133/1236] CXX obj/src/inspector/inspector/Console.o
[134/1236] CXX obj/third_party/inspector_protocol/crdtp/dispatch.o
[135/1236] CXX obj/src/inspector/inspector/Profiler.o
[136/1236] CXX obj/src/inspector/inspector/Schema.o
[137/1236] CXX obj/src/inspector/inspector/HeapProfiler.o
[138/1236] CXX obj/torque_base/implementation-visitor.o
[139/1236] CXX obj/src/inspector/inspector/Debugger.o
[140/1236] CXX obj/src/inspector/inspector/Runtime.o

Which is a segmentation fault without any additional messages.

Envoy Branch: release/v1.18

Machine Information:
AWS Graviton: c6g.2xlarge

The text was updated successfully, but these errors were encountered:

github-actions · 2021-07-23T16:01:25Z

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

github-actions · 2021-07-30T20:01:13Z

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

maikoulin · 2021-11-24T09:42:10Z

I have encountered this problem, have you solved it?

timonwong · 2022-01-05T05:57:47Z

I have encountered this problem, have you solved it?

Nope, it's still segement fault...

moderation · 2022-01-05T17:37:47Z

I'm building on aarch64 Linux on a Raspberry Pi and MacOS M1. I don't use Docker to build and I compile out a bunch of functionality I don't need include the WASM support that causes your segfault. From my ~/.bazelrc

build --define wasm=disabled

More info at https://github.com/envoyproxy/envoy/tree/main/bazel#enabling-and-disabling-extensions

This doesn't fix the segfault and is a work around however this approach will unblock you if you don't require WASM filters

timonwong · 2022-01-06T08:31:41Z

@moderation thanks, I did have success building without any wasm support for arm64 builds. But we are looking for build with wasm support...

The last update is I have successfully built wee8 under huaweicloud Kunpeng. (still cannot build with AWS Graviton, very strange...)

moderation · 2022-01-06T16:59:23Z

Any ideas for WASM support for aarch64 @lizan @PiotrSikora @mathetake ?

PiotrSikora · 2022-01-06T17:12:39Z

It builds and passes all the tests on the CI on Linux-aarch64, so I have no idea what could be the issue. Perhaps too little memory on RPi?

You could try building from #19275. Maybe that will work, or at least provide a more helpful error.

timonwong · 2022-01-07T04:28:49Z

@PiotrSikora
I think it's not a build system issue, maybe some hardware or host machine operating system related problem:
I just copied the bytecode_builtins_list_generator binary to the environment which fails, it still gives me SIGSEGV even without giving any command line arguments.

It should:

# ./bytecode_builtins_list_generator
Usage: ./bytecode_builtins_list_generator <output filename>

However it produces:

# ./bytecode_builtins_list_generator
Segmentation fault

timonwong · 2022-01-07T04:55:05Z

It's interesting that on machine that's ok running ldd(one fail) gives me:

objdump returns the same result:

# objdump -p ./bytecode_builtins_list_generator

./bytecode_builtins_list_generator:     file format elf64-littleaarch64

Program Header:
    PHDR off    0x0000000000000040 vaddr 0x0000000000000040 paddr 0x0000000000000040 align 2**3
         filesz 0x00000000000002a0 memsz 0x00000000000002a0 flags r--
  INTERP off    0x00000000000002e0 vaddr 0x00000000000002e0 paddr 0x00000000000002e0 align 2**0
         filesz 0x000000000000001b memsz 0x000000000000001b flags r--
    LOAD off    0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**12
         filesz 0x0000000000034f1c memsz 0x0000000000034f1c flags r--
    LOAD off    0x0000000000034f20 vaddr 0x0000000000035f20 paddr 0x0000000000035f20 align 2**12
         filesz 0x000000000001c020 memsz 0x000000000001c020 flags r-x
    LOAD off    0x0000000000050f40 vaddr 0x0000000000052f40 paddr 0x0000000000052f40 align 2**12
         filesz 0x00000000000027a8 memsz 0x00000000000027a8 flags rw-
    LOAD off    0x00000000000536e8 vaddr 0x00000000000566e8 paddr 0x00000000000566e8 align 2**12
         filesz 0x0000000000000018 memsz 0x0000000000000100 flags rw-
     TLS off    0x0000000000050f40 vaddr 0x0000000000051f40 paddr 0x0000000000051f40 align 2**3
         filesz 0x0000000000000000 memsz 0x0000000000000010 flags r--
 DYNAMIC off    0x0000000000053010 vaddr 0x0000000000055010 paddr 0x0000000000055010 align 2**3
         filesz 0x00000000000001f0 memsz 0x00000000000001f0 flags rw-
   RELRO off    0x0000000000050f40 vaddr 0x0000000000052f40 paddr 0x0000000000052f40 align 2**0
         filesz 0x00000000000027a8 memsz 0x00000000000030c0 flags r--
EH_FRAME off    0x000000000002de60 vaddr 0x000000000002de60 paddr 0x000000000002de60 align 2**2
         filesz 0x00000000000017f4 memsz 0x00000000000017f4 flags r--
   STACK off    0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**0
         filesz 0x0000000000000000 memsz 0x0000000000000000 flags rw-
    NOTE off    0x00000000000002fc vaddr 0x00000000000002fc paddr 0x00000000000002fc align 2**2
         filesz 0x0000000000000038 memsz 0x0000000000000038 flags r--

Dynamic Section:
  NEEDED               libpthread.so.0
  NEEDED               libstdc++.so.6
  NEEDED               libgcc_s.so.1
  NEEDED               libc.so.6
  NEEDED               ld-linux-aarch64.so.1
  FLAGS                0x0000000000000008
  FLAGS_1              0x0000000000000001
  DEBUG                0x0000000000000000
  RELA                 0x0000000000024208
  RELASZ               0x00000000000062b8
  RELAENT              0x0000000000000018
  RELACOUNT            0x0000000000000406
  JMPREL               0x000000000002a4c0
  PLTRELSZ             0x0000000000000be8
  PLTGOT               0x00000000000552d8
  PLTREL               0x0000000000000007
  SYMTAB               0x0000000000000338
  SYMENT               0x0000000000000018
  STRTAB               0x0000000000009bf0
  STRSZ                0x000000000001a618
  GNU_HASH             0x0000000000007f10
  INIT_ARRAY           0x0000000000052f48
  INIT_ARRAYSZ         0x0000000000000010
  FINI_ARRAY           0x0000000000052f40
  FINI_ARRAYSZ         0x0000000000000008
  INIT                 0x0000000000051708
  FINI                 0x000000000005171c
  VERSYM               0x00000000000074a8
  VERNEED              0x0000000000007e1c
  VERNEEDNUM           0x0000000000000005

Version References:
  required from libpthread.so.0:
    0x06969197 0x00 07 GLIBC_2.17
  required from libstdc++.so.6:
    0x08922974 0x00 04 GLIBCXX_3.4
    0x0297f864 0x00 08 GLIBCXX_3.4.14
    0x0297f868 0x00 09 GLIBCXX_3.4.18
    0x0297f871 0x00 03 GLIBCXX_3.4.21
    0x056bafd3 0x00 06 CXXABI_1.3
  required from libgcc_s.so.1:
    0x0b792650 0x00 10 GCC_3.0
    0x09276060 0x00 11 GCC_4.2.0
  required from libc.so.6:
    0x06969197 0x00 02 GLIBC_2.17
  required from ld-linux-aarch64.so.1:
    0x06969197 0x00 05 GLIBC_2.17
private flags = 0:

# ldd ./bytecode_builtins_list_generator
        linux-vdso.so.1 (0x0000ffff9f401000)
        libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000ffff9f352000)
        libstdc++.so.6 => /usr/lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000ffff9f1be000)
        libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000ffff9f19a000)
        libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff9f041000)
        /lib/ld-linux-aarch64.so.1 (0x0000ffff9f3d5000)
        libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffff9ef88000)

However on break one:

# ldd ./bytecode_builtins_list_generator
not a dynamic executable

UPDATE: Using strace

On node which is ok:

# strace ./bytecode_builtins_list_generator
execve("/root/wee8/out/wee8/bytecode_builtins_list_generator", ["/root/wee8/out/wee8/bytecode_bui"...], 0xffffd69b60b0 /* 31 vars */) = 0
brk(NULL)                               = 0xaaab114f8000
faccessat(AT_FDCWD, "/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
faccessat(AT_FDCWD, "/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=36263, ...}) = 0
mmap(NULL, 36263, PROT_READ, MAP_PRIVATE, 3, 0) = 0xffff8605b000
close(3)                                = 0
faccessat(AT_FDCWD, "/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/aarch64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0\330\\\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=137872, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffff86059000
mmap(NULL, 177104, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xffff8600d000
mprotect(0xffff86024000, 61440, PROT_NONE) = 0
mmap(0xffff86033000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0xffff86033000
mmap(0xffff86035000, 13264, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xffff86035000
close(3)                                = 0
faccessat(AT_FDCWD, "/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/aarch64-linux-gnu/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0\20\317\10\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=1578656, ...}) = 0
mmap(NULL, 1654416, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xffff85e79000
mprotect(0xffff85fee000, 65536, PROT_NONE) = 0
mmap(0xffff85ffe000, 49152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x175000) = 0xffff85ffe000
mmap(0xffff8600a000, 11920, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xffff8600a000
close(3)                                = 0
faccessat(AT_FDCWD, "/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/aarch64-linux-gnu/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0\20)\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=80168, ...}) = 0
mmap(NULL, 144440, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xffff85e55000
mprotect(0xffff85e68000, 61440, PROT_NONE) = 0
mmap(0xffff85e77000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x12000) = 0xffff85e77000
close(3)                                = 0
faccessat(AT_FDCWD, "/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/aarch64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0`\10\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1341248, ...}) = 0
mmap(NULL, 1409880, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xffff85cfc000
mprotect(0xffff85e3b000, 65536, PROT_NONE) = 0
mmap(0xffff85e4b000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13f000) = 0xffff85e4b000
mmap(0xffff85e51000, 13144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xffff85e51000
close(3)                                = 0
faccessat(AT_FDCWD, "/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/aarch64-linux-gnu/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0@\257\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=691544, ...}) = 0
mmap(NULL, 753784, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xffff85c43000
mprotect(0xffff85ceb000, 61440, PROT_NONE) = 0
mmap(0xffff85cfa000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa7000) = 0xffff85cfa000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffff86057000
mprotect(0xffff85e4b000, 16384, PROT_READ) = 0
mprotect(0xffff85cfa000, 4096, PROT_READ) = 0
mprotect(0xffff85e77000, 4096, PROT_READ) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffff85c41000
mprotect(0xffff85ffe000, 40960, PROT_READ) = 0
mprotect(0xffff86033000, 4096, PROT_READ) = 0
mprotect(0xaaaae62bd000, 16384, PROT_READ) = 0
mprotect(0xffff86066000, 4096, PROT_READ) = 0
munmap(0xffff8605b000, 36263)           = 0
set_tid_address(0xffff86057dc0)         = 4172
set_robust_list(0xffff86057dd0, 24)     = 0
rt_sigaction(SIGRTMIN, {sa_handler=0xffff86012760, sa_mask=[], sa_flags=SA_SIGINFO}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {sa_handler=0xffff86012828, sa_mask=[], sa_flags=SA_RESTART|SA_SIGINFO}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
brk(NULL)                               = 0xaaab114f8000
brk(0xaaab11519000)                     = 0xaaab11519000
futex(0xffff8600b060, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0xffff8600b064, FUTEX_WAKE_PRIVATE, 2147483647) = 0
write(2, "Usage: ", 7Usage: )                  = 7
write(2, "/root/wee8/out/wee8/bytecode_bui"..., 52/root/wee8/out/wee8/bytecode_builtins_list_generator) = 52
write(2, " <output filename>\n", 19 <output filename>
)    = 19
exit_group(1)                           = ?
+++ exited with 1 +++
root@ecs-kc1-large-2-linux-20220106145753:/tmp# cd /root/wee8/out/weei
-bash: cd: /root/wee8/out/weei: No such file or directory
root@ecs-kc1-large-2-linux-20220106145753:/tmp# cd /root/wee8/out/wee8
root@ecs-kc1-large-2-linux-20220106145753:~/wee8/out/wee8# strace ./bytecode_builtins_list_generator
execve("./bytecode_builtins_list_generator", ["./bytecode_builtins_list_generat"...], 0xffffc693c950 /* 31 vars */) = 0
brk(NULL)                               = 0xaaab00d76000
faccessat(AT_FDCWD, "/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
faccessat(AT_FDCWD, "/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=36263, ...}) = 0
mmap(NULL, 36263, PROT_READ, MAP_PRIVATE, 3, 0) = 0xffff8d827000
close(3)                                = 0
faccessat(AT_FDCWD, "/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/aarch64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0\330\\\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=137872, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffff8d825000
mmap(NULL, 177104, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xffff8d7d9000
mprotect(0xffff8d7f0000, 61440, PROT_NONE) = 0
mmap(0xffff8d7ff000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0xffff8d7ff000
mmap(0xffff8d801000, 13264, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xffff8d801000
close(3)                                = 0
faccessat(AT_FDCWD, "/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/aarch64-linux-gnu/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0\20\317\10\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=1578656, ...}) = 0
mmap(NULL, 1654416, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xffff8d645000
mprotect(0xffff8d7ba000, 65536, PROT_NONE) = 0
mmap(0xffff8d7ca000, 49152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x175000) = 0xffff8d7ca000
mmap(0xffff8d7d6000, 11920, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xffff8d7d6000
close(3)                                = 0
faccessat(AT_FDCWD, "/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/aarch64-linux-gnu/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0\20)\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=80168, ...}) = 0
mmap(NULL, 144440, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xffff8d621000
mprotect(0xffff8d634000, 61440, PROT_NONE) = 0
mmap(0xffff8d643000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x12000) = 0xffff8d643000
close(3)                                = 0
faccessat(AT_FDCWD, "/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/aarch64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0`\10\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1341248, ...}) = 0
mmap(NULL, 1409880, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xffff8d4c8000
mprotect(0xffff8d607000, 65536, PROT_NONE) = 0
mmap(0xffff8d617000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13f000) = 0xffff8d617000
mmap(0xffff8d61d000, 13144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xffff8d61d000
close(3)                                = 0
faccessat(AT_FDCWD, "/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/aarch64-linux-gnu/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\267\0\1\0\0\0@\257\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=691544, ...}) = 0
mmap(NULL, 753784, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xffff8d40f000
mprotect(0xffff8d4b7000, 61440, PROT_NONE) = 0
mmap(0xffff8d4c6000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa7000) = 0xffff8d4c6000
close(3)                                = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffff8d823000
mprotect(0xffff8d617000, 16384, PROT_READ) = 0
mprotect(0xffff8d4c6000, 4096, PROT_READ) = 0
mprotect(0xffff8d643000, 4096, PROT_READ) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xffff8d40d000
mprotect(0xffff8d7ca000, 40960, PROT_READ) = 0
mprotect(0xffff8d7ff000, 4096, PROT_READ) = 0
mprotect(0xaaaae9896000, 16384, PROT_READ) = 0
mprotect(0xffff8d832000, 4096, PROT_READ) = 0
munmap(0xffff8d827000, 36263)           = 0
set_tid_address(0xffff8d823dc0)         = 4177
set_robust_list(0xffff8d823dd0, 24)     = 0
rt_sigaction(SIGRTMIN, {sa_handler=0xffff8d7de760, sa_mask=[], sa_flags=SA_SIGINFO}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {sa_handler=0xffff8d7de828, sa_mask=[], sa_flags=SA_RESTART|SA_SIGINFO}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
brk(NULL)                               = 0xaaab00d76000
brk(0xaaab00d97000)                     = 0xaaab00d97000
futex(0xffff8d7d7060, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0xffff8d7d7064, FUTEX_WAKE_PRIVATE, 2147483647) = 0
write(2, "Usage: ", 7Usage: )                  = 7
write(2, "./bytecode_builtins_list_generat"..., 34./bytecode_builtins_list_generator) = 34
write(2, " <output filename>\n", 19 <output filename>
)    = 19
exit_group(1)                           = ?
+++ exited with 1 +++

On node which fails:

# strace ./bytecode_builtins_list_generator
execve("./bytecode_builtins_list_generator", ["./bytecode_builtins_list_generat"...], 0xffffdb1b0870 /* 14 vars */) = -1 EINVAL (Invalid argument)
--- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=NULL} ---
+++ killed by SIGSEGV +++

PiotrSikora · 2022-01-07T05:19:50Z

@timonwong thanks for investigating!

I think it's not a build system issue, maybe some hardware or host machine operating system related problem:

It's not only about the build system, but those builds are quite different in terms of compiler flags, etc., so you never know.

Could you see if debug build helps? Perhaps run ./bytecode_builtins_list_generator under gdb?

On the main branch:

$ bazel build -c dbg @com_googlesource_chromium_v8//:build

In #19275:

$ bazel build -c dbg @v8//:wee8

timonwong · 2022-01-07T09:32:45Z

@PiotrSikora

Could you see if debug build helps? Perhaps run ./bytecode_builtins_list_generator under gdb?

gdb does not help much, the bytecode_builtins_list_generator crash during startup, even before loading any :

gdb bytecode_builtins_list_generator -batch -ex 'set disable-randomization off' -ex 'catch throw' -ex 'catch signal SIGSEGV' -ex 'break main' -ex 'run gen/builtins-generated/bytecodes-builtins-list.h'

warning: Could not find DWO CU obj/bytecode_builtins_list_generator/generate-bytecodes-builtins-list.dwo(0x755ec394d72caf7b) referenced by CU at offset 0x0 [in module /workspace/bytecode_builtins_list_generator]
Catchpoint 1 (throw)
Catchpoint 2 (signal SIGSEGV)
Breakpoint 3 at 0x3ccec: file ../../src/builtins/generate-bytecodes-builtins-list.cc, line 110.
During startup program terminated with signal SIGSEGV, Segmentation fault.

What is more interesting is, after I compile the bytecode_builtins_list_generator using g++, it just works...

The build command:

buildtools/linux-arm64/gn gen out/wee8 '--args= is_debug=true v8_symbol_level=2 v8_optimized_debug=false is_clang=false custom_toolchain="//build/toolchain/linux/unbundle:default" use_custom_libcxx=false use_custom_libcxx_for_host=true use_sysroot=false use_glib=false v8_expose_symbols=true is_component_build=false v8_enable_i18n_support=false v8_enable_gdbjit=false v8_use_external_startup_data=false v8_enable_shared_ro_heap=false target_cpu="arm64"'

# Just build the bytecode_builtins_list_generator binary
ninja -C out/wee8 bytecode_builtins_list_generator

I'm wondering whether it's a page error

timonwong · 2022-01-07T10:17:00Z

Perhaps it's a PAGESIZE issue?

on good host machine (debian/ubuntu):

getconf PAGESIZE # 4096

on problem host machine (CentOS/RHEL):

getconf PAGESIZE # 65536

Some issues maybe related after searching:

[Bug]: Electron segmentation fault on CentOS7 aarch64(arm64) electron/electron#25387
Electron segmentation fault on CentOS/RHEL 8 arm64 electron/electron#21395
VSCode segmentation fault on CentOS8 non-4k pagesize aarch64(arm64) microsoft/vscode#108509

timonwong · 2022-01-07T13:54:45Z

My guess:

The sections headers from the generated binary (using llvm+clang):

 Program Headers:
   Type           Offset             VirtAddr           PhysAddr
                  FileSiz            MemSiz              Flags  Align
   PHDR           0x0000000000000040 0x0000000000000040 0x0000000000000040
                  0x0000000000000268 0x0000000000000268  R      0x8
   INTERP         0x00000000000002a8 0x00000000000002a8 0x00000000000002a8
                  0x000000000000001b 0x000000000000001b  R      0x1
       [Requesting program interpreter: /lib/ld-linux-aarch64.so.1]
   LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                  0x000000000001da4c 0x000000000001da4c  R      0x1000
   LOAD           0x000000000001da50 0x000000000001ea50 0x000000000001ea50
                  0x000000000000b750 0x000000000000b750  R E    0x1000
   LOAD           0x00000000000291a0 0x000000000002b1a0 0x000000000002b1a0
                  0x00000000000027c8 0x00000000000027c8  RW     0x1000
   LOAD           0x000000000002b968 0x000000000002e968 0x000000000002e968
                  0x0000000000000018 0x00000000000000f0  RW     0x1000
   DYNAMIC        0x000000000002b2e8 0x000000000002d2e8 0x000000000002d2e8
                  0x00000000000001f0 0x00000000000001f0  RW     0x8
   GNU_RELRO      0x00000000000291a0 0x000000000002b1a0 0x000000000002b1a0
                  0x00000000000027c8 0x0000000000002e60  R      0x1
   GNU_EH_FRAME   0x000000000001beac 0x000000000001beac 0x000000000001beac
                  0x00000000000005bc 0x00000000000005bc  R      0x4
   GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                  0x0000000000000000 0x0000000000000000  RW     0x0
   NOTE           0x00000000000002c4 0x00000000000002c4 0x00000000000002c4
                  0x0000000000000038 0x0000000000000038  R      0x4

Generated (PT_)LOAD overlaps for 64k page size:
PT_LOAD (RE) 0x1ea50 + 0x0b750 = 0x2a1a0
PT_LOAD (RW) Starts at 0x2b1a0. which just overlaps previous 0x0x2a1a0 with 64k page for different purposes (execute & write).

However, for gcc builds:

   LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                  0x000000000002d738 0x000000000002d738  R E    0x10000
   LOAD           0x000000000002d920 0x000000000003d920 0x000000000003d920
                  0x00000000000026f8 0x00000000000027d8  RW     0x10000

As you can see, PT_LOAD(RW) - PT_LOAD(RE) do not overlap under 64k page size

So it's ok for gcc builds...

Generated binaries (gcc & llvm):
xxx.tar.gz

timonwong · 2022-01-07T14:44:36Z

It's basically caused by the gn (or ninja), which adds max-page-size=4096 to the linker options:

[53/53] python3 "../../build/toolchain/gcc_link_wrapper.py" --output="./bytecode_builtins_list_generator" -- clang++ -pie -fuse-ld=lld -Wl,--fatal-warnings -Wl,--build-id -fPIC -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -Wl,-z,max-page-size=4096 -Wl,--icf=all -Wl,--color-diagnostics -Wl,--no-call-graph-profile-sort --target=aarch64-linux-gnu -no-canonical-prefixes -Werror -rdynamic -Wl,-z,defs -Wl,--as-needed -pie -Wl,--disable-new-dtags -Wl,-O2 -Wl,--gc-sections -o "./bytecode_builtins_list_generator" -Wl,--start-group @"./bytecode_builtins_list_generator.rsp"  -Wl,--end-group  -latomic -ldl -lpthread -lrt

Maybe switching to bazel will solve this problem, I'll try it later in a few days.

PiotrSikora · 2022-01-07T15:03:48Z

Thanks for investigating! This flag is added by the existing gn-based build system, so it shouldn't be an issue with #19275.

Otherwise, v8 compilation failed with the symptoms in envoyproxy/envoy#17100

timonwong added the triage Issue requires triage label Jun 23, 2021

alyssawilk added area/build and removed triage Issue requires triage labels Jun 23, 2021

github-actions bot added the stale stalebot believes this issue/PR has not been touched recently label Jul 23, 2021

github-actions bot closed this as completed Jul 30, 2021

timonwong mentioned this issue Jan 5, 2022

Unable to build arm64 generate_bytecode_builtins_list #19089

Closed

morlay mentioned this issue May 8, 2022

compile arm64 envoy error , error is com_googlesource_chromium_v8/BUILD.bazel:31:8 resf/istio#69

Closed

visualzhou added a commit to 10gen/envoy-serverless that referenced this issue Jan 12, 2023

Update wee8 compile script on ARM.

33e5417

Otherwise, v8 compilation failed with the symptoms in envoyproxy/envoy#17100

visualzhou added a commit to 10gen/envoy-serverless that referenced this issue Jan 12, 2023

Update wee8 compile script on ARM.

cba861c

Otherwise, v8 compilation failed with the symptoms in envoyproxy/envoy#17100

davidjb mentioned this issue Nov 30, 2023

[REQUEST] arm64 support Sparticuz/chromium#186

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to build arm64 binaries (segfault during v8 bytecode_builtins_list_generator) #17100

Unable to build arm64 binaries (segfault during v8 bytecode_builtins_list_generator) #17100

timonwong commented Jun 23, 2021 •

edited

Loading

github-actions bot commented Jul 23, 2021

github-actions bot commented Jul 30, 2021

maikoulin commented Nov 24, 2021

timonwong commented Jan 5, 2022

moderation commented Jan 5, 2022

timonwong commented Jan 6, 2022

moderation commented Jan 6, 2022

PiotrSikora commented Jan 6, 2022

timonwong commented Jan 7, 2022

timonwong commented Jan 7, 2022 •

edited

Loading

PiotrSikora commented Jan 7, 2022

timonwong commented Jan 7, 2022 •

edited

Loading

timonwong commented Jan 7, 2022 •

edited

Loading

timonwong commented Jan 7, 2022 •

edited

Loading

timonwong commented Jan 7, 2022 •

edited

Loading

PiotrSikora commented Jan 7, 2022

Unable to build arm64 binaries (segfault during v8 bytecode_builtins_list_generator) #17100

Unable to build arm64 binaries (segfault during v8 bytecode_builtins_list_generator) #17100

Comments

timonwong commented Jun 23, 2021 • edited Loading

github-actions bot commented Jul 23, 2021

github-actions bot commented Jul 30, 2021

maikoulin commented Nov 24, 2021

timonwong commented Jan 5, 2022

moderation commented Jan 5, 2022

timonwong commented Jan 6, 2022

moderation commented Jan 6, 2022

PiotrSikora commented Jan 6, 2022

timonwong commented Jan 7, 2022

timonwong commented Jan 7, 2022 • edited Loading

PiotrSikora commented Jan 7, 2022

timonwong commented Jan 7, 2022 • edited Loading

timonwong commented Jan 7, 2022 • edited Loading

timonwong commented Jan 7, 2022 • edited Loading

timonwong commented Jan 7, 2022 • edited Loading

PiotrSikora commented Jan 7, 2022

timonwong commented Jun 23, 2021 •

edited

Loading

timonwong commented Jan 7, 2022 •

edited

Loading

timonwong commented Jan 7, 2022 •

edited

Loading

timonwong commented Jan 7, 2022 •

edited

Loading

timonwong commented Jan 7, 2022 •

edited

Loading

timonwong commented Jan 7, 2022 •

edited

Loading