[#10942, #11033, #11034, #10964] Enable link-time optimization with C…

…lang 12

Summary:
Link-time optimization ( https://llvm.org/docs/LinkTimeOptimization.html ) allows for more aggressive optimizations, including inlining, compared to the shared library based model that we currently use.

This diff enables link-time optimization for the Clang 12 Linuxbrew-based release build for the yb-tserver executable only, producing a binary that statically links all object files needed by yb-tserver, including those that are included in the yb_pgbackend library. Third-party libraries are being linked statically but they are not LTO-enabled yet.

The linking of the final LTO-enabled binary is currently being done outside of the CMake build system, using the dependency_graph.py tool that can access the dependency graph of targets and object files, and therefore has all the information needed to construct the linker command line. This also gives us more flexibility customizing the linker command line compared to attempts to do this in the CMake build system. Moving this linking step to CMake may be a future project.

Refactored the dependency_graph.py script into multiple modules: dependency_graph.py, dep_graph_common.py, source_files.py, as well as lto.py (with the new LTO logic).

Also refactored master_main.cc and tablet_server_main.cc and extracted common initialization code to tserver/server_main_util.cc. It is in the tserver directory because the master code currently uses the tserver code.

For building LTO-enabled binaries, we need to use LLVM's lld linker. It has issues with our distributed compilation framework ( #11034 ). Fixing this by always running LLD-enabled linking commands locally and not on a remote build worker.

Various static initialization issues were identified as fixed as part of this work. If not fixed, these would result in the yb-tserver binary crashing immediately with a core dump.
- In consensus_queue.cc, the RpcThrottleThresholdBytesValidator function for validating the rpc_throttle_threshold_bytes flag was trying to access other flags before they were fully initialized. Moved this validation to the main program.
- The webserver_doc_root flag was calling yb::GetDefaultDocumentRoot() to determine its default value. Moved that default value determination to where the flag is being used.
- [ #11033 ] The INTERNAL_TRACE_EVENT_ADD_SCOPED macro, when invoked during static initialization, led to a crash in std::string construction. Added a new atomic trace_events_enabled for enabling trace events and only turned it on after main() started executing. The INTERNAL_TRACE_EVENT_ADD_SCOPED is a no-op before trace_events_enabled is set to true.
- [ #10964 ] The kGlobalTransactionTableName global constant of the YBTableName type relied on the statically initialized string constant, kGlobalTransactionsTableName, which turned out to be empty during initialization. As a result, the transaction status table could not be properly located. Changed kGlobalTransactionsTableName to be a `const char*`.

In addition, in the LTO-enable build, it became apparent that some symbols were duplicated between the gperftools library and the gutil part of YugabyteDB code ( #10956 ):
- AtomicOps_Internalx86CPUFeatures -- renamed to YbAtomicOps_Internalx86CPUFeatures
- RunningOnValgrind -- renamed to YbRunningOnValgrind
- ValgrindSlowdown -- renamed to YbValgrindSlowdown
- base::internal::SpinLockDelay, base::internal::SpinLockWake -- added a top-level yb namespace

To enable easily switching between regular and LTO binaries, we are updating yb-ctl to support YB_CTL_TSERVER_DAEMON_FILE_NAME and YB_CTL_MASTER_DAEMON_FILE_NAME environment variables. For example, by setting YB_CTL_TSERVER_DAEMON_FILE_NAME=yb-tserver-lto, you can tell yb-ctl to launch the tablet server using build/latest/bin/yb-tserver-lto. However, for the release package, the LTO-enabled yb-tserver executable will still be named yb-tserver, replacing the previous shared library based executable.

Another tooling change in this diff is how we handle the `--no-tests` flag passed to `yb_build.sh`. That flag results in setting the YB_DO_NOT_BUILD_TESTS environment variable to 1, and our CMake scripts skip all the test targets. However, it is easy to forget to keep specifying that flag. In this diff, we are storing the variable BUILD_TESTS in CMake's build cache, and reuse it during future CMake runs, without the developer having to specify `--no-tests`. It can be reset by setting YB_DO_NOT_BUILD_TESTS=0.

Test Plan:
Jenkins

```
# ./yb_build.sh --clang12 release
# build-support/tserver_lto.sh
# ldd build/latest/bin/yb-tserver-lto
	linux-vdso.so.1 (0x00007fff535bf000)
	libm.so.6 => /opt/yb-build/brew/linuxbrew-20181203T161736v9/lib/libm.so.6 (0x00007f1b85b7d000)
	libgcc_s.so.1 => /opt/yb-build/brew/linuxbrew-20181203T161736v9/lib/libgcc_s.so.1 (0x00007f1b85966000)
	libc.so.6 => /opt/yb-build/brew/linuxbrew-20181203T161736v9/lib/libc.so.6 (0x00007f1b855ca000)
	/opt/yb-build/brew/linuxbrew-20181203T161736v9/lib/ld.so => /lib64/ld-linux-x86-64.so.2 (0x00007f1b85e80000)
	libdl.so.2 => /opt/yb-build/brew/linuxbrew-20181203T161736v9/lib/libdl.so.2 (0x00007f1b853c6000)
	libpthread.so.0 => /opt/yb-build/brew/linuxbrew-20181203T161736v9/lib/libpthread.so.0 (0x00007f1b851a9000)
	librt.so.1 => /opt/yb-build/brew/linuxbrew-20181203T161736v9/lib/librt.so.1 (0x00007f1b84fa1000)
```
The yb-tserver-lto is ~326 MiB.

Microbenchmark
--------------
The test was done on a dual-socket Xeon E5-2670 machine (16 cores total, 32 hyper-threads) running AlmaLinux 8.5.
Details: https://gist.githubusercontent.com/mbautin/7f9784fb2ea4173539d2e2656cfe117f/raw
Results (CassandraKeyValue workload): 78K ops/sec with GCC 5.5, 85K ops/sec with Clang 12 without LTO, 104K ops/sec with Clang 12 with LTO.

Reviewers: sergei

Reviewed By: sergei

Subscribers: sergei, bogdan

Differential Revision: https://phabricator.dev.yugabyte.com/D14616

CMakeLists.txt

@@ -35,7 +35,6 @@ cmake_minimum_required(VERSION 3.7.0)
 
 project(YugabyteDB)
 
-
 if("$CACHE{YB_PCH_ON}" STREQUAL "")
   if("$ENV{YB_USE_PCH}" STREQUAL "1")
     set(YB_USE_PCH ON)
@@ -102,17 +101,31 @@ set(YB_ALL_DEPS "" CACHE INTERNAL "All dependencies" FORCE)
 # This is used to let the add_executable wrapper know if we're adding a test.
 set(YB_ADDING_TEST_EXECUTABLE "FALSE" CACHE INTERNAL "" FORCE)
 
-if ("$ENV{YB_DO_NOT_BUILD_TESTS}" STREQUAL "1")
-  message("Skipping building tests (YB_DO_NOT_BUILD_TESTS environment variable is set to 1)")
-  set(BUILD_TESTS OFF)
-else()
+if(NOT "$ENV{YB_DO_NOT_BUILD_TESTS}" STREQUAL "")
+  if("$ENV{YB_DO_NOT_BUILD_TESTS}" STREQUAL "1")
+    message("YB_DO_NOT_BUILD_TESTS is set to 1, will not build tests")
+    set(BUILD_TESTS OFF)
+  elseif("$ENV{YB_DO_NOT_BUILD_TESTS}" STREQUAL "0")
+    message("YB_DO_NOT_BUILD_TESTS is set to 0, will build tests")
+    set(BUILD_TESTS ON)
+  else()
+    message(FATAL_ERROR
+            "Invalid value of the YB_DO_NOT_BUILD_TESTS environment variable, expected 0 or 1 but "
+            "got '$ENV{YB_DO_NOT_BUILD_TESTS}'.")
+  endif()
+  set(BUILD_TESTS ${BUILD_TESTS} CACHE BOOL "Whether to build tests")
+elseif("$CACHE{BUILD_TESTS}" STREQUAL "")
   set(BUILD_TESTS ON)
+  message("Will build tests by default")
+else()
+  message("BUILD_TESTS from cache: ${BUILD_TESTS} (set YB_DO_NOT_BUILD_TESTS to 0 or 1 to change)")
 endif()
 
 parse_build_root_basename()
 if("${YB_BUILD_TYPE}" STREQUAL "")
   message(FATAL_ERROR "YB_BUILD_TYPE still not set after parse_build_root_basename")
 endif()
+
 message("YB_BUILD_TYPE: ${YB_BUILD_TYPE}")
 message("CMAKE_MAKE_PROGRAM: ${CMAKE_MAKE_PROGRAM}")
 
@@ -244,6 +257,7 @@ elseif (IS_GCC)
 else()
   message(FATAL_ERROR "Unknown compiler family: '${COMPILER_FAMILY}'.")
 endif()
+
 message("THIRDPARTY_INSTRUMENTATION_TYPE=${THIRDPARTY_INSTRUMENTATION_TYPE}")
 
 # Make sure third-party dependency is up-to-date.
@@ -301,6 +315,7 @@ endif()
 
 VALIDATE_COMPILER_TYPE()
 DETECT_BREW()
+enable_lto_if_needed()
 
 message("Using COMPILER_FAMILY=${COMPILER_FAMILY}")
 

build-support/common-build-env.sh

@@ -292,6 +292,11 @@ if [[ -n ${YB_LINUXBREW_DIR:-} ]]; then
   yb_linuxbrew_dir_origin=" (from environment)"
 fi
 
+yb_llvm_toolchain_url_origin=""
+if [[ -n ${YB_LLVM_TOOLCHAIN_URL:-} ]]; then
+  yb_llvm_toolchain_url_origin=" (from environment)"
+fi
+
 yb_llvm_toolchain_dir_origin=""
 if [[ -n ${YB_LLVM_TOOLCHAIN_DIR:-} ]]; then
   yb_llvm_toolchain_dir_origin=" (from environment)"
@@ -1146,7 +1151,9 @@ save_var_to_file_in_build_dir() {
     if [[ ! -d $BUILD_ROOT ]]; then
       mkdir -p "$BUILD_ROOT"
     fi
-    echo "$value" >"$BUILD_ROOT/$file_name"
+    if ! echo "$value" >"$BUILD_ROOT/$file_name"; then
+      fatal "Could not save value '$value' to file '$BUILD_ROOT/$file_name'"
+    fi
   fi
 }
 
@@ -1217,14 +1224,16 @@ download_thirdparty() {
 
 download_toolchain() {
   local toolchain_urls=()
-  if [[ -n ${YB_THIRDPARTY_URL:-} && ${YB_THIRDPARTY_URL##*/} == *linuxbrew* ]]; then
-    # TODO: get rid of this and always include linuxbrew_url.txt in the thirdparty archives that are
-    # built for Linuxbrew.
-    toolchain_urls+=( "https://github.com/yugabyte/brew-build/releases/download/\
-20181203T161736v9/linuxbrew-20181203T161736v9.tar.gz" )
-    if [[ -f $BUILD_ROOT/llvm_url.txt ]]; then
-      toolchain_urls+=( "$(<"$BUILD_ROOT/llvm_url.txt")" )
-    fi
+  local linuxbrew_url=""
+  if [[ -n ${YB_THIRDPARTY_DIR:-} && -f "$YB_THIRDPARTY_DIR/linuxbrew_url.txt" ]]; then
+    local linuxbrew_url_file_path="${YB_THIRDPARTY_DIR}/linuxbrew_url.txt"
+    linuxbrew_url="$(<"${linuxbrew_url_file_path}")"
+  elif [[ -n ${YB_THIRDPARTY_URL:-} && ${YB_THIRDPARTY_URL##*/} == *linuxbrew* ||
+          -n ${YB_THIRDPARTY_DIR:-} && ${YB_THIRDPARTY_DIR##*/} == *linuxbrew* ]]; then
+    # TODO: get rid of the hard-coded URL below and always include linuxbrew_url.txt in the
+    # thirdparty archives that are built for Linuxbrew.
+    local linuxbrew_url="https://github.com/yugabyte/brew-build/releases/download/"
+    linuxbrew_url+="20181203T161736v9/linuxbrew-20181203T161736v9.tar.gz"
   else
     for file_name_part in linuxbrew toolchain; do
       local url_file_path="$YB_THIRDPARTY_DIR/${file_name_part}_url.txt"
@@ -1234,6 +1243,14 @@ download_toolchain() {
       fi
     done
   fi
+
+  if [[ -n ${linuxbrew_url:-} ]]; then
+    toolchain_urls+=( "$linuxbrew_url" )
+  fi
+  if [[ -n ${YB_LLVM_TOOLCHAIN_URL:-} ]]; then
+    toolchain_urls+=( "${YB_LLVM_TOOLCHAIN_URL}" )
+  fi
+
   if [[ ${#toolchain_urls[@]} -eq 0 ]]; then
     return
   fi
@@ -1250,7 +1267,7 @@ download_toolchain() {
       is_linuxbrew=true
     else
       fatal "Unable to determine the installation parent directory for the toolchain archive" \
-            "named '$toolchain_url_basename'. Toolchain URL: '$toolchain_url'."
+            "named '$toolchain_url_basename'. Toolchain URL: '${toolchain_url}'."
     fi
 
     download_and_extract_archive "$toolchain_url" "$toolchain_dir_parent"
@@ -1890,6 +1907,9 @@ log_thirdparty_and_toolchain_details() {
     if is_linux && [[ -n ${YB_LINUXBREW_DIR:-} ]]; then
       echo "    YB_LINUXBREW_DIR: $YB_LINUXBREW_DIR$yb_linuxbrew_dir_origin"
     fi
+    if [[ -n ${YB_LLVM_TOOLCHAIN_URL:-} ]]; then
+      echo "    YB_LLVM_TOOLCHAIN_URL: $YB_LLVM_TOOLCHAIN_URL$yb_llvm_toolchain_url_origin"
+    fi
     if [[ -n ${YB_LLVM_TOOLCHAIN_DIR:-} ]]; then
       echo "    YB_LLVM_TOOLCHAIN_DIR: $YB_LLVM_TOOLCHAIN_DIR$yb_llvm_toolchain_dir_origin"
     fi
@@ -2437,8 +2457,16 @@ set_prebuilt_thirdparty_url() {
         fatal "Could not automatically determine the third-party archive URL to download."
       fi
       log "Setting third-party URL to $YB_THIRDPARTY_URL"
-
       save_var_to_file_in_build_dir "$YB_THIRDPARTY_URL" thirdparty_url.txt
+
+      if [[ -f $llvm_url_file_path ]]; then
+        YB_LLVM_TOOLCHAIN_URL=$(<"$llvm_url_file_path")
+        export YB_LLVM_TOOLCHAIN_URL
+        yb_llvm_toolchain_url_origin=" (determined automatically based on the OS and compiler type)"
+        log "Setting LLVM toolchain URL to $YB_LLVM_TOOLCHAIN_URL"
+        save_var_to_file_in_build_dir "$YB_LLVM_TOOLCHAIN_URL" llvm_url.txt
+      fi
+
     else
       log "YB_THIRDPARTY_URL is already set to '$YB_THIRDPARTY_URL', not trying to set it" \
           "automatically."
@@ -2553,6 +2581,10 @@ is_apple_silicon() {
   return 1
 }
 
+should_use_lto() {
+  using_linuxbrew && [[ "${YB_COMPILER_TYPE}" == "clang12" && "${build_type}" == "release" ]]
+}
+
 # -------------------------------------------------------------------------------------------------
 # Initialization
 # -------------------------------------------------------------------------------------------------

build-support/compiler-wrappers/compiler-wrapper.sh

@@ -287,6 +287,10 @@ has_yb_c_files=false
 compiler_args_no_output=()
 analyzer_checkers_specified=false
 
+linking=false
+use_lld=false
+lld_linking=false
+
 while [[ $# -gt 0 ]]; do
   is_output_arg=false
   case "$1" in
@@ -335,16 +339,31 @@ while [[ $# -gt 0 ]]; do
     -analyzer-checker=*)
       analyzer_checkers_specified=true
     ;;
+    -fuse-ld=lld)
+      use_lld=true
+    ;;
   esac
   if ! "$is_output_arg"; then
     compiler_args_no_output+=( "$1" )
   fi
   shift
 done
 
-if [[ $output_file != *.o && ${#library_files[@]} -gt 0 ]]; then
-  input_files+=( "${library_files[@]}" )
-  library_files=()
+if [[ $output_file == *.o ]]; then
+  # Compiling.
+  linking=false
+else
+  linking=true
+fi
+
+if [[ $linking == "true" && $use_lld == "true" ]]; then
+  lld_linking=true
+fi
+
+if [[ $YB_COMPILER_TYPE == clang* && $output_file == "jsonpath_gram.o" ]]; then
+  # To avoid this error:
+  # https://gist.github.com/mbautin/b943fb426bfead7388dde17ddb1b0fa7
+  compiler_args+=( -Wno-error=implicit-fallthrough )
 fi
 
 # -------------------------------------------------------------------------------------------------
@@ -362,9 +381,12 @@ if [[ $HOSTNAME == build-worker* ]]; then
   is_build_worker=true
 fi
 
+# If linking with LLVM's lld linker, do it locally.
+# See https://github.com/yugabyte/yugabyte-db/issues/11034 for more details.
 if [[ $local_build_only == "false" &&
       ${YB_REMOTE_COMPILATION:-} == "1" &&
-      $is_build_worker == "false" ]]; then
+      $is_build_worker == "false" &&
+      $lld_linking == "false" ]]; then
 
   trap remote_build_exit_handler EXIT
 
@@ -552,6 +574,12 @@ run_compiler_and_save_stderr() {
 # -------------------------------------------------------------------------------------------------
 # Local build
 
+if [[ $output_file =~ libyb_pgbackend*  ]]; then
+  # We record the linker command used for the libyb_pgbackend library so we can use it when
+  # producing the LTO build for yb-tserver.
+  echo "${compiler_args[*]}" >"link_cmd_${output_file}.txt"
+fi
+
 if [[ ${build_type:-} == "asan" &&
       $PWD == */postgres_build/src/backend/utils/adt &&
       # Turn off UBSAN instrumentation in a number of PostgreSQL source files determined by the

build-support/jenkins/build-and-test.sh

@@ -725,6 +725,22 @@ export YB_GZIP_PER_TEST_METHOD_LOGS=1
 export YB_GZIP_TEST_LOGS=1
 export YB_DELETE_SUCCESSFUL_PER_TEST_METHOD_LOGS=1
 
+if should_use_lto; then
+  log "Using LTO"
+  (
+    set -x
+    "$YB_SRC_ROOT/python/yb/dependency_graph.py" \
+        --build-root "$BUILD_ROOT" \
+        --file-regex "^.*/yb-tserver$" \
+        --lto-output-suffix="" \
+        link-whole-program
+    ls -l "$BUILD_ROOT/bin/yb-tserver"
+    ldd "$BUILD_ROOT/bin/yb-tserver"
+  )
+else
+  log "Not using LTO: YB_COMPILER_TYPE=${YB_COMPILER_TYPE}, build_type=${build_type}"
+fi
+
 if [[ $YB_COMPILE_ONLY != "1" ]]; then
   if spark_available; then
     if [[ $YB_BUILD_CPP == "1" || $YB_BUILD_JAVA == "1" ]]; then
@@ -740,11 +756,14 @@ if [[ $YB_COMPILE_ONLY != "1" ]]; then
         test_conf_path="$BUILD_ROOT/test_conf.json"
         # YB_GIT_COMMIT_FOR_DETECTING_TESTS allows overriding the commit to use to detect the set
         # of tests to run. Useful when testing this script.
-        "$YB_SRC_ROOT/python/yb/dependency_graph.py" \
-            --build-root "$BUILD_ROOT" \
-            --git-commit "${YB_GIT_COMMIT_FOR_DETECTING_TESTS:-$current_git_commit}" \
-            --output-test-config "$test_conf_path" \
-            affected
+        (
+          set -x
+          "$YB_SRC_ROOT/python/yb/dependency_graph.py" \
+              --build-root "$BUILD_ROOT" \
+              --git-commit "${YB_GIT_COMMIT_FOR_DETECTING_TESTS:-$current_git_commit}" \
+              --output-test-config "$test_conf_path" \
+              affected
+        )
         run_tests_extra_args+=( "--test_conf" "$test_conf_path" )
         unset test_conf_path
       fi

build-support/tserver_lto.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+
+. "${BASH_SOURCE%/*}/common-build-env.sh"
+
+activate_virtualenv
+set_pythonpath
+
+set -x
+"$YB_SRC_ROOT/python/yb/dependency_graph.py" \
+    --build-root "$YB_SRC_ROOT/build/release-clang12-linuxbrew-dynamic-ninja" \
+    --file-regex "^.*/yb-tserver$" \
+    link-whole-program \
+    "$@"

build-support/yb_release.py

@@ -47,7 +47,7 @@ def main():
                              'false if --build_target is specified, true otherwise.')
     parser.add_argument('--destination', help='Copy release to Destination folder.')
     parser.add_argument('--force', help='Skip prompts', action='store_true')
-    parser.add_argument('--commit', help='Custom specify a git commit.')
+    parser.add_argument('--commit', help='Specifies a custom git commit to use in archive name.')
     parser.add_argument('--skip_build', help='Skip building the code', action='store_true')
     parser.add_argument('--build_target',
                         help='Target directory to put the YugaByte distribution into. This can '
@@ -221,8 +221,13 @@ def main():
     # This points to the release manifest within the release_manager, and we are modifying that
     # directly.
     release_util = ReleaseUtil(
-        YB_SRC_ROOT, build_type, build_target, args.force, args.commit, build_root,
-        args.package_name)
+        repository=YB_SRC_ROOT,
+        build_type=build_type,
+        distribution_path=build_target,
+        force=args.force,
+        commit=args.commit,
+        build_root=build_root,
+        package_name=args.package_name)
 
     system = platform.system().lower()
     library_packager_args = dict(

cmake_modules/YugabyteFunctions.cmake

@@ -729,3 +729,30 @@ function(parse_build_root_basename)
         "but CMAKE_SYSTEM_PROCESSOR is ${CMAKE_SYSTEM_PROCESSOR}")
   endif()
 endfunction()
+
+macro(enable_lto_if_needed)
+  if(NOT DEFINED COMPILER_FAMILY)
+    message(FATAL_ERROR "COMPILER_FAMILY not defined")
+  endif()
+  if(NOT DEFINED USING_LINUXBREW)
+    message(FATAL_ERROR "USING_LINUXBREW not defined")
+  endif()
+  if(NOT DEFINED YB_BUILD_TYPE)
+    message(FATAL_ERROR "YB_BUILD_TYPE not defined")
+  endif()
+
+  if("${YB_BUILD_TYPE}" STREQUAL "release" AND
+     "${COMPILER_FAMILY}" STREQUAL "clang" AND
+     "${YB_COMPILER_TYPE}" STREQUAL "clang12" AND
+     USING_LINUXBREW AND
+     NOT APPLE)
+    message("Enabling full LTO and lld linker")
+    ADD_CXX_FLAGS("-flto=full -fuse-ld=lld")
+  else()
+    message("Not enabling LTO: "
+            "YB_BUILD_TYPE=${YB_BUILD_TYPE}, "
+            "COMPILER_FAMILY=${COMPILER_FAMILY}, "
+            "USING_LINUXBREW=${USING_LINUXBREW}, "
+            "APPLE=${APPLE}")
+  endif()
+endmacro()

codecheck.ini

@@ -38,11 +38,15 @@ included_regex_list =
     ^python/yb/command_util[.]py$
     ^python/yb/common_util[.]py$
     ^python/yb/compile_commands[.]py$
+    ^python/yb/dep_graph_common[.]py$
     ^python/yb/dependency_graph[.]py$
     ^python/yb/library_packager[.]py$
     ^python/yb/linuxbrew[.]py$
+    ^python/yb/lto[.]py$
     ^python/yb/os_detection[.]py$
+    ^python/yb/release_util[.]py$
     ^python/yb/run_pvs_studio_analyzer[.]py$
+    ^python/yb/source_files[.]py$
     ^python/yb/thirdparty_tool[.]py$
     ^python/yb/tool_base[.]py$
     ^python/yb/yb_dist_tests[.]py$

ent/build-support/upload_package.sh

@@ -100,8 +100,9 @@ upload_package() {
       return
     fi
 
-    if is_linux && [[ $YB_COMPILER_TYPE != "gcc" ]]; then
-      log "Skipping package upload for a non-gcc build on Linux (compiler type: $YB_COMPILER_TYPE)"
+    if is_linux && [[ $YB_COMPILER_TYPE != "gcc" &&
+                      $YB_COMPILER_TYPE != "clang12" ]]; then
+      log "Skipping package upload on Linux (compiler type: $YB_COMPILER_TYPE)"
       return
     fi
   fi